Why we removed Inbox delivery tests from our status page

Last week we decided to remove inbox rates on our status page. In order to avoid confusion, I want to explain why we did it. 

Subscribe to receive future product updates.

If you are not familiar with the tests, we used to have a section on our status page that showed delivery rates to AOL, Outlook, Yahoo, Apple, and Gmail. Using a service called 250ok, it would report whether our emails were getting to the inbox, going to spam, or went missing entirely. It works by having a list of “seed” addresses at each ISP, which we then send email to and they monitor if it was delivered (or not). It was a nice visualization:

After nearly two years of running our transparent status page we realized these inbox tests were misleading. In most cases it would always show 100% inbox rates. And in some rare cases it might show issues only for us to find out it was a problem with our tool that sends the tests or with the seed address bouncing due to other circumstances. What we realized is that seed tests are not a great tool for showing overall delivery results for Postmark. Instead, our Time to Inbox metrics provide extremely relevant data. 

There are several reasons why seed tests are not a great general metric for our status page:

1. Email delivery is specific to domain reputation, content, and even timing. So a general seed test is misleading unless we are using your own domain and data.
2. Most seed tests only run a few times a day, which is not granular enough for monitoring email streams. In comparison, our Time to Inbox tests run every five minutes.
3. With a good domain reputation, and simple content, a consistent seed test will rarely show missing or spam issues to a major ISP. 
4. If a seed test does show missing emails, it is usually due to queue build up and delays. Our Time to Inbox tests show this better than anything else. 
5. In the cases where our status page displayed missing emails for a provider, it was always due to a problem with the actual seed address bouncing due to an unrelated issue. It was a recipient issue, not an ESP / ISP issue.
6. And finally, if a seed test is showing spam rates at larger ISPs, it means things are on fire for an ESP, in which case we would have posted a status message about it.

With these reasons in mind, we constantly found ourselves fixing the seed addresses when problems showed up on our status page. So when inbox rates always showed 100% people might roll their eyes thinking it is bogus, but when inbox rates displayed missing emails, it was not a real issue. 

Time to Inbox is a better gauge of performance  #

So instead of seed tests and inbox rates, we kept our Time to Inbox graphs. These tests use our open source MailHandler tool to send emails every five minutes to the major ISPs. We then track the time it takes from sending the request to our servers, to the time it arrives in the inbox at the ISP. Compared to seed tests, this brings a few advantages:

1. They are more frequent, so we can see changes over time based on our application and queue performance. 
2. We assume 100% delivery, because we use a domain with good reputation and simple content.
3. The tests go across all of our shared IPs for an accurate model of our queues and performance. 
4. If issues arise, it will show the spike and length of time, which is easy for customers to react on.
5. The tests show data that better represents the responsibility of the ESP, to maintain great performance and reputation to deliver emails quickly to the inbox. 

A good example is what happened with AOL last Friday. In the morning they had issues processing email on their side. The emails were accepted immediately from Postmark, but the time to arrive in the inbox spiked dramatically. We (and you) could clearly see this on our status page:

Realizing this, we posted a status message to inform our customers of the issue and we updated it once AOL fixed the issues. A seed test might have never caught this, and the duration would have been difficult to see. By using Time to Inbox tests, we were able to quickly inform customers and give them some peace of mind. It makes a big difference when you can communicate these things with an infrastructure product. 

So when are seed tests relevant? #

We still use 250ok seed testing internally. I see it more as a pulse on delivery overall since we send it across multiple domains, multiple sets of IPs, and 100+ providers including the corporate like Mimecast, Trendmicro, Symantec, etc. It’s a single tool of many that helps us monitor and understand our trends. There are times when we do see drops from certain providers on certain domains or IPs and we have to investigate. 

I also recommend seed testing for your own emails. This way you can track your individual reputation across many ISPs. And since we open sourced MailHandler, go ahead and set that up as well.