Better bounce categorization in Postmark

Bounces are a big part of troubleshooting delivery issues. We provide our customers a lot of information about bounces:

• We provide a specific bounce type so that there is a specific reason the bounce occurred.
• We provide the full bounce message, exactly what our servers receive.
• We store bounced messages forever because of their value in troubleshooting.

When we detect a Hard Bounce or a Spam Complaint for a particular address, we will deactivate the address. This helps us keep delivery rates high as it gives Postmark a good reputation with ISPs. It also helps our customers so that credits are not spent to send email to addresses that don’t exist. However, it becomes a problem when we incorrectly parse a bounce as a Hard Bounce. Then a legitimate user’s address is deactivated and won’t receive emails.

I’ll describe the recent efforts Postmark has put into reworking our bounce processing and categorization so that our customers have the best deliverability we can provide.

Testing #

Bounce processing is very important to us and our customers. To have confidence in our changes, we tested, tested and tested again. Our efforts were focused around answering two questions: how to verify our bounce processing continued to function properly and how could we get the most accurate categorization.

Testing Functionality #

To make sure our new bounce categorization method worked as we expected it to work, we launched it side-by-side our old bounce categorization. Our code compared the results of the two categorization methods and logged out the information when they differed. After a few weeks, we were able to verify our new bounce categorization was functioning as we expected it to and take out the old categorization method.

Testing Accuracy #

We tested out several different code libraries for bounce processing. Since Postmark stores bounces forever, our testing portfolio consisted of tens of thousands of different bounces and compared the new result to the old result. After a lot of csv files were analyzed, we came to the conclusion that no one tool was accurate enough. We needed a dynamic way to pull in multiple sources of input, which I’ll describe in the next section.

Multiple Sources of Input #

ListNanny.Net #

List Nanny is a .NET library we use to parse bounces. It parses out the SMTP diagnostic code and other valuable information from the raw message. It also provides a base bounce type that we use. List Nanny has worked very well for us and provides a base for the rest of the bounce parsing process.

PowerMTA Categories #

Postmark uses PowerMTA as part of our SMTP server infrastructure. When a bounce goes through PowerMTA, it adds a bounce category to the message. PowerMTA bounce categories are interesting because it doesn’t try to bin messages into Hard Bounce/Soft Bounce/etc. It uses categories like “policy-related” and “quota-issues,” which makes the category very useful in cases where we want to recategorize the bounce to something more meaningful.

Regular Expressions #

Any parsing solution wouldn’t be complete without some regexes! Luckily, Postmark only has a few regexes to detect certain diagnostic codes and specific content in the message body.

Tying It All Together #

Take the following example:

Action: failed
Status: 5.7.1 (delivery not authorized)
Remote-MTA: dns;example.com (X.X.X.X)
Diagnostic-Code: smtp;550 5.7.1 The user or domain that you are sending to (or from) has a policy that prohibited the mail that you sent.
X-PowerMTA-BounceCategory: policy-related

This is technically a Hard Bounce because the diagnostic code is in the 5.X.X range and the recipient didn’t receive the message. Using the diagnostic code and the easy to parse PowerMTA category, we can see that this individual message was blocked for policy reasons on the recipient’s domain, but future messages might have different content and not be blocked. In this case, we don’t want to deactivate the recipient’s address and we will recategorize the bounce type to Blocked instead of Hard Bounce. We have a lot of special cases like this in Postmark that take the form of:

If   List Nanny says it's a hard bounce
And  the PowerMTA category is policy-related
Then recategorize the bounce as blocked

And these types of If-Then statements were hard coded in our application. This works fine, but when we want to add more recategorization rules or change existing ones it means we have to redeploy the application. These types of If-Then statements were fairly easy to formalize in a JSON document that gets stored in CouchDB.

{
  "Description": "Policy Related",
  "Preconditions": {
    "BounceType": "HardBounce",
    "PowerMtaBounceCategory": "policy-related"
  },
  "ActualBounceType": "Blocked"
}

Our bounce processing code now just reads these documents from CouchDB and parses them into functions. Adding or changing a rule is a simple task.

What This Means to Our Customers #

This means we are able to drastically reduce the number of messages that get miscategorized as a Hard Bounce and so fewer addresses that get deactivated. It means we are able to react much faster when we discover areas where our categorization isn’t 100% accurate. It means more messages getting to their intended recipients. It means our customers are able to spend less time worrying about bounces and more time working on their business!