Why pre-defined success metrics are so important in product development

Last month we released a new DNS Settings page. In the announcement post I mentioned that our goal with the redesign was to help customers send more compliant emails, while also addressing the fear of messing with your DNS. Instead of saying “just trust us and do it!” we wanted to face the challenge head-on by providing clear instructions and feedback on what you need to do, and why. A couple of weeks ago Chris explained another major part of this process — why we don’t require customers to add SPF records to their DNS any more.

That all sounds very nice, but if you’re like me you might read those posts and say to yourself, “ok fine… but did it actually work!?” Companies are less inclined to share their success metrics with the world. 

In fact, it’s often common practice to not even have success metrics for new features, which makes it impossible to even figure out if what you did had a positive impact.

So in this post I wanted to follow up and talk a little bit about our process for establishing success metrics, how we defined them for the DNS Settings project, and how we’re currently doing on those.

Every major feature we work on at Postmark starts with a job story, instead of the usual “user story”. This keeps us focused on the outcomes that customers are trying to achieve when they use our product. For the DNS Settings page our job story was this:

When I set up authentication for my email domains I want to know exactly what to do and get real-time feedback on progress so I can be confident that I follow best practices for email.

This was derived from the primary goal of helping customers send more compliant email, which helps to keep deliverability high. In order to have good delivery for your email you need to do stuff with your DNS. But DNS changes are often hard to get right, and never fun. So our goal was to go way further than just “do this”, and give you real-time feedback on what might be going wrong along the way.

Setting the goal (the job story) is only one part of the equation, though. The other part is the all-important question teams so often forget to ask when we work on projects… How will we know if this feature was successful? For us, this all came down to the number of people who set up the necessary records in their DNS. Increasing that measure would mean that more people are implementing best practices for email delivery. That is good for customers, and good for us. So we started with a simple metric:

Our ultimate goal is to increase the number of users who have DKIM records and Custom Return-Paths set up for their domains by 40%

The job story, along with the success metrics, become the guiding light for the design and implementation of the feature. It’s a framework for decision-making when we need to make choices. It’s the constraints we need to make sure we don’t add things to the product that we don’t need.

You can read more about the eventual solution in the announcement blog post. In this post I want to share some of the results we saw. Once the new page was live for a few weeks, we were able to pull some metrics and see how we did… 

The graph below shows the number of Return-Path and DKIM confirmations per week, starting in July:

Since we launched the new DNS Settings page on October 16 we have seen a definite increase in the number of people who are able to successfully add these records. The big jump in Return-Path confirmations was expected since we essentially replaced SPF with Return-Path. But we were especially excited to see the increase in DKIM confirmations a well.

There are other secondary metrics we’re looking at as well (such as customer support request volume and the speed of adding records once an account is created), but based on this we’re ready to call the project a success.

Improving the DNS Settings page might not, on its surface, sound like the most exciting feature in the world. But it’s something that we know has a real impact on our customers’ ability to send effective email, as well as our ability to maintain and strengthen our overall delivery reputation. And that’s what makes it worth investing time and effort in. We’re not done with this page, either. There are always going to be some outstanding tasks that we didn’t get to in the first release, and it’s dangerous to just let those sit and move on to something else. 

But that’s a topic for another post. For now, we celebrate…