Outbound SMTP Outage on September 15-16, 2024
As a followup to the SMTP Outage on September 15, 2024, we felt it important to provide additional detail and context into this incident. During this time, some customers were unable to send emails, the core functionality of our platform. This is unacceptable and as a valued Postmark customer, we apologize for the impact this has had on your business operations.
A brief timeline and summary of the incident: #
- The SSL certificates for the Postmark SMTP servers expired at midnight GMT on Monday September 16th. As a result, customers using SMTP were unable to send emails from 0:00 UTC until 9:40 UTC.
- A synthetic test detected that the SMTP endpoint was not sending emails and an alert was triggered at ~00:08 UTC. An engineer responded, but misdiagnosed the problem and believed the alert was a false alarm.
- At ~4:19 UTC another engineering team responded to an escalation and began addressing the issue.
- At ~9:16 UTC the certificates were fully updated and service was again functional.
What are we doing to prevent this in our future? #
- An audit of all certificates is being performed to ensure automatic renewal.
- Updating our escalation paths within our engineering team to ensure faster time to resolution in any future incidents.
- Updating our internal communication channels between teams to ensure more frequent and timely updates to our customers.
