MailHandler — A stopwatch for email delivery

We have been testing email delivery for a very long time, but we lacked a simple Ruby library for email delivery testing. To meet our needs, we built MailHandler, a simple Ruby library and we decided to share it with you.

History behind MailHandler #

When testing Postmark, the first thing on our minds is always successful and fast email delivery. This is why email delivery is the main focus of our test suite.

In order to test email delivery easily, we needed a library which would allow us to send and search for an email, and gather/expose statistics about sending and receiving at the same time.

Mail gem and MailHandler #

We found a library that did most of the job for us - the Ruby Mail gem. It's truly an awesome library, it allows you to do bunch of things, parse an email from string, send emails, search for emails and so on.

The problem with the mail gem is that the search, although not limited, did not work for us. This is what a search with the Mail gem would look like:

Mail.defaults do
  retriever_method :imap, :address    => "imap.example.com",
                          :port       => 993,
                          :user_name  => 'username',
                          :password   => 'password',
                          :enable_ssl => true
end

emails = Mail.find(:count => 10, :keys => ['SUBJECT','email subject'])

The code snippet represents how we would search for an email, and retrieve the emails found. The problem here was that these are one time only searches.

We needed a library which can be configured to search for an email for a given amount of time. The search should finish once the email is found or time for email searching exceeded. Plus we wanted the search interface to be easier to use.

In order to have these search options, we built an interface on top of the Mail gem.

With MalHandler search calls would look like this:

email_receiver = MailHandler.receiver(:imap) do |checker|
  checker.address = 'imap.example.com'
  checker.port = 993
  checker.username = 'username'
  checker.password = 'password'
  checker.use_ssl  =  true
end

email_receiver.find_email(:by_subject => 'Welcome to Postmark', :archive => true)

Now, these are very similar, but with one significant difference - this search will search for an email by default for a maximum of four minutes (the length of time is configurable of course).

If an email isn't found after four minutes, the search will be finished and with result that email was not found. Also, the search options in MailHandler were more natural for us to use.

Making MailHandler more versatile #

Solving the search problem was one of our goals, but we also wanted to be able to search for emails locally. We wanted to be able to search for emails on your local machines, for example when a mailbox is setup to be a simple folder on your computer and not a remote mailbox (which you would reach by protocols like IMAP). Example of this would be if you have something like postfix setup on your side, where emails would be stored as files in a folder.

Search by local folder would allow us to test delivery fast, in case we want to ignore big email providers like gmail in our testing. We needed to extend our library to add a support to one more type of search, which would be a search by files in certain folder on local machine.

This is where the power of our library would come in. All you would do is initiate email_receiver with folder searcher, instead of searcher by IMAP protocol.

email_receiver = MailHandler.receiver(:folder) do |checker|
    checker.inbox_folder = "/folder/mailbox/inbox/"
    checker.archive_folder = "/folder/mailbox/inbox/archive"
end

The checker variables in the example above are configured in the following way:

• "checker.inbox_folder" accepts string which represents the local path where the emails are stored
• "checker.archive_folder" accepts string which represents the local path where the emails which are found as part of the search are moved.

In this example, MailHandler will search for an email content through files in "/folder/mailbox/inbox/" folder.

Once the email content is found, files where the content is found would be moved to the new location "/folder/mailbox/inbox/archive" and deleted from original location. This is similar to the way the archive option works for the IMAP protocol.

Please note that these are local folders, meaning that if you use this type of search with MailHandler, the search has to be executed on the same machine where the folders live. Because the search is not remote like it would be using IMAP, it can be much faster.

Other than configuring the folders and handling the files, from library side the search would work the same way as searching by IMAP protocol. You would be able to search for an email easily in a mailbox by IMAP protocol, or in a local mailbox by folders and files. The choice can be made by only by changing couple of lines of code. Only limitation of searching by folder is that it doesn't support unicode characters for now.

MailHandler search goodies #

Once we added the main feature of the library, searching, we also needed to save search details by gathering data from our search method. After each search, MailHandler will give you results like when the search was started, finished, and how long it took.

Now that we had a simple search tool, we wanted to be able to get some sort of notifications about how the search is doing while it's running. In order to solve that, we added two types of notifications to the MailHandler gem:

• Notification by console on how email search is doing
• Notification by email if email search is taking too long

Notifications by email would be sent out to email addresses configured to receive the notifications. By default, if the search takes more than one minute, an email notification would be sent to those addresses. If an email is found, but the search took longer than one minute, we would send a second notification to the configured email addresses.

What we wanted to do is make MailHandler more than just an email delivery test tool. Since we already had search by period of time in place, we figured why not make the tool more versatile, and provide option to send notifications about any long searches. This lets us use MailHandler outside of our test suite.

By default, notifications are not used when searching for an email. They can be added optionally, as needed. For example, if you would like to receive notifications, you could use the following code:

email_receiver = MailHandler.receiver(:folder) do |checker|
    checker.inbox_folder = "/folder/mailbox/inbox/"
    checker.archive_folder = "/folder/mailbox/inbox/archive"
end

email_sender = MailHandler.sender(type) do |dispatcher|
    dispatcher.api_token = '1234567a-1234-1234-1234'
end

email_receiver.add_observer(MailHandler::Receiving::Notification::Email.new(email_sender,'ibalosh@example.com', you@example.com'))

With the code above, we create MailHandler email searching object — in this case we would use folder search. After that we create MailHandler sender object, which we will use for sending notifications. In the case above, notifications would be sent by Postmark API.

Once the notification sender is configured, we can add it to email searching object along with the list of email addresses we would like to be notified about any delays.

In the diagram below, we can see how the email search notification logic looks like:

An email notifications would be sent once each of the thresholds are reached. In our case, we have thresholds of 60 seconds. The second threshold is based on how long the search lasts, in the example above that would be 240 seconds. This is the default time in MailHandler to search for email.

We would send three notification emails taking the above thresholds in the account:

• Email was not found in over 60 seconds
• Email was found after 80 seconds for example
• Email was not found in 240 seconds

Adding console notifications is similar, except there is no need for email addresses or senders, since the data about the status of the delay notification is displayed on the screen where the test is running. That code would look like this:

email_receiver.add_observer(MailHandler::Receiving::Notification::Console.new)

Console notifications are simpler, since we need only the status on how search is doing. They will show every fifteen seconds how long the search has taken to receive an email up until the search is done.

MailHandler supports sending email too #

In order to make MailHandler an all around tool for email delivery testing, we added an interface for sending email too. The sending interface was built on top of mail gem and postmark gem.

This allowed an easy switch between Postmark sending and simple SMTP sending through a single tool. On top of that, analytics are available here too just like when searching, you can see how long it took to send an email, when sending was started, and when was it finished.

How does it work? #

How MailHandler works can be seen in code snippets above. In the example here though, we will show you how you could use the library in Ruby with RSpec to test delivery.

require 'mailhandler'
require 'rspec'

describe 'Email' do

  let(:email) {
    Mail.new do
      from     'me@example.com'
      to       'you@example.com'
      subject  'Here is the image you wanted'
      body     'test email'    
    end
  }

  let(:email_receiver) {

    MailHandler.receiver(:imap) do |checker|
      checker.address = 'imap.example.com'
      checker.port = 993
      checker.username = 'username'
      checker.password = 'password'
      checker.use_ssl  =  true
    end

  }

  let(:email_sender) {
    MailHandler.sender(type) do |dispatcher|
      dispatcher.api_token = '1234567a-1234-1234-1234'
    end
  }

  it 'search by subject' do

    emailsender.send_email(email)

    aggregate_failures "email delivery verification" do
      expect(email_receiver.find_email(:by_subject => email.subject, :archive => true)).to be true
      expect(email_receiver.search.duration).to be < 240
    end

  end

end

In the code example above, there are couple of important sections. We will describe each for you.

let(:email)

#

This section describes a memoized helper method in which we will create an email we will send in the test case.

let(:email_receiver)

#

This section describes how we will define the MailHandler email receiver, which we will use for searching for a specific email in certain mailbox.

let(:email_sender)

#

This section describes how we will define the MailHandler sender, which we will use for sending an email. In the example above, we will use the Postmark API to send an email.

The last section starting with "it" keyword describes the test. In this code section, we will send an email which we described in one of memoized helper methods. Once we send an email, we will search for it by "find" method. The search will be done by searching by subject. We will verify that the email was found and we will make sure that the email was found in less than 240 second.

Let us know what you think? #

MailHandler is available at https://github.com/activecampaign/mailhandler with detailed documentation about all it's features. You can download it from GitHub or use it as a Ruby gem. Let us know what you think? Don't hesitate to ask any questions or suggest any improvements.