For a few years now, I've run a hosting co-operative with a few friends. Although the cost savings versus all renting VMs individually are probably marginal at best these days, one of the nice things about it is the chance to run things like our incoming MX on one machine only, instead of all having to run our own anti-spam and other measures. The incoming mail is handled by Exim, and each user of our system can add domains for which mail is processed. They get to toggle SMTP-time rejection of spam and viruses, and specify the final destination machine for incoming mail to their domain.

This has all been working well for over two years, but occasionally something has to change: a few months ago, we got rid of sender verify callouts, now widely considered abusive by SMTP server admins, and more recently we added support for tagging messages with headers to say if they passed or failed DKIM verification. And every time I make such a change, I worry that I might have inadvertently broken something. This server handles mail for 30 domains and 8 people, some of who rely on it to run businesses! Panic!

I usually end up reassuring myself by doing some ad-hoc testing by hand after reconfiguring the server. At the most basic level, whatever your SMTP server is, you can use netcat to have a conversation with it on port 25:

d@s:~$ nc localhost 25
220 mailserver.splice.org.uk ESMTP Exim 4.71 Sat, 17 Mar 2012 09:51:20 +0000
HELO localhost
250 mailserver.splice.org.uk Hello localhost [127.0.0.1]
MAIL FROM: <>
250 OK
RCPT TO: [email protected]
550-Callout verification failed:
550 550 Unrouteable address
QUIT
221 mailserver.splice.org.uk closing connection

And there, I've just convinced myself that one of our features is still working: the mailserver should call forward to the final destination for mail to @dnorth.net addresses to check the local part ('someaddress' in this case) is valid, and reject the message up-front if it's not.

Exim also has a load of other toys you can take advantage of: say I want to check how mail to [email protected] is routed:

d@s:~$ exim4 -bt [email protected]
R: hubbed_hosts for dnorth.net
[email protected]
 router = hubbed_hosts, transport = remote_smtp
 host mx-internal.dnorth.net [192.0.2.100]

(IP addresses changed for example purposes, obviously)

And finally, there's debug mode: you can run

exim4 -bhc

to run a complete 'fake' SMTP session as though you were connecting from the given IP address. You can send messages, but they won't actually go through, and exim prints a lot of debug output to give you a clue as to its inner workings as it decides how to route the message.

This is all very well, but a quick brainstorming session gives a list of over 30 things I might want to check about my mailserver:

  • Basic check that mail is accepted to our domains
  • Only existent addresses on our domains should have mail accepted
  • Domains with SMTP-time spam rejection on should have spam rejected
  • Same for viruses
  • Same for greylisting
  • ...

Testing all these by hand isn't going to fly, so what tools can we find for automating it? A bit of Googling turns up swaks, which looks quite handy, but suffers from two drawbacks for me: first, it's a bit low-level, and a collection of scripts calling it will be a bit difficult to read and maintain for testing all 30 of my assertions. Second, it really sends the e-mails in the success case, and I don't want my users to get test messages or have to set up aliases for receiving and discarding them. swaks will definitely become my tool of choice for ad-hoc testing in future, but meanwhile...

The other promising Google result is Test::MTA::Exim4, which is a Perl wrapper for testing an exim config file. However, a few problems: (1) it's Perl, and I Don't Do Perl. (2), it's limited to testing the routing of addresses, so it's not going to cut it for checking spam rejection etc.

Having at least pretended not to be suffering from NIH syndrome, let's spec out a fantasy system for doing what I want: I would like to be able to write some nice high-level tests in my favourite language, Python, which look a bit like this

class HubbedDomainTests(EximTestCase):
    """
    Tests for domains our server acts as the 'proxy MX' for, doing
    scanning etc before forwarding the mail to the destination machine
    """

    def testProxiedMailAccepted(self):
        """Proxied mail should be accepted"""
        session = self.newSession()
        session.mailFrom('[email protected]').rcptTo('[email protected]').randomData()

    def testLocalPartsVerifiedWithDestinationMachine(self):
        """Local parts should be verified with the destination machine"""
        session = self.newSession()
        session.mailFrom('[email protected]').assertRcptToRejected('[email protected]')

I could then run these in the usual manner for Python unit tests, and lastly, I want them backed by an exim4 -bhc session so that they're as realistic as possible without actually sending messages.

This post is long enough already, so I'll cut to the chase and say that I've made a start on writing it, and you can find out more at Bitbucket. In a follow-up post, I'll talk about how it was done.