Greylisting

One of the new features in FTGate is the option of SMTP Greylisting. Greylisting is a way of filtering out the large majority of spam and virus sources on the net. In order to understand how it works we need to look at how mail is normally passed around the internet and how we can use this mechanism to help us filter the mail.

Mail is passed between machines on the net using SMTP, and the SMTP protocol is designed to be resilient to failures of both the net and of individual servers. A mail server that complies to the SMTP RFC’s will try to send a message to a destination a number of times before giving up and returning a failure. The exception to this is if the destination rejects the message, in which case the sender will bounce it immediately. Part of the protocol allows a destination to tell a sender that it is currently busy and the sender should try again later, when this occurs the sender should hold the message in its queue and try again after a time delay. This allows the destination to delay mail when its load it too high or there is a server problem.

How does this help us to filter out the spam and viruses?

The main source of spam and viruses are zombie relays that are not true SMTP mail servers, they are designed to try an address and then move onto the next address. If an address fails then they don’t retry. Thus if we reject the connection with a busy signal, they will never retry and we will never get the spam or virus message.

So how does it work?

When a server tries to send a message to FTGate it makes a note of the senders IP, senders address and the recipient address. If it has never seen these three before, it rejects the connection with a busy message telling the sender to try again later. When the sender retries it will accept the message.

There are a few modifications to this simple approach to make sure that the sender is really a true mail relay and not just a slightly smarter zombie. A slightly smarter zombie might retry the same connection immediately after a failure, in which case it would get through, so we add a little dead time to the Greylist entry so that any retry within the dead period is also rejected. This means that if a spam or virus zombie author wants to send the mail to us they have to make their software quite a bit more sophisticated.

The other problem that we face is what to do with the thousands of bad connections that we reject. In one 4 day period the FTGate main server rejected 3500 connections with Greylisting. These were connections that never retried. Obviously if we never clean out these connections from the Greylist it will get very large indeed. So FTGate has two timeout periods for Greylist entries. The first is for those connections that never try again, this is a short timeout period that drops the zombies from the list. The second is a longer timeout for connections that did try again and have been passed. This ensures that the list doesn’t grow too large and that good connections are saved to prevent delays.

Any greylist system will require the ability to add whitelist entries. FTGate allows you to add both whitelist and blacklist entries by IP, sender, recipient and HELO address of the sending server. In addition each filter policy has the option of bypassing the greylist if the sender has been SPF Validated.

What are the disadvantages of Greylisting?

As with any system that is so good at wiping out spam there is going to be some problems.

1. The sender doesn’t retry!
   There are some mail servers that are simply badly written. They don’t care about internet RFC’s and assume that any send error is a complete failure and simply bounce the message. These servers are broken must be explicitly whitelisted.

2. The sender retry is longer than the zombie timeout!
   While not broken, these servers don’t put much value on trying to send the message through. The default for the zombie period is 24hrs and any real server that doesn’t retry a message within 24hrs is very poorly configured. Again, these servers must be explicitly whitelisted.

3. The sender is part of a server farm!
   Some servers try to send a message and then, when it fails, punt it to another server, which will then retry later. Obviously, if the time taken to go round all the punt servers is longer than the zombie timeout, the message will never be accepted. Thus the timeout must be chosen appropriately or the sender must be whitelisted.

4. I have MX relays!
   If you have MX relays then they should all implement greylisting. While most zombies will not retry after a given time a large number are designed to automatically try the MX backup for a domain in the event of a failure. Thus if you do not greylist on the relays the spam will simply flow in through them instead. In addition, you don’t want mail that comes in through a relay to be delayed so you must also whitelist your own relays.

5. My mail will be delayed!
   The first time someone sends to you there will be a delay, and the period is under control of their mail system administrator. Thus if their administrator sets their mail system to retry every 8 hours there will be an 8 hour delay. Subsequent ends will be instant. This is the price of cutting the spam and viruses, the rate at which you get the first email depends on how important the senders administrator thinks their mail is. However, in real terms it is irrelevant for most of the time. If you have a contact that you know will send you messages, such a supplier, you can whitelist them. If it is another source then immediate response is probably not an issue as many things can occur to delay a message, even with no whitelisting, and most users are aware of this. If you really must accept all mail immediately, then you would just turn off greylisting and except that you will get more spam.

Greylisting is a powerful tool new tool in fighting spam and viruses but it also has some potential issues that should be considered before using it on your system. You should review the above points and decide if their impact is acceptable to your needs before implementing greylisting on your server.

For the original Greylisting whitepaper please see this whitepaper by Evan Harris:

http://projects.puremagic.com/greylisting/whitepaper.html

Additional information is available here:  http://www.greylisting.org/