SPAM: Change is coming

Why is change needed?

When FTGate Technology started supplying mail servers, over ten years ago, there was no such thing as spam. When you received a message you knew that it was most likely to be a genuine message that you should take time to read. The world was a nice place where everyone was trusted to only send you messages if they thought you wanted to get them. eMail was cheap, quick and efficient. The Internet was designed with this in mind, protocols were open, easy to implement and had no security at all.

Then things began to change. The low cost of sending an email, essentially nil, made it very cost effective to send millions of emails with a marketing message. At first no one really took any notice, one odd email of spam was not a problem. But it didn’t stop there, it grew.

Now the problem has escalated to the point where there is more spam on the Internet than real mail, and the open protocols, that assumed trust, offer no means to protect ourselves from the deluge.

The problem is exacerbated by viruses. Many of these viruses are sent from machines whose owners do not know they are infected. They use random from addresses and often random to addresses and can come from anywhere on the Internet. They don’t require a mail client or mail server to run.

Organised crime has also joined the game. They use virus infected machines called zombies to source spam to millions of addresses from machines whose users are unaware that this is happening. They use the machines to probe for addresses, phish for bank account details and launch denial of service attacks on companies.

As the problem has grown FTGate Technology have successfully introduced more and more features with which to fight spam; word filters, phrase filters, UbeBlock,  blacklists, RBL lists and so on. These are all very effective methods of blocking spam which work on trying to identify which messages contain material that we would rather not receive or identifying sources of messages which are known to be bad.

However, the spammers are an ingenious bunch and at every stage they have found a means to obscure their message (html, word soup,etc), hide their IP addresses (zombies, open relays, etc), and this has produced an arms race in trying to identify the messages as being spam. We improve detection, they hide the message more skillfully, and it goes on, and on.

A shift in approach

There is a complete shift in approach going on, and FTGate Technology are part of this being the first Mail server company to officially sign the SPF Community Position pledge.

The shift is from a world where we try to identify spam to one where we identify legitimate messages, and assume everything else is junk.

There are several approaches to this, some of which are already used:

  • White lists (current)
    Used to identify addresses that we know are good and always want to receive.
     

  • Safe words (current)
    Words that have special meaning, such as product names, that are unlikely to be part of a junk email
     

  • SPF (new in FTGate4 and being deployed throughout the Internet in 2004/2005) SPF
    This seeks to verify that a machine sending a message is authorised to send mail for that domain.
     

  • Encryption/Signing (being deployed throughout the Internet 2005/2006)
    This seeks to verify that the sender is who they say they are.
     

  • Inverse Spam detection.
    Determine that a message is good rather than it is bad.

A combination of these features can result in a world where spam, viruses and other junk are eliminated completely.

Cleaning up the junk

Once we decide to reverse the problem, assume that most of the mail is junk, and try to find the good stuff we can make some big improvements in the way the mail is handled.

SPF

At the top level we can have our mail servers check that there are valid SPF records for the senders of email, this allows us to reject mail which the sending domain owner says should not be sent, and prevent your domain being used to send mail which is not from you. It works like this:

  1. A spammer connects to your server from address a.b.c.d and sends a junk email to you pretending to be from richard@ftgate.com

  2. The server calls the ftgate.com DNS server and asks “Is a.b.c.d a valid sender for domain ftgate.com”

  3. The ftgate.com DNS says no

  4. The spam message is rejected

or

  1. FTGate Technology send a message from 195.224.16.245 to your server and says it is from richard@ftgate.com

  2. The server calls the ftgate.com DNS server and asks “Is 195.224.16.245 a valid sender for ftgate.com”

  3. The FTGate server says yes

  4. The message is accepted

  5. The message bypasses filtering as it is known to be from a good address

or

  1. A customer sends a message to your server from address a.b.c.d saying its from bob@a.com

  2. The server calls the a.com DSN server and says “Is a.b.c.d a valid sender for a.com”

  3. The server says “I dont know” (either they do not support SPF or they do not know if the address is good for them)

  4. The message is accepted

  5. The message is passed to the remaining filters for analysis

This shows that as SPF is rolled out through the Internet community the level of trust for incoming messages will rise. Zombie machines, and open relays will be blocked immediately, while spammers will be forced to use traceable domains and addresses which can then be blocked using the RBL systems or blacklists currently in place.

White lists

After the message arrives we can decide if we will filter it or not. A white list tells the server that we trust this address. The server can then deliver the message directly to the users.

The problem for an administrator is that they must maintain a white list which for large numbers of users can be very time consuming. FTGate4 has addressed this by allowing the administrator to include the entire server contact address book in the white list, thus allowing users to add their own white list entries through either WebMail or via SolSight.

UbeBlock spam analysis

The latest version of UbeBlock adds the ability to add a weight to unknown words. This makes training of the system very simple. Rather than trying to find every possible example of spam and train the system to identify it, we simply train it with a good sample of valid messages, which we all have in abundance. From that point on a message that contains words that are not in our normal emails will have a higher rating applied to it. Couple this with rating for HTML content and its overall rating and you can practically eliminate junk mail from your system.

Moving Forward

These features are effective, however, there is a down-side. If you will only accept messages from addresses that are SPF validated or white listed users, you can expect other administrators to do the same. This means that you will be expected to authenticate your mail clients and vouch that their IP addresses are valid. This is not hard to do.

  1. If you have your own domain name, you should publish an SPF record, or have your hosting company do it for you.
    If you send directly to the Internet you should list your server addresses
    If you use an ISP or hosting company you should send through their servers and list their server addresses
     

  2. Have all your mail clients authenticate with SMTP and force them to send using the authenticated address.
    Do not let them authenticate as bob@a.com and send as fred@b.com unless you are sure that they have the right to do this, in which case, they should really authenticate as fred@b.com anyway.
    (In the security policy set the SA and AR flag, clear the AA flag)
     

  3. Have all your mail clients send ONLY through your server. This will prevent anyone spoofing your domain as SPF will then block all spoofed mail.
     

  4. If you forward mail, you must change the envelope sender address to a local address, otherwise you will fail the SPF checking because your server will not be valid for the original domain. FTGate has done this for some time .
     

  5. If you implement MX forwarding (FTGate remote domains) you should ensure that the receiving server WILL NOT perform SPF checking on the MX relay machines, as this would definitely fail SPF checking. (In the appropriate security policy, add the MX machines IP range and clear the SPF flag).

SmartPop

SmartPop is the poor relation when it comes to anti-spam handling. Because all the mail has already been accepted by your ISP and the IP address information is most likely lost or obscured it becomes much harder to validate that the message is good. For this reason SmartPop does not have any SPF facilities. However, if your ISP implements SPF filtering and adds the required SPF header to the message, the main filters can be bypassed as if the message had been received and validated directly by FTGate.

The future

Over the course of the next few years a variety of techniques designed to limit junk and authenticate users will be tested by the Internet community. They vary from Yahoo’s DomainKeys, Microsoft’s PRA, IIM and others.

As the technology stabilises we will continue to integrate their requirements into our systems. You can be sure that, as usual, FTGate mail servers will deliver your mail reliably and limit the junk you see.