Why OIT does NOT whitelist domains

There used to be a time when there was no such thing as spam email in the way we know it today.  Back then, referring to an email as spam usually meant that someone you knew had sent you a message that wasn’t of interest to you.  There were no rate limits or blacklists or any of the other tools that have been developed to block unwanted or dangerous email, because the problem didn’t exist.

Eventually people realized that the system could be exploited and thus spammers were born. What was once a manageable amount of email traversing the mail servers became an unending deluge of money scams, email-born viruses, un-targeted advertising and phishing attempts. The volume of email processed by mail servers has risen dramatically every year since then and quickly reached a point where without effective spam countermeasures in place, they simply would not be able to keep up.

Various anti-spam measures have been developed over the years and many have been rendered ineffective by the growing sophistication of the spammers who attempt to counter them.  Modern strategies to fight spam include a two-phase approach, which I call “drop and scan”.  During the first phase, the majority of email is simply refused based on the reputation of the sending host and domain. Reputation is determined over time by data collection services that gather information about any mail servers they see who send email.  The reputation numbers get worse if they see large amounts of spam coming from a mail server and get better for trends of non-spam email from that same server. Based on reputation alone an incoming mail-delivery connection will be dropped immediately, allowed in but at a lower rate, or allowed in at the normal rate.  Reputation scores are constantly evolving and cannot be adjusted locally.

Once email has passed the “drop” phase, it comes into the “scan” phase.  No spam scanner is perfect, but they can do a pretty good job of catching what the reputation blockers cannot.  Modern scanners analyze messages using complex heuristics that also evolve over time.  They too are updated from a central authority trying to improve their effectiveness against the spammers who are also evolving their own messages to trick the scanners into letting the spam through.  Scanning spam is a lot more resource-intensive to the gateway server and so the drop phase becomes even more important as a means of keeping the scanning workload under control.

The question sometimes comes up, “Why didn’t I receive email that I know was sent to me?”  There are a many possible answers to this one.  A common reason is that the mail host the person was attempting to send through was flagged as suspect because someone else recently used the same mail server to send out a high volume of spam.  The subsequent normal message was then refused or throttled for delivery based on the prior actions of the sending host.  When this happens, often the reputation will quickly return to normal once the spam from that host has been stopped.

When a Williams user has been successfully tricked into giving up their username and password, their email account is often immediately used as a source for sending out huge amounts of spam.  This in turn affects our reputation scores with other sites who then begin to deny mail sent from us.  If they use a good reputation service, then our scores will improve shortly after we’ve cleaned up the issue.  However some large sites such as AOL base their policy on the feedback of their users who flag these messages as spam.  The problem with this is that once a site such as Williams is associated with sending out spam and subsequently blocked, it’s very difficult to get that policy reversed to allow mail from our domain once again. It is definitely not a timely self-healing process like reputation scores.

Another question we commonly get is why don’t we whitelist mail from XYZ.  Whitelisting means making an exception to the rules that allows all mail to come in from a site without being checked.  This circumvents our protection and opens us up to any problems on site XYZ.  It is better to let the anti-spam appliance do its job and keep the protection that it offers.