Jump to content United States-English
HP.com Home Products and Services Support and Drivers Solutions How to Buy
» Contact HP

HP.com home

September 2004

Putting spam in the slow lane

Experimental system separates out and delays suspected spam to keep e-mail flowing freely


» 

HP Labs

» Research
» News and events
» Technical reports
» About HP Labs
» Careers @ HP Labs
» People
» Worldwide sites
» Downloads
Content starts here
What this does is put the junk email into a slow lane before it reaches the anti-spam filters, allowing the good mail to go into the filtering process without delay.

by Julian Richards

If you have ever waited, frustrated, for an important e-mail message while endless spam and junk pour into your inbox, then HP Labs researchers could have the answer. It is an experimental system that can reduce delays to legitimate e-mail delays caused by an ever-mounting volume of spam and junk messages by giving good mail priority over bad.

A prototype of the system is now running in a live trial on one of the mail servers at HP Laboratories in Bristol, U.K., where it was developed.

Good mail gets through

The system is deployed upstream of conventional mail-server spam filters. It rapidly separates out messages from suspicious IP addresses to reduce backlogs during spam surges and then places those messages in a different queue.

The system could be used to complement current spam-filtering technology, speeding the delivery of legitimate e-mail messages by sending good e-mail through the filters and into the network first. Suspected spam would go into a lower-priority queue.

“It puts junk e-mail into a slow lane before it reaches the anti-spam filters, allowing the good mail to go into the filtering process without delay,” says Miranda Mowbray, one of the researchers who developed the technology.

Bottlenecks a problem

As anyone with an e-mail address knows, the quantity of junk mail is soaring. Some 64 percent of all e-mail is spam -- and this figure is expected to rise, according to Brightmail, a maker of anti-spam tools. The increasing number of spam surges and virus attacks tie up mail servers and can delay e-mail delivery for hours, including messages that could be of critical business importance.

Part of the problem is that, upon arrival at the e-mail gateway, incoming messages are put in a queue so that the content can be scanned for viruses and spam before it is passed on to the recipient.

During spam surges,the scanning process can create a performance bottleneck in the system and legitimate mail is delayed waiting for junk mail to be filtered out.

The HP Labs team has devised an approach that gives delivery priority to mail it categorises as good over mail it tags as junk.

Separating wheat from chaff

The HP Labs prioritization technology classifies a server as good if fewer than half the e-mails received from that IP address were junk – spam, virus or undeliverable. The classification is based on statistics from SpamAssassin, an anti-spam application, Sophos, the anti-virus scanner, plus the results of the first delivery attempts.

This classification can be used to speed up the delivery of non-junk e-mail by creating two queues. The queue of e-mail from good servers is given priority access to the existing content scanner. This ensures that there are virtually no delays to e-mail in this queue, even if the server is heavily loaded with junk mail.

During a recent virus attack, delays of 2.7 hours were experienced to all mail processed by a large commercial mail server. If the prioritization technology had been in operation, the delay would have been only 22 seconds for the good queue, but four hours for the junk queue. The results come from performance data from real mail that was passed through the experimental system.

Successful trial

At the moment the system predicts good messages with 74 percent accuracy and junk messages with 95 percent accuracy. This is sufficient to select suspect messages to be delayed in favour of legitimate e-mails before they are scanned for content.

“The power of this classification method is speed. We know almost immediately if a message should be tagged as good mail or junk,” says researcher Dan Twining. “This immediacy, along with the system’s relative accuracy and its lightweight implementation, allows us to optimize all other anti-spam techniques.”

Researchers ran a three-month live trial of the technology using the sendmail mail transfer agent in a large-scale e-mail system. The team has now developed and is preparing to trial a Postfix-based version of its prioritization technology. The Postfix mail transfer agent is more commonly used by large companies and enterprises.

In addition, the team -- Mowbray, Twining, Maher Rahmouni and Matt Williamson – published an academic paper (Email Prioritization: Reducing Delays on Legitimate Mail Caused by Junk Mail), which was presented at the Usenix conference in Boston in June 2004. The paper is available as an HP Labs Technical Report .

 

Related links

» E-mail prioritization technical report

News and events

» Recent news stories
» Archived news stories

Researchers from left to right, Miranda Mowbray, Dan
Twining and Maher Rahmouni

Researchers from left to right, Miranda Mowbray, Dan Twining and Maher Rahmouni











































































     



Printable version
Privacy statement Using this site means you accept its terms Feedback to HP Labs
© 2009 Hewlett-Packard Development Company, L.P.