by Julian Richards
If you have ever waited, frustrated, for an important e-mail message while
endless spam and junk pour into your inbox, then HP Labs researchers could have
the answer. It is an experimental system that can reduce delays to legitimate
e-mail – delays caused by an ever-mounting volume of spam and junk messages – by
giving good mail priority over bad.
A prototype of the system is now running in a live trial on one of the mail
servers at HP Laboratories in Bristol, U.K., where it was developed.
The system is deployed upstream of conventional mail-server
spam filters. It rapidly separates out messages from suspicious
IP addresses to reduce backlogs during spam surges and then places
those messages in a different queue.
The system could be used to complement current spam-filtering
technology, speeding the delivery of legitimate e-mail messages
by sending good e-mail through the filters and into the network
first. Suspected spam would go into
a lower-priority queue.
“It puts junk e-mail into a slow lane before it reaches the anti-spam filters,
allowing the good mail to go into the filtering process without delay,” says
Miranda Mowbray, one of the researchers who developed the technology.
As anyone with an e-mail address knows, the quantity of junk mail
is soaring. Some 64 percent of all e-mail is spam -- and this figure
is expected to rise, according to Brightmail, a maker
of anti-spam tools. The increasing number of spam
surges and virus attacks tie up mail servers and can delay e-mail
delivery for hours, including messages that could be of critical
business importance.
Part of the problem is that, upon arrival at the e-mail gateway,
incoming messages are put in a queue so that the content can be
scanned for viruses and spam before it is passed on to the recipient.
During spam surges,the scanning process can create a performance
bottleneck in the system and legitimate mail is delayed waiting
for junk mail to be filtered out.
The HP Labs team has devised an approach that gives delivery priority
to mail it categorises as good over mail it tags as junk.
The HP Labs prioritization technology classifies a server as good if
fewer than half the e-mails received from that IP address were junk – spam,
virus or undeliverable. The classification is based on statistics from
SpamAssassin, an anti-spam application, Sophos, the anti-virus scanner,
plus the results of the first delivery attempts.
This classification can be used to speed up the delivery of non-junk
e-mail by creating two queues. The queue of e-mail from good servers
is given priority access to the existing content scanner. This ensures
that there are virtually no delays to e-mail in this queue, even if the
server is heavily loaded with junk mail.
During a recent virus attack, delays of 2.7 hours were experienced to
all mail processed by a large commercial mail server. If the prioritization
technology had been in operation, the delay would have been only 22 seconds
for the good queue, but four hours for the junk queue. The results come
from performance data from real mail that was passed through the experimental
system.
At the moment the system predicts good messages with 74 percent accuracy
and junk messages with 95 percent accuracy. This is sufficient to select suspect
messages to be delayed in favour of legitimate e-mails before they are scanned
for content.
“The power of this classification method is speed. We know almost immediately
if a message should be tagged as good mail or junk,” says researcher Dan
Twining. “This immediacy, along with the system’s relative accuracy
and its lightweight implementation, allows us to optimize all other anti-spam
techniques.”
Researchers ran a three-month live trial of the technology using the sendmail
mail transfer agent in a large-scale e-mail system. The team has now developed
and is preparing to trial a Postfix-based version of its prioritization technology.
The Postfix mail transfer agent is more commonly used by large companies and
enterprises.
In addition, the team -- Mowbray, Twining, Maher Rahmouni and Matt Williamson – published
an academic paper (Email Prioritization: Reducing Delays on Legitimate
Mail Caused by Junk Mail), which was presented at the Usenix conference
in Boston in June 2004.
The paper is available as an HP
Labs Technical Report
.
|
Printable version
