Is Challenge/Response filtering a good or bad thing?

A small number of people who encounter challenge/response (C/R) spam filtering systems, including my own, disapprove of them as a concept. Some of their complaints have merit, so I felt it worthwhile to explore these issues regarding the tools, since I've been running them longer than anybody.

The fight against Spam, as I see it, is a battle to stop the spam flood while doing the least "collateral damage" to legitimate mail and the principles we want from our mail system.

One of the greatest collateral damages is the "false positive" -- when a spam-blocker stops a piece of legitimate mail from reaching its target, particularly when it simply drops the mail on the floor with no diagnostic to an interested party. If we have false positives, we abandon one of the most fundamental features we want from a mail system -- reliable delivery.

It is for this reason -- the potential for reducing false positives, that I continue to use C/R as a spam-blocking system.

Important Update: Autoresponders being blacklisted

Even though there's no particular evidence of a large problem with C/R systems, some spam blacklists including spamcop.net at spews, reportedly blacklist operators of autoresponders, including C/R and with a particular focus on it. It is a concern when anti-spammers decide to fight other anti-spammers instead of fighting spammers, and also when great collateral damage is the result.

Buggy C/R

One of the main reasons for complaints about C/R is actually due to buggy C/R. No denying that some recent implementers of C/R have done a poor job, in particular doing things like issuing challenges to mailing list mail, which can easily swamp a mailing list owner.

That's a shame because e-mail autoresponders are a well established software genre, going back to the "Vacation" programs of the 70s. Any C/R system needs to at least follow the principles of a proper autoresponder, along with many others. To help identify and stop these bugs, I wrote a white paper on best practices for challenge/response anti-spam systems and an associated general principles for all autoresponders.

Buggy C/R that challenges replies, well-formed, mailing list mail other challenges etc. should be strongly discouraged. However, the flaws of buggy C/R are not apropos to the question of whether well-behaved C/R is good or bad.

Burden on senders

Many complain, "how dare you make the sender to the work of managing your spam filtration?" It is sometimes decried as "selfish" for this reason. It's true that the extra burden of a challenge is a mild form of collateral damage. However, I remain unconvinced that this burden is something special, or even inappropriate. For a century, strangers calling busy people on the telephone have had to face a screening process with a secretary, answering machine or IVR system.

It's also the case that on mail from a stranger, there's no set pattern as to who the mail benefits. Often a stranger mails me seeking something from me, information or advice. Other times a stranger mails me to offer me something I want. More often, the relationship could be viewed as an even exchange. I see nothing that says that there is more or less merit to putting the screening burden on one party or another. If the stranger seeks my help, it certainly seems more fair to put the burden on the sender.

Of course, the burden should be as small as possible. (See the best practices document.) Proper C/R only sends challenges on the first mail from an unknown party, when that mail would otherwise have been blocked by an anti-spam system unable to be sure if it is or isn't spam, which is a relatively tiny fraction of total legitimate mail. As such, C/R challenges should be something that occur rarely, an average of well under 1/day and thus not a great burden in any event.

Forged 'From' addresses

A legitimate complaint against all mail autoresponders -- and this even includes all the mail servers of the world which will autorespond with a 'bounce' in case of error -- is that they will respond to a forged From address and thus send mail to an innocent 3rd party.

This is most commonly happening today with mail worms and viruses, which have taken to sending mail to one person pretending to be "From" another semi-random person. Less commonly, there are spammers who also put in such fake addresses. When the mail bounces or hits an autoresponder like a C/R system, the innocent party gets some extra mail.

As the level of faked addressing continues to grow, this will need to be seriously addressed. One option is to consider not autoresponding to mail that contains an attachment like a windows binary. Anti-virus software, which actually knows for sure the mail is part of an address forging mail-worm, has no excuse for autoresponding. Autoresponders now must consider integrating together with spam and virus-detection software, so that they only respond to mail which is in the "unknown" (spam or not) category.

It's also true that autoresponses to spam or worms with fake addresses adds an extra burden on mail servers, by sending out a small message for each such message in. In the case of spam, many of the addresses are simply invalid altogether, so the challenge is never sent.

One common suggestion is to improve E-mail authentication to feel better about sending autoresponses when it can be confirmed the mail came from the address it says it came from. This can be useful, though we must avoid the risk of building an E-mail system that only allows people who use ID to send mail.

Some worry that blackhats might use autoresponders to mailbomb a person, though this is not particularly efficient since you need to send one message out for every message reflected at the target -- you could just hit the target directly. At present I do not believe such an attack has taken place so it's not something we would want to take drastic measures to stop in the hypothetical future.

Mailing list whitelisting

All spam filters need to be able to whitelist the mailing lists a user has legitimately subscribed to. This is an issue for all spam filters, but the only special issue for C/R filters is that since they must not challenge mailing list mail, they need to take special care to assist the user in whitelisting those lists.

Privacy issues

A 3rd party C/R system ends up keeping a database of a user's contacts on an outside system. This is to be avoided, or the risks of it made plain. Systems that run on your own server are safer. There is a trade-off. Running on the client provides the best privacy protection, but if the client is not full-time connected, can result in delays as indicated below.

Speed Issues

C/R can slow down mail. Particularly if the challenge is not sent immediately. If it is, risk of a long slowdown is slow -- the user would need to send the mail that gets challenged and immediately disconnect without checking mail again. As noted, this can happen but it's rare enough to not outweigh the benefits.

Lowest false positive rate

The big reason C/R is a good system -- in particular when combined, as it should be with other spam-scoring and virus detecting algorithms -- is that it has such a low false positive rate, perhaps the best that exists.

An ideal C/R-enabled spam filter system first attempts to score the incoming mail. If it is certain it is spam, it can be discarded without challenge. If it is very likely not spam, it should also be directly forwarded without challenge.

Thus, challenges only will occur on a fraction of new mails from unknown correspondents, where the spam score is inconclusive.

Systems without C/R, faced with an inconclusive score, take several poorer routes. Some forward through the mail, causing amounts of spam to reach the mailbox. Some discard the mail, causing serious quantities of real mail to be lost. Some put the mail in a "maybe spam" filter which is scanned less frequently, resulting in possible long delays or loss of the mail. Some bounce the mail, informing the sender but forcing a re-send -- in effect a hard to manage challenge.

C/R systems offer the option of a challenge, providing the sender the chance to arrange immediate delivery of the mail to the target's main mailbox. The worst case, of not responding to the challenge, moves the mail to a delayed "maybe spam" mailbox where it faces the same fate such boxes have in a system without C/R. In practice, C/R provides that fast delivery most of the time.

In my experience, C/R has both the best spam blocking rate and lowest false positive rate of any system, and combined with the best spam scoring systems, it provides performance levels that are very difficult to beat, worthy even in spite of the drawbacks that exist.

While there are those who react to C/R by saying they find the challenges annoying or selfish, I believe the question should be posed somewhat differently:

If you have just sent an E-mail to a stranger, and for whatever reason the target's spam filter has decided your mail is possibly, but not certainly spam, which of these would you prefer?

1. The mail be discarded.
2. The mail be put in an infrequently-checked "maybe-spam" mailbox
3. The sender be send a challenge which gives them the option of confirming the mail for immediate delivery to the real mailbox, or doing nothing and having it stay in the maybe-spam mailbox.

Phrased this way, I don't think very may would judge the C/R system as annoying or selfish. The selfish act may be having a spam filter system that wasn't good enough to identify the mail as clearly not-spam, but given that those are going to exist, which choice would users prefer in these rare cases? Loss of e-mail, or a challenge?

Whitelisting has incompatability with other systems

There are some anti-spam systems that try to use a different email address for every email you send out. This lets them track replies, shut off addresses that are getting spammed, and also "age" addresses so that recent addresses get less filtering and older ones get more.

These systems have a problem with all whitelisting tools, including C/R filters. (For example, you can't really participate in a mailing list that insists all mail come from a subscriber's address.)

My judgement is these address-changing tools need to establish standards to point to the real address, so that whitelisters can identify them. A simple one they could use would be to use the "Sender:" field or create a new field. They could also establish a public key to sign messages so that nobody could forge it, and use the same key to sign all messages. Only after this standard was established could whitelisting tools continue to whitelist such systems. Whitelisting has become very common (not just in C/R systems) so I think the multi-address systems have the burden of dealing with the question.

Anonymous mail

Truly anonymous mail is a hard question for all anti-spam systems. You can't provide a challenge on it. Alternate routes, such as CPU-stamps are probably the only route. Pseudonymous mail is fine with C/R systems.