Autresponse Standards

E-mail autoresponse is a highly useful tool. You mail an address and you get back an automatic response. This is used for customer support, and just about every mailing list today mails you back to get a confirmation that you really wanted to subscribe before putting you on the list. (This is known as confirmed opt-in, or sometimes double opt-in.) The anti-spam community essentially demands this autoresponse behaviour.

In addition, web sites which take E-mail addresses (either using web e-mail as an anti-spam technique, or those that add people to mailing lists or those that use an e-mail address as an userid or to confirm a new account) all mail that address to confirm the owner of the address really wants it used in this fashion.

Challenge/response spam filtering, which allows senders to confirm they really sent an email when the email would otherwise be blocked by a spam filter, also depends on autoresponse. A set of best practices for challenge/response spam filters details special rules for these autoresponders.

Vacation programs, which inform senders that mail may not be attended to quickly, are a popular form of autoresponse.

Finally, the detection of errors in the mail system itself depends on autoresponse. If errors can be spotted immediately on a peer to peer connection from the sender's mail server directly to the recipient's, the errors can be sent via the protocol, but if E-mail is forwarded, or the "MX" system is used to route the mail through a clearing or backup mail server, errors and nondelivery reports can only be sent via e-mail autoresponse. In a robust system, errors must be reported to somebody who will care -- it is not acceptable software engineering to just drop mail on the floor.

Unfortunately spammers are indirectly creating trouble with autoresponse. Spammers and viruses often send messages with a faked "From" address or envelope. This means that the autoresponse can go back to the randomly chosen person whose address has been forged. The autoresponder doesn't intend this of course, and is the spammer's first victim, but it is also annoying to the spammer's 2nd victim.

In addition, autoresponders can be used to deliberately send mail to the party whose address is forged on the envelope. In certain cases, it may be possible to have more than one autoresponse sent for a single delivery attempt, thus multiplying the effect. This multiplication attack has not yet been used in the wild, but the "joejob" attack to flood a user, while not particularly powerful, has been known to occur. Should this become an actual DDOS attack in the wild, it would be a matter of concern.

Because of these real and hypothetical threats, some parties have gone so far as to blacklist sites for running autoresponders, even though such autoresponders are an important and standard part of the mail system -- in fact RFC 2821 says that mail agents MUST send an autoresponse for any failure detected after acceptance of a message. Such blacklisting is controversial. It causes sites to reject all mail from the site with the autoresponder, not just the autoresponses.

Since autoresponders are not going away, it makes sense to examine some practices which may help with these problems.

Some issues were addressed in RFC3834.

Standardize autoresponses

There are some standards in place for certain types of autoresponses. A uniform standard should be declared and adhered to.

Identification

There should be one simple universal identification that an E-mail is machine generated and is an autoresponse. RFC3834 introduces the use of the Auto-Submitted field. It may make sense to extend this to indicate the type of autoresponder, with classes such as: error, warning, challenge, comment etc.

In-reply-to:

all autoresponses MUST include an In-Reply-To header to indicate the message-ID of the message they are autoresponding to. (As per RFC3834 and RFC2822.)

The presence of this header can make it easy for sites to determine if the response is in fact due to a message that went out from the site. Other responses can be considered as likely due to forgeries, or at least be given a poorer spam score.

Received:

Autoresponders SHOULD include in the response a copy of the "Received" headers from the original message. This allows the determination of the actual IP that sent the message being replied to. Normally that should be a server used by the sender. This can be subject to automatic scan, or simply for forensic work in tracking down spammers or virus infected machines.

Priority

Autoresponders SHOULD pass through the priority of the original message. Urgent messages may require urgent treatment of an autoresponse, for example, particularly with spam challenges or non-delivery reports.

Single Autoresponse

Autoresponders SHOULD assure that on a message with multiple recipients, only one response is sent. This stops autoresponders from multiplying their actions on the initial autoresponse.

Request to disable autoresponses

Any autoresponse which might engender an additional autoresponse MUST include a means to disable the further responses. A typical example is a "Temporary failure" message, which is followed by additional temporary failure updates and typically a final permanent failure update.

For this, the autoresponse would include a special address to which one could mail a message which references the message-id of the original message. This message would say, "Send no more autoresponses" or possibly, "Send no more autoresponses except permanent failure." (See also autoresponse preferences.)

Autoresponse Preferences

Outgoing mails MAY contain a header which would indicate their preferences on autoresponses. They may indicate that they wish to see the usual responses. They may indicate this wish to see no responses, or only a single permanent failure response. They may indicate they do not wish to receive spam challenges or other classes of autoresponses.

They may also indicate they would like a verbose autoresponse, which may be useful for debugging. In order to assure that is not misused, such a request might only apply to verified, signed E-mails.

It may also be possible to use a DNS based system to record, on a per-domain basis, what level of autoresponse should be applied to messages claiming to originate from that domain.

Note that proper software engineering principles require that the request for no autoresponses NOT be the default. Users who do nothing must get error reports. Unfortunately this means they also get autoresponses to forgeries.

Spammers could put in "please autorespond" on their forgeries, but they have no particular motive to. It gains them nothing and creates more liability for them. Mailbombers attempting to DOS attack using autoresponders would of course apply such a header. Fortunately this has not yet taken place.

For message delivery notifications, the NOTIFY tag on SMTP RCPT commands allows the specification of what sort of failure and success notices should be delivered, including asking they never be sent. However, this is not standardly available to programs further down the chain. The nonstandard Errors-to header provides some facility here.

E-mail signature

We may have to consider autoresponding only to signed mail. This means that anonymous mailers, and those not signing their mail, would lose the ability to bypass spam filters, determine if their mail was not delivered and various other functions. As such it's a very drastic step.

As such it is recommended it be combined with a DNS based preference system, so that one can say, "For this domain, autorespond only if messages are signed or otherwise authenticated as being from this domain."

Autorespond minimally

The volume of autoresponses should be reduced. In particular, challenge/response spam filters should do extensive spam scoring first, and only challenge mail whose spam status is uncertain, and which is from unknown parties. This minimizes the number of challenges in general, possibly eliminating any problems with the issue. At the same time, parties can assure their mail will be delivered.

A global mail response system

For those who feel the problem has become particularly dire, it may make sense to develop a global system for mail autoresponses. In such a system, rather than responding with E-mail, a new protocol would be developed. Again, DNS records could indicate that autoresponses of various classes should go to the new protocol.

For example, the response may simply be appended to a database record for the user in question. Then responses can be batched together into a single mail, such as, "Here are all the mails you sent today that failed delivery after the first server." Coming once a day, such messages would never reach a level of annoyance.

This could be imagined as something akin to "MX." The alternate MX would be used only for autoresponses, or might even dictate another protocol.

It's also possible, within the confines of the existing system, to simply gather all identified autoresponses for a user and batch them rather than deliver them all at once.

If the priority of the original message is passed through, the batcher can use this to determine when to deliver any autoreponses. The class of autoresponse can also help. Vacation program messages will have low priority. Spam challenges may have higher priority. Temporary non-delivery warnings will tend to have low priority.

A global opt-out for autoresponses

In the event that the problem of autoresponses from conforming autoresponders overwhelming mail servers gets serious, a global opt-out database of those who wish to not receive autoresponses could be considered. Implementation of such a system is non-trivial, however, and might require some level of centralization and a mechanism to update the addresses on the opt-out list.

Challenge/response filters could decide that since they will not send a challenge to such users, that their messages of unknown state may be put in a special queue of "mail from those who can't be responded to" for examination by the user. However, spammers would put their addresses on this list, but they would have to be real addresses, which spammers try to avoid using.

Since a well-formed autoresponse will contain headers identifying it as an autoresponse, it is easy for mail software to discard or not display autoresponses for those who do not wish to see them, or certain classes of them. This is probably the most effectively implemented opt-out mechanism as it requires no centralization or network queries. It does mean a modest amount of extra network traffic if autoresponders are properly limiting the volume of their responses.

Techniques for spam filters & blacklists

Spam filters wish to block autoresponses that were generated by mail with a forged address.

Spot your responses

Ideally, spot responses to your own mail by looking at the Message-ID found in the References or In-Reply-to header of the autoreponse.

This can be done by keeping a database of recent outgoing messages, or by generating Message-IDs that all follow a non-public rule that can be tested. This may not be possible if you don't control your Message-IDs.

As a secondary test, one can, as many spam filters do, track all recipients of outgoing mail and subject lines of outgoing messages to whitelist those items so that replies, including valid autoresponses, are not spam-filtered.

Narrowly blacklist and filter

Blacklisting is controversial and is far from endorsed by the author of this document. However, those who do decide to blacklist should blacklist as narrowly as possible to block the least possible real mail.

As such, if a site is added to a blacklist for sending large volumes of autoresponses to forged addresses, it should be added to a special blacklist so that only autoresponses, and not regular mail, are blocked from that site.

Correctly formed autoresponses (featuring the Auto-Submitted header) can be easily detected and treated as unwanted if this is the desire of a destination site. If a site runs an autoresponder that does not provide headers that allow its autoresponses to be identified as such, behaviour of the spam filter is beyond the scope of this recommendation.

In general, blacklisting is appropriate only for mail that can't be readily classified by a recipient site or which comes in a volume that causes a significant load problem. Blacklisting should not be used on properly formed autresponses which use an Auto-Submitted header and other appropriate headers which come in moderate volumes.

In addition, blacklisters should work to assure their blacklists can't be used as a DOS mechanism. Since a considerable number of web sites and mailing lists will send an E-mail to an uncheckable e-mail address (for example to confirm if it is real), the automatic blacklisting of sites which mail spamtrap addresses presents an easy method to DOS such sites that attempt to confirm addresses. Even if the blacklisting follows the rule above and only filters autoreponses, it could make a mailing list unable to add subscribers since it can't confirm the requests of those subscribers.