vthrottle README

$Id: README,v 1.12 2004/06/02 03:30:55 jose Exp $

/*
 * an implementation of m. williamson's rate throttling mechanism for
 * mail bourne worms and viruses. basically we keep track of what hosts
 * and what senders have been sending mail. if they attempt to send mail
 * from one machine or using one address more than once a minute we
 * back them off by sending a temporary failure. well behaved hosts
 * are expected to not trip this detection/throttle ...
 *
 * references:
 * Williamson, M. HPL-2002-172
 *      HP Labs Tech Report HPL-2002-172
 *      http://www.hpl.hp.com/techreports/2002/HPL-2002-172.html
 * Twycross, J., and Williamson, M.
 *      in Proc. 12th USENIX Security Symposium, pp 285-294
 *      http://www.usenix.org/events/sec03/tech/twycross.html
 */

a big hearty thank you to matt williamson and the HP management team
for being ok with an independent implementation being developed and
released. their technique and implementations are patent pending. this
is just an approximation of their methods.

build the milter using the Makefile. you may have to adjust your paths,
it's set up for OpenBSD's default paths to the milter include files and
their sys/queue.h file (for linked list macros). a copy of sys/queue.h 
is included here for convenience.

you'll have to adjust your sendmail configuration file to use this as
an input filter. in this case this shows a UNIX domain socket:

  INPUT_MAIL_FILTER(`filter1', `S=unix:/var/run/milter/vthrottle.sock, F=R')
  define(`confINPUT_MAIL_FILTERS', `filter1')

regen your config and install it, restart sendmail to use that configuration
file. you can also use an IPv4 or IPv6 socket for the filter, see the
sendmail and milter docs for how to do this. 

once it's built and sendmail is up and running, you can launch the filter:

  $ sudo ./vthrottle -p unix:/var/run/milter/vthrottle.sock 

it should start working on your mail ... it will attempt to send a temporary
failure if any single host or source tries to send more than one mail
a second. this should help you slow down mail bourne viruses and worms
that use your servers. when they do get throttled you'll see messages
like this in your system logs:

    Feb  3 22:50:17 gibbs vthrottle: throttling HELO host localhost: observed 
	interval: 0, expected: 1 

it is not effective against outgoing worms/viruses from your own network
that set up their own SMTP servers ("direct to MX" worms). firewall rules 
which enforce a site's mail server policy (ie egress port 25 filtering) 
can help here. vthrottle may catch inbound worms/viruses, but only after 
significant damage has been done. 

for hosts that need to be able to send mail as often as they want (at
a rate of more than once a minute), you can use the whitelist feature.
a sample whitelist, which also explains the format, is included. if you
would like to have a whitelist automatically generated for you, you can
run "vmeasure" in much the same way you would run vthrottle. after a 
suitable measurement period, you can send it a USR1 signal and the 
new whitelist will be stored in /var/tmp/vmeasure.stats.

vmeasure runs instead of vthrottle, and you can either add a second socket
to your sendmail.cf or you can reuse the vthrottle socket. the latter
is probably easier, so you never have to run them in tandem.

to dump the in-memory statistics, send the vthrottle process a USR1
signal. the statistics are written out in the file /var/tmp/vthrottle.stats .

the mail server's statistics are kept as a doubly linked list and is 
stored in memory. after a while, busy servers will see the performance 
of vthrottle decrease as this list size increases. to help alleviate this,
entry expiration (via the -x argument) can be used to prune stale
list entries. set this value to at least 5-10 times the interval period
for best effect.

LOG MESSAGES

vthrottle will log to the normal syslog mechanism. these are some log
messages you will see and what they mean:

	"(w_helo, w_connect), cannot set reply to 451"
	   vthrottle attempted to set a custom error message for
	   connection and HELO throttling to 451 (as opposed to 5xx)
	   but failed. failures will generate 5xx error codes and mail
	   will bounce.  mail should be bounced with a temporary
	   failure and almost all SMTP clients will try again a short
  	   while later.

	"throttling connecting host %s: observed interval: %d, expected: %d"
	   vthrottle has seen the host connect too frequently and has
	   blocked it from proceeding with the SMTP transaction.

	"removing expired entry for connect host %s, age %d is older than %d"
	   vthrottle found an entry that is older than the expiration
	   time and has pruned it from the list. informational.

	"throttling HELO host %s: observed interval: %d, expected: %d"
	   vthrottle has seen the host say HELO too frequently and has
	   blocked it from proceeding with the SMTP transaction. note
	   that this is based on what the client said as "HELO", which
	   can be almost anything. mail should be bounced with a temporary
	   failure and almost all SMTP clients will try again a short
	   while later.

	"removing expired entry for HELO host %s, age %d is older than %d"
	   vthrottle found an entry that is older than the expiration
	   time and has pruned it from the list. informational.

	"throttling mail address %s: observed interval: %d, expected: %d"
	   vthrottle has seen the destination mail address too frequently
	   and blocked the client from proceeding with the SMTP transaction.

	"removing expired entry for mail %s, age %d is older than %d"
	   vthrottle found an entry that is older than the expiration
	   time and has pruned it from the list. informational.

when an SMTP transaction is throttled it is temporarily blocked from
proceeding. the nature of SMTP means that it will try again in the near
future and that message will just be delayed.

THANKS

ben lindstrom, niels provos, marco hyman for help in debugging a scalability
problem, claus assmann for looking over vthrottle. todd fries has done
some real world testing of vthrottle and found significant bugs, both in
the code and the docs. thanks for the feedback. vmeasure was based on a 
conversation with todd fries, who wanted as statistically significant 
whitelists as he could find.

TODO

implement williamson's application layer throttle described in 
  http://www.hpl.hp.com/techreports/2003/HPL-2003-118.html
this means a deferrment queue ...

run as non-root ... gotta figure out a trusted user to try this as.

handle RSET command

fix bug relating to whitelist insertions into the TAILQ structure

