Re: What I did in April

[Christian Gut <cycloon_(_at_)_is-root_(_dot_)_org>]

WOW! Great work henning, especially the dhcp suite auditing and
rewriting is something typical for OpenBSD. Just a bit ahead.. ;)

Thanks.

On Tue, 04 May 2004, Henning Brauer wrote:

> I've been flying to Canada early in April, to help with the release and 
> to attend the cansecwest conference and the pf hackathon at Ryan's in 
> Sechelt, April 23 - 28.
> 
> There was quite some time in between, and I could work on various 
> things.
> 
> Input queue congestion
> ======================
> 
> On the day I left germany, one of our main firewalls was attacked using 
> a dDoS attack. The machine was completely unresponsive and I have been 
> unable to log in and fix things, so it took a bit to solve the issue.
> Analysis showed that all the packets to randon ports on one IP caused 
> that many pf ruleset evaluations that the CPU was exhausted. A "block 
> in quick to $target_ip" on top of the ruleset was good enough to solve 
> the exhaustion problem.
> Now when this happens the input queue gets filled completely. There is 
> one input queue per address family, and the packets arriving on all 
> network interfaces are placed there. When it is full, we obviously have 
> a CPU problem - when we're in network congestion, the output queues run 
> full, but never the input queue, we simply do not have enough 
> (CPU) ressources to drain it fast enough.
> This observation let to a diff that sets a flag when the input queue is 
> full, and a timer that kicks in after 10 ms and unsets that flag.
> Whenever that flag is set pf stops evaluating the ruleset and returns a 
> stringt block, causing the packet to be dropped.
> This does not make the situation any worse: when the input queue is 
> full, newly arriving packets are dropped anyway. Handling the situation 
> this way allows for more selective drops; the state table is still 
> evaluated and thus established connections (for which a state table 
> entry exists) are not affected, and, IMHO even more important, leaves 
> enough CPU time to log in via console or already established ssh 
> connection and fix things.
> The 10ms timeout has been chosen experimentally; we started with a 1s 
> timeout and started lowering it. It turned out there was nearly no 
> difference in the machine load and behaviour depending on the timeout. 
> 10ms is the smallest timeout we can use on a standard kernel so that 
> was chosen.
> 
> Reproducing the condition was the hardest part - Bob and myself spent 
> more than a day on that. In the end it was Chris Kuethe who came up 
> with a little tool that could attack our poor little victim machine 
> fast enough.
> 
> DHCP
> ====
> 
> We started factoring out dhclient before already and cleaned it up big 
> time. I also added code to check for the interface link status and not 
> trying to send requests on interfaces without a link.
> We wanted to do the same to dhcpd too, and investigate privilege 
> seperation or -drop for it too. I started factoring out dhcrelay and 
> privilege revoked it, plus a lot of cleaning of course. The next target 
> was naturally dhcpd. After seperating it from the former usr.sbin/dhcp/ 
> mess I looked at privileve revoking it, and it turned out I had to 
> rewrite the leases file handling. dhcpd opened and closed the file all 
> the time, now instead we open it once and rewind/ftruncate it for 
> rewrites. Then I could let dhcpd drop privileges.
> canacar@ provided the bpf filters and bpf lock code. dhcpd and dhcrelay 
> open a bpf descriptor and install a read filter to see dhcp 
> requests/replies, and send requests/replies via bpf too. We added a 
> write filter and lock the bpf device, which results in the filters via 
> unchangeable, before dropping privs.
> Should anyone be able to exploit something in dhcpd now he will find 
> himself:
> -running as use _dhcp, which has no rights in the system whatsover
> -chrooted to /var/empty, where he has nothing to play but a dev/log
> -one open bpf descriptor, which only can send and receive packets 
>  from/to the dhcp ports
> 
> Later, after much prodding from theo, I attacked dhclient as well. It 
> executes a shell script on each status change which sets up the 
> interface, /etc/resolv.conf etc. That script has to run as root.
> dhclient now forks a child which keeps running as root on startup and 
> drops privileges to _dhcp and chroots in /var/emtpy. Whenever it needs 
> the script to ru it sends a message with the parameters to the 
> privileged child, which verifies the messages and executes the script 
> if the evaluation was positive. Care has been taken to not just send 
> the script's environment variables, but the data in binary form. The 
> privileged child exits (and thus kills the unprivileged process too) on 
> any inconsistency/corruption in the messages. After executing the 
> script it sends ithe script's exit code back to the unprivileged 
> process.
> We will soon add a bpf write filter and lock the bpf descriptor in 
> dhcrelay too.
> 
> Interestingly enough, as a side effect of the cleaning and especially 
> the removal of the fallback socket stuff (which we do not need on 
> OpenBSD as the bpf stuff works fine), the dhcp programs and especially 
> dhclient got much faster.
> 
> mopd and tftpd
> ==============
> 
> In some beer mood theo and myself made a little deal - he cleans up & 
> privilege seperates / revokes rarpd, and I do mopd. Turned out to be a 
> pretty good deal for me - after a few brain freezing hours of cleaning 
> up mopd, priv-revoking it looked easy. In fact priv revoking it only 
> took me like 10 minutes a few days later, mopd now drops privileges to 
> _mopd and chroots itself to /tftpboot/mop, where the files it serves 
> live.
> On the drive from calgary to edmonton with pval I briefly looked over 
> the small daemons we have in tree looking for ones that need to be 
> privsep'd/-revoked. I stumbled over tftpd which already is privilege 
> revoking, but used "nobody". "nobody" is overloaded, so this was 
> changed to "_tfpd".
> 
> bgpd
> ====
> During some boring windows talk I finally got back to a couple of diffs 
> I got from Brent Graveland for IPv6 in bgpd. While I solved a lot of 
> things differently, his diffs were of great help and inspiration, and I 
> had a sucessfull session over IPv6 between two bgpds within an hour.
> We do not yet support exchanging IPv6 routes in bgpd, but this will 
> come soon too.
> With claudio's arrival for the pf hackathon we got into the right bgpd 
> hacking mood again, and we got quite some things done - besides a lot 
> of smaller ones, claudio added route refresh support per RFC 2918, and 
> I wrote the capabilities advertising code and fleshed out the 
> capability parser. I also added neighbor cloning, where you can specify 
> an address/mask pair instead of a single IP address for a neighbor, 
> turning that neighbor into a "template". When a session from an 
> address within that range comes up there is a new neighbor entry cloned 
> from the template and the address/mask replaced by the connecting 
> peer's address. A small mod also allows to accept the AS the neighbor 
> advertizes itself as remote-as. This is helpfull in route server 
> setups, and when (ab)using bgpd to distribute prefix/netmask pairs for 
> means other then routing.
> With markus arrival we also started hacking on ipsec integration for 
> bgpd. With a couple of rather easy config file statements bgpd can now 
> set up ipsec encryption for the session to its neighbor with static 
> keys. There is also support for dynamic keying, which we are especially 
> proud of as it is unbelievable easy to set up - in the 
> neighbor statement in bgpd.conf, you add
>   ipsec [esp|ah] ike
> and add a default 2 lines isakmpd.policy file, copy the public keys 
> (which are now generated on the fist boot, just like the ssh host keys) 
> to the neighbor, respectively, and start up bgpd and isakmpd. That's 
> all - bgpd does the complicated part and sets up the flows.
> The need for the policy file might vanish soon too to make it even 
> easier...
> 
> 
> Besides these there was a lot of cool stuff happening at the pf 
> hackathon around the network stack and pf itself - many of those 
> changes are the result of discussions between all developers who've 
> been there, and the very special climate at this event did the rest. 
> Listing them all here would be too much; quite some stuff went in 
> already, and a lot is about to hit the tree.
> It was a great event.
> 
> 
> All in all, I had a wonderful month in Edmonton, Calgary, Vancouver and 
> Sechelt. Many thanks to Bob & Chenelle, Theo & Nadine and Ryan & Emily 
> for housing me (and a lot of fun), and Dragos for cansecwest.