[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: What I did in April
- To: misc_(_at_)_openbsd_(_dot_)_org
- Subject: Re: What I did in April
- From: Christian Gut <cycloon_(_at_)_is-root_(_dot_)_org>
- Date: Wed, 5 May 2004 13:52:38 +0200
- Mail-followup-to: misc_(_at_)_openbsd_(_dot_)_org
WOW! Great work henning, especially the dhcp suite auditing and
rewriting is something typical for OpenBSD. Just a bit ahead.. ;)
On Tue, 04 May 2004, Henning Brauer wrote:
> I've been flying to Canada early in April, to help with the release and
> to attend the cansecwest conference and the pf hackathon at Ryan's in
> Sechelt, April 23 - 28.
> There was quite some time in between, and I could work on various
> Input queue congestion
> On the day I left germany, one of our main firewalls was attacked using
> a dDoS attack. The machine was completely unresponsive and I have been
> unable to log in and fix things, so it took a bit to solve the issue.
> Analysis showed that all the packets to randon ports on one IP caused
> that many pf ruleset evaluations that the CPU was exhausted. A "block
> in quick to $target_ip" on top of the ruleset was good enough to solve
> the exhaustion problem.
> Now when this happens the input queue gets filled completely. There is
> one input queue per address family, and the packets arriving on all
> network interfaces are placed there. When it is full, we obviously have
> a CPU problem - when we're in network congestion, the output queues run
> full, but never the input queue, we simply do not have enough
> (CPU) ressources to drain it fast enough.
> This observation let to a diff that sets a flag when the input queue is
> full, and a timer that kicks in after 10 ms and unsets that flag.
> Whenever that flag is set pf stops evaluating the ruleset and returns a
> stringt block, causing the packet to be dropped.
> This does not make the situation any worse: when the input queue is
> full, newly arriving packets are dropped anyway. Handling the situation
> this way allows for more selective drops; the state table is still
> evaluated and thus established connections (for which a state table
> entry exists) are not affected, and, IMHO even more important, leaves
> enough CPU time to log in via console or already established ssh
> connection and fix things.
> The 10ms timeout has been chosen experimentally; we started with a 1s
> timeout and started lowering it. It turned out there was nearly no
> difference in the machine load and behaviour depending on the timeout.
> 10ms is the smallest timeout we can use on a standard kernel so that
> was chosen.
> Reproducing the condition was the hardest part - Bob and myself spent
> more than a day on that. In the end it was Chris Kuethe who came up
> with a little tool that could attack our poor little victim machine
> fast enough.
> We started factoring out dhclient before already and cleaned it up big
> time. I also added code to check for the interface link status and not
> trying to send requests on interfaces without a link.
> We wanted to do the same to dhcpd too, and investigate privilege
> seperation or -drop for it too. I started factoring out dhcrelay and
> privilege revoked it, plus a lot of cleaning of course. The next target
> was naturally dhcpd. After seperating it from the former usr.sbin/dhcp/
> mess I looked at privileve revoking it, and it turned out I had to
> rewrite the leases file handling. dhcpd opened and closed the file all
> the time, now instead we open it once and rewind/ftruncate it for
> rewrites. Then I could let dhcpd drop privileges.
> canacar@ provided the bpf filters and bpf lock code. dhcpd and dhcrelay
> open a bpf descriptor and install a read filter to see dhcp
> requests/replies, and send requests/replies via bpf too. We added a
> write filter and lock the bpf device, which results in the filters via
> unchangeable, before dropping privs.
> Should anyone be able to exploit something in dhcpd now he will find
> -running as use _dhcp, which has no rights in the system whatsover
> -chrooted to /var/empty, where he has nothing to play but a dev/log
> -one open bpf descriptor, which only can send and receive packets
> from/to the dhcp ports
> Later, after much prodding from theo, I attacked dhclient as well. It
> executes a shell script on each status change which sets up the
> interface, /etc/resolv.conf etc. That script has to run as root.
> dhclient now forks a child which keeps running as root on startup and
> drops privileges to _dhcp and chroots in /var/emtpy. Whenever it needs
> the script to ru it sends a message with the parameters to the
> privileged child, which verifies the messages and executes the script
> if the evaluation was positive. Care has been taken to not just send
> the script's environment variables, but the data in binary form. The
> privileged child exits (and thus kills the unprivileged process too) on
> any inconsistency/corruption in the messages. After executing the
> script it sends ithe script's exit code back to the unprivileged
> We will soon add a bpf write filter and lock the bpf descriptor in
> dhcrelay too.
> Interestingly enough, as a side effect of the cleaning and especially
> the removal of the fallback socket stuff (which we do not need on
> OpenBSD as the bpf stuff works fine), the dhcp programs and especially
> dhclient got much faster.
> mopd and tftpd
> In some beer mood theo and myself made a little deal - he cleans up &
> privilege seperates / revokes rarpd, and I do mopd. Turned out to be a
> pretty good deal for me - after a few brain freezing hours of cleaning
> up mopd, priv-revoking it looked easy. In fact priv revoking it only
> took me like 10 minutes a few days later, mopd now drops privileges to
> _mopd and chroots itself to /tftpboot/mop, where the files it serves
> On the drive from calgary to edmonton with pval I briefly looked over
> the small daemons we have in tree looking for ones that need to be
> privsep'd/-revoked. I stumbled over tftpd which already is privilege
> revoking, but used "nobody". "nobody" is overloaded, so this was
> changed to "_tfpd".
> During some boring windows talk I finally got back to a couple of diffs
> I got from Brent Graveland for IPv6 in bgpd. While I solved a lot of
> things differently, his diffs were of great help and inspiration, and I
> had a sucessfull session over IPv6 between two bgpds within an hour.
> We do not yet support exchanging IPv6 routes in bgpd, but this will
> come soon too.
> With claudio's arrival for the pf hackathon we got into the right bgpd
> hacking mood again, and we got quite some things done - besides a lot
> of smaller ones, claudio added route refresh support per RFC 2918, and
> I wrote the capabilities advertising code and fleshed out the
> capability parser. I also added neighbor cloning, where you can specify
> an address/mask pair instead of a single IP address for a neighbor,
> turning that neighbor into a "template". When a session from an
> address within that range comes up there is a new neighbor entry cloned
> from the template and the address/mask replaced by the connecting
> peer's address. A small mod also allows to accept the AS the neighbor
> advertizes itself as remote-as. This is helpfull in route server
> setups, and when (ab)using bgpd to distribute prefix/netmask pairs for
> means other then routing.
> With markus arrival we also started hacking on ipsec integration for
> bgpd. With a couple of rather easy config file statements bgpd can now
> set up ipsec encryption for the session to its neighbor with static
> keys. There is also support for dynamic keying, which we are especially
> proud of as it is unbelievable easy to set up - in the
> neighbor statement in bgpd.conf, you add
> ipsec [esp|ah] ike
> and add a default 2 lines isakmpd.policy file, copy the public keys
> (which are now generated on the fist boot, just like the ssh host keys)
> to the neighbor, respectively, and start up bgpd and isakmpd. That's
> all - bgpd does the complicated part and sets up the flows.
> The need for the policy file might vanish soon too to make it even
> Besides these there was a lot of cool stuff happening at the pf
> hackathon around the network stack and pf itself - many of those
> changes are the result of discussions between all developers who've
> been there, and the very special climate at this event did the rest.
> Listing them all here would be too much; quite some stuff went in
> already, and a lot is about to hit the tree.
> It was a great event.
> All in all, I had a wonderful month in Edmonton, Calgary, Vancouver and
> Sechelt. Many thanks to Bob & Chenelle, Theo & Nadine and Ryan & Emily
> for housing me (and a lot of fun), and Dragos for cansecwest.
Visit your host, monkey.org