Help with pf + pfsync + carp

[Matthew Grooms <mgrooms_(_at_)_seton_(_dot_)_org>]

All,

     I am evaluating OpenBSD as a potential replacement for some 
checkpoint fw equipment and am having some difficulty getting pf + 
pfsync + carp to reliably handle transparent failover. Im sure its just 
my lack of experience with the system and have most likely goofed some 
configuration parameter somewhere. My setup is simple as it is a test 
case in our lab. I have two firewalls configured to simulate failover by 
shutting down switchports randomly. The hardware consists of two Dual 
AMD64 3+GZ Processor with 4GB of Ram, LSI Controllers w/ RAID5 and Intel 
Quad Port Gigabit cards ( all overkill for firewall I know ) running the 
stock 3.6 UP i386 kernel and with both boxes configured like so ...

em0 - public
em1 - private
em5 - pfsync interconnect
carp0 - public
carp1 - public
pfsync0 - for state replication

fw1 interface configs ...

hostname.em0 = 192.168.253.2 255.255.255.0 NONE
hostname.em1 = 192.168.254.2 255.255.255.0 NONE
hostname.em5 = 192.168.252.1 255.255.255.0 NONE
hostname.carp0 = 192.168.253.1 255.255.255.0 NONE vhid 1 advskew 1 ...
hostname.carp1 = 192.168.254.2 255.255.255.0 NONE vhid 2 advskew 1 ...
hostname.pfsync0 = up syncif em5

fw2 interface configs ...

hostname.em0 = 192.168.253.3 255.255.255.0 NONE
hostname.em1 = 192.168.254.3 255.255.255.0 NONE
hostname.em5 = 192.168.252.2 255.255.255.0 NONE
hostname.carp0 = 192.168.253.1 255.255.255.0 NONE vhid 1 advskew 100 ...
hostname.carp1 = 192.168.254.2 255.255.255.0 NONE vhid 1 advskew 100 ...
hostname.pfsync0 = up syncif em5

my pf.conf on both firewalls look like this ...

nat on carp0 inet from !em0 -> carp0
pass quick on em0 inet proto carp
pass quick on em1 inet proto carp
pass quick on em5 inet
pass quick log-all inet keep state

     The CARP failover portion works great. Its a bit confusing to 
configure since the man page is a bit non-verbose, but when an active 
interface is unplugged, the secondary firewalls carp interface 
transitions to a MASTER state and everything looks groovy. After the 
failover, pf passes traffic well when new connections are open by a 
client in the private network to the public network. However, if I have 
an open TCP connection, ( like an ftp download ) it will stall and 
eventually fail. Using tcpdump to monitor the em0 interface on the 
backup firewall reveals that the ftp tcp connection is alive and packets 
are being forward to it as soon as its carp interface transitions to 
MASTER. But when tcpdump is used to monitor the pflog0 interface, it 
does not show any matched rules. It seems to me that the NAT and rule 
state should be present on the backup firewall due to pfsync and the 
traffic should first get translated and then get passed to the client 
endpoint of the tcp stream. At first I though the NAT rule was to blame 
but the tcpdump output from em0 shows the packets addressed as ...

[ftp server address] > 192.168.254.100

... which looks like the packet should look post translation. I tried 
changing the rule ...

nat on carp0 inet from !em0 -> carp0

... to ...

nat on em0 inet from !em0 -> em0

... for grins but it did not appear to help. I can see that state info 
is being replicated to the backup firewall via pfsync. For example, when 
use pfctl -s state to view the ftp connections state entries ...

self tcp 192.168.254.100:2432 -> [ftp server address]:63703 
ESTABLISHED:ESTABLISHED
self tcp [ftp server address]:63703 <- 192.168.254.100:2432 
ESTABLISHED:ESTABLISHED

... shows up on both the primary and backup firewall before the failover 
happens. Should I be able to glean any NAT related state info from this 
commands output? Anyhow, when I try the same sort of test with a more 
interactive connection ( like ssh ), the connection will stall until the 
client generates a bit of outbound traffic. At that point all the 
inbound traffic ( like if you had started a ping on the remote machine 
via the ssh terminal session to generate constant screen activity ) 
pours in at once.

    So, am I missing something fundamental or doing something obviously 
wrong? Any suggestions would be greatly appreciated. Please include me 
in responses as I do not subscribe to this list.

--
Matthew Grooms
mgrooms_(_at_)_seton_(_dot_)_org