FIXED! Connection pool oddities and PF - a solution.

[Dylan Smith <dylan_(_at_)_iompost_(_dot_)_co_(_dot_)_im>]

Finally - I've discovered a solution to my problem (mainly through error 
rather than trial, and a thorough re-reading of the manpage).

To recap: I have 2 ADSL connections with separate /29 netblocks and no means 
for proper load balancing (i.e. BGP). So a connection pool it was. The basic 
connection pool for *outgoing* traffic worked well from the example in the 
documentation, but for incoming traffic (see the rdr rule) the reply traffic 
was getting round-robined so half the time the reply packets were being sent 
out on the wrong external interface.

The rdr rule in the below example of a connection pool with an incoming 
redirection rule is using port 10101 so it was easy to distinguish between 
the outbound SSH traffic for my SSH session to a remote machine to test the 
incoming redirection rule.

OK - in the interests of global understanding, I will go through the whole 
process.
The example PF ruleset is basically very similar to this:

# NAT outgoing connections.
nat on $ext_if1 from $dmz_net to any -> ($ext_if1)
nat on $ext_if2 from $dmz_net to any -> ($ext_if2)
rdr on $ext_if1 proto tcp from any to $ext_addr1a port 10101 -> 172.16.1.2 
port 22

pass out on $int_if from any to $dmz_net

pass in on $int_if route-to { ( $ext_if1 $ext_gw1), ( $ext_if2 $ext_gw2 ) } \
        round-robin keep state
pass out on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 to any
pass out on $ext_if2 route-to ($ext_if1 $ext_gw1) from $ext_if1 to any

What I think was happening here is the incoming packet on port 10101 was 
hitting the redirect rule, and getting sent to 172.16.1.2 port 22 just fine. 
However, there was no state data with it. The reply packet, as far as pf was 
concerned, had no state associated with it yet so it just matched the 'pass 
on $int_if route-to...' round robin load balancing rule for outbound traffic.
This meant half the time the reply packet ended up being sent out on the 
internet on the wrong interface. It wouldn't get very far as the first 
upstream router would drop it as it didn't belong to the right network.

Yanko Karkalichev replied to my original post on the subject and suggested 
changing the pass out on $int_if... to the following:

pass out on $int_if from any to $dmz_net keep state

I think this fixed part of the problem - the problem that the redirected 
packet had no state information with it. It now had state information, but 
when the reply packet came back, the OpenBSD machine replied to 172.16.1.2 
with an ICMP Destination host unreachable message (as shown by tcpdump 
running on 172.16.1.2). I had a little think - well, of course - the system 
gets a reply to packet, but since the machine has no default route (all the 
routing is done via pf's route-to rules) when the reply packet hit the 
OpenBSD gateway it didn't know how to route it and so replied ICMP 
Destination Unreachable.

I tried many permutations of this to try and route the reply packets:
pass out on $int_if reply-to ( $ext_if1 $ext_gw1) from any to $dmz_net \
	keep state

But for some reason I'm still unsure of, this resulted in the reply packet 
still going out on the wrong ext_if 50% of the time.

So the next thing I tried was tagging the packets so I knew where they came 
from. So I added this experimental ruleset. From reading the documentation, I 
guessed that a packet would hit any rules for ext_ifN first, then the 
redirection rule, then any rules for int_if (i.e. where the redirection rule 
would spit the packet out). I felt that the trouble was occuring when it hit 
int_if - information about the packet's origin was getting forgotten about 
and so the reply was getting sent out on the wrong interface half the time.
So I decided to see if tagging the packets would help, which resulted in this 
experimental rule set. As the packets hit an external interface, the rules 
put a tag on saying which one it hit. Therefore, when it's been redirected 
and hits the int_if, we can still know from which ext_if it originated from, 
so only the correct rule (and reply-to route) will be matched. (Note: 
$ext_addr1a is a single IP address that belongs to the netblock that $ext_if 
is on - $ext_if is aliased to all five free IP addresses in the /29 
netblock).

# NAT outgoing connections.
nat on $ext_if1 from $dmz_net to any -> ($ext_if1)
nat on $ext_if2 from $dmz_net to any -> ($ext_if2)
rdr on $ext_if1 proto tcp from any to \
	$ext_addr1a port 10101 -> 172.16.1.2 port 22

# Keep track of what interface a packet arrived on.
pass in on $ext_if1 tag EXT_ONE keep state
pass in on $ext_if2 tag EXT_TWO keep state

# Connection pool rules
block out on $int_if
pass out on $int_if reply-to ($ext_if1 $ext_gw1) tagged EXT_ONE keep state
pass out on $int_if reply-to ($ext_if2 $ext_gw2) tagged EXT_TWO keep state

pass in on $int_if route-to { ( $ext_if1 $ext_gw1), ( $ext_if2 $ext_gw2 ) } \
        round-robin keep state
pass out on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 to any
pass out on $ext_if2 route-to ($ext_if1 $ext_gw1) from $ext_if1 to any

This ruleset worked perfectly, both directions!
Of course, for a full connection pooling firewall, a big 'pass in on...' on 
the external interface is probably not such a good idea, but this makes a 
good starting point that can be modified such that it makes a useful rule 
set.

-- 
Dylan Smith, Isle of Man Post, Spring Valley.
#include <std/disclaimer.h>