[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

PF, Bridge, and IP on bridged interface [more]



Okay, I can get my bridge and pf rules working if I need to, but
I'd still like to understand WHY they work they way they do.  So
I ran some test cases.

My configuration is this:

OpenBSD/Sparc (SS20).  I have one external interface, and two
internal interfaces.  There's NAT to the external, but that's
working flawlessly, so we'll leave that out of the equation for now.

The two internal interfaces are bridged, and I have various pf rules
restricting what can and cannot pass between them (as well as
between the internal and external world).  However, I'm not entirely
clear on what's going on... where the bridge acts in the general
process flow, whether bridge rules act before or after frames are
copied to the bridged interfaces, exactly how the bridge tagging
rules work, etc.

What I'm looking to do is construct something like the process flow
diagrams at:
	http://mniam.net/pf/pf.png
	and
	http://homepage.mac.com/quension/pf/flow.png

except with a bridge in the model.

So I decided to run some test cases with rule sets that pass and log
everything.  Now that I have some data, and I've looked it over, I
have a few questions.

So, here's my first, simple case:

$ cat /etc/bridgename.bridge0
add le0
add le2
blocknonip le0
blocknonip le2
rule pass in on le0 tag t_lan
rule pass in on le2 tag t_wap
up
$ cat /etc/hostname.le0
inet 192.168.1.1 255.255.255.0 NONE
$ cat /etc/hostname.le2
up

pf rules:

	@0 pass out quick on lo0 all
	@1 pass in quick on lo0 all
	@2 pass in log-all quick on le0 all
	@3 pass in log-all quick on le2 all
	@4 pass out log-all quick on le0 all
	@5 pass out log-all quick on le2 all

Router: 192.168.1.1
LAN client: 192.168.1.9
WLAN client: 192.168.1.130



And I proceeded to execute some pings, from the console of each
machine.

LAN machine pings router:
Passes in on le2 (incorrect), reply passes out on le0 (correct)

Feb 16 08:28:43.378979 rule 3/0(match): pass in on le2: 192.168.1.9 > 192.168.1.1 : icmp: echo request (id:0f5a seq:0) (ttl 64, id 3081)
Feb 16 08:28:43.379197 rule 4/0(match): pass out on le0: 192.168.1.1 > 192.168.1.9 : icmp: echo reply (id:0f5a seq:0) (ttl 255, id 36469)




LAN machine pings WLAN machine:
Request passes in on le0 (correct) first, then out on le2 (correct)
Reply   passes in on le2 (correct) first, then out on le0 (correct)

Feb 16 08:28:54.881680 rule 2/0(match): pass in on le0: 192.168.1.9 > 192.168.1.130: icmp: echo request (id:0f5b seq:0) (ttl 64, id 3108, bad cksum 0!)
Feb 16 08:28:54.881737 rule 5/0(match): pass out on le2: 192.168.1.9 > 192.168.1.130: icmp: echo request (id:0f5b seq:0) (ttl 64, id 3108)
Feb 16 08:28:54.882440 rule 3/0(match): pass in on le2: 192.168.1.130 > 192.168.1.9 : icmp: echo reply (id:0f5b seq:0) (ttl 64, id 48170, bad cksum 0!)
Feb 16 08:28:54.882487 rule 4/0(match): pass out on le0: 192.168.1.130 > 192.168.1.9 : icmp: echo reply (id:0f5b seq:0) (ttl 64, id 48170)




WLAN machine pings router:
Passes in on le2 (correct), reply passes out on le0 (incorrect)

Feb 16 08:29:22.925161 rule 3/0(match): pass in on le2: 192.168.1.130 > 192.168.1.1 : icmp: echo request (id:028b seq:0) (ttl 64, id 48190)
Feb 16 08:29:22.925292 rule 4/0(match): pass out on le0: 192.168.1.1 > 192.168.1.130: icmp: echo reply (id:028b seq:0) (ttl 255, id 53634)




WLAN machine pings LAN machine:
Request passes in on le2 (correct) first, then out on le0 (correct)
Reply   passes in on le0 (correct) first, then out on le2 (correct)

Feb 16 08:29:32.830865 rule 3/0(match): pass in on le2: 192.168.1.130 > 192.168.1.9 : icmp: echo request (id:028c seq:0) (ttl 64, id 48200, bad cksum 0!)
Feb 16 08:29:32.830917 rule 4/0(match): pass out on le0: 192.168.1.130 > 192.168.1.9 : icmp: echo request (id:028c seq:0) (ttl 64, id 48200)
Feb 16 08:29:32.831474 rule 2/0(match): pass in on le0: 192.168.1.9 > 192.168.1.130: icmp: echo reply (id:028c seq:0) (ttl 64, id 3191, bad cksum 0!)
Feb 16 08:29:32.831523 rule 5/0(match): pass out on le2: 192.168.1.9 > 192.168.1.130: icmp: echo reply (id:028c seq:0) (ttl 64, id 3191)




Router pings LAN machine:
Passes out on le0 (correct), reply passes in on le2 (incorrect)

Feb 16 08:29:48.837588 rule 4/0(match): pass out on le0: 192.168.1.1 > 192.168.1.9 : icmp: echo request (id:16ee seq:0) (ttl 255, id 62936)
Feb 16 08:29:48.838269 rule 3/0(match): pass in on le2: 192.168.1.9 > 192.168.1.1 : icmp: echo reply (id:16ee seq:0) (ttl 64, id 3223)


Router pings WLAN machine:
Passes out on le0 (incorrect), reply passes in on le2 (correct)

Feb 16 08:29:56.494341 rule 4/0(match): pass out on le0: 192.168.1.1 > 192.168.1.130: icmp: echo request (id:7ff8 seq:0) (ttl 255, id 60383)
Feb 16 08:29:56.495223 rule 3/0(match): pass in on le2: 192.168.1.130 > 192.168.1.1 : icmp: echo reply (id:7ff8 seq:0) (ttl 64, id 48215)



1a. My pf rules are all log-all

1b. bridge (4), and all of the literature I can find online says
	that frames on the bridge will pass through pf twice.
	"Bridged frames pass through pf(4) twice.  They can be filtered
	on any interface, in both directions."

-> They do not appear to do so. They appear to pass through pf only once.
That is, they pass in once and out once, exactly as they do when
routing between two Subnets in an unbridged configuration. Why?
Is the documentation wrong? Are there frames that are not showing
up in my pflog, despite all rules being log-all? For this
"dual pass" through pf, does it only make a second pass if it
has not found a destination? When running these tests, I first
did a `brconfig bridge0 flushall` in the hopes of eliminating such
a scenario.




2a. traffic passing ACROSS the bridge uses all the "correct" rules
	for the machines and the interfaces they are on

2b. traffic from the LAN to the router appears to come from the
	wrong interface

2c. traffic from the router to the LAN appears to go out on the
	correct interface

2d. traffic from the WLAN to the router appears to come from the
	correct interface

2d. traffic from the router to the WLAN appears to go out on the
	wrong interface

-> Half of this -- the outbound traffic, can be explained, I
	believe, by the fact that le0 is assigned the network number &
	IP.  All outbound traffic from the router, whether to the LAN or
	the WLAN, goes out on le0.  That makes sense.  After all, le2
	has no network.  It wouldn't show up in system routing policy.
	Only the bridge knows about it.
	
-> This does not explain why all inbound traffic from both LAN and
	WLAN appears to come from le2.  Why is this?  It's obviously
	some function of the bridge interface.  I could see if the
	bridge made it appear as though all frames were inbound on both
	interfaces -- that's what I was expecting; it would mesh with
	the statement that frames pass through pf "twice."  But I'm not
	seeing them pass through pf twice, only "once" (once inbound,
	once outbound, just like unbridged interfaces).

-> Obviously, this is the bridge at work...
	Is this behaviour determinate?
	Can I expect all inbound traffic to the router to always appear
	as though it's from le2, regardless of which interface it
	actually arrived on?
	
	Is which interface gets picked as the one on which all frames
	arrive due to the order in which the interfaces are added to the
	bridge?  Is it due to the interface number, or some other
	deterministic mechanism?  Or can I expect this to change from
	boot to boot, from one box to another, if I swap out interface
	cards, or if it is run on another hardware platform (e.g. is it
	endian)?  Or, perhaps, does it depend on which interface I
	assign the IP address to (IP is assigned to le0)?


Next, trying to learn more about what's going on behind the scenes, I looked at a more complex ruleset for pf.

Here I attempt to emply my bridge tagging rules to clear up the mess
from before, where frames destined for the router itself appeared to
always be coming in on le2, regardless of which physical device they
actually arrived on.

bridge rules:

	pass in on le0 tag t_lan
	pass in on le2 tag t_wap

pf rules:

	@0 pass out quick on lo0 all
	@1 pass in quick on lo0 all
	@2 pass in log-all quick on le0 all keep state tagged t_wan
	@3 pass in log-all quick on le2 all keep state tagged t_lan
	@4 pass in log-all quick on le2 all keep state tagged t_wan
	@5 pass in log-all quick on le0 all keep state tagged t_lan
	@6 pass out log-all quick on le0 all keep state tagged t_wan
	@7 pass out log-all quick on le0 all keep state tagged t_lan
	@8 pass out log-all quick on le2 all keep state tagged t_wan
	@9 pass out log-all quick on le2 all keep state tagged t_lan
	@10 pass in log-all quick on le0 all keep state
	@11 pass in log-all quick on le2 all keep state
	@12 pass out log-all quick on le0 all keep state
	@13 pass out log-all quick on le2 all keep state

LAN machine pings router:

Passes in on le2 (incorrect), reply passes out on le0 (correct)

This behaves just as in the simple-rules case before.

Incidentally, rule #11 catches both inbound and outbound frames,
Which makes sense only if the rule applies to the "reply" simply
because it applied to the request (due to 'keep state).

The troubling bit is that, ALL of the 'tagged' rules plainly fail.
The ICMP request should have been tagged t_lan by the bridge, even
if the reply doesn't get tagged as anything by the bridge.

Does the bridge interface NOT tag frames that are to be routed
elsewhere (e.g. to the router itself, or to some interface that is
not a part of the bridge)?

Feb 16 08:31:27.701271 rule 11/0(match): pass in on le2: 192.168.1.9 > 192.168.1.1 : icmp: echo request (id:0f5f seq:0) (ttl 64, id 3350)
Feb 16 08:31:27.701551 rule 11/0(match): pass out on le0: 192.168.1.1 > 192.168.1.9 : icmp: echo reply (id:0f5f seq:0) (ttl 255, id 8988)




LAN machine pings WLAN machine:

Request passes in on le0 (correct) first, then out on le2 (correct)
Rule 5 (pass in on le0 tagged t_lan) and rule 9 (pass out on le2
tagged t_lan) are the rules that catch here.  This is what I would
expect.

The reply passes in on le2 (correct) first, then out on le0 (correct)

Rules 5 & 9 fire again.  I might have expected that rules 4 & 6 to
catch (pass in on le2 tagged t_wan and pass out on le0 tagged
t_wan), since the reply comes from le2.

However, there are possible expanations for this.  First, they
remain tagged as t_lan because the connection "originated" on the
LAN, because that's where the ICMP dialogue was initiated, and 'keep
state' takes over.

Am I correct in suspecting that pass in on le2 matches rule 9
because rule 9 was what allowed the ICMP echo request to pass out
that interface, and that rule 5 matches pass out on le0 because rule
5 was what allowed the request to pass in on that interface?

The other possibility is more unsettling:

	brconfig (8):
		"If no source or destination address is specified, the rule
		will match all frames (good for creating a catchall policy)."

If this statement is correct, then if no addresses are given, the
rule matches ALL frames, regardless of the interface.  Thus my
bridge rules:

	pass in on le0 tag t_lan
	pass in on le2 tag t_wap

Would see all frames arriving, whether on le0 or le2, and match the
first rule and get tagged t_lan.  Strictly speaking, that's what
the excerpt from brconfig (8) says.  Is this the case?

Feb 16 08:31:36.303072 rule 5/0(match): pass in on le0: 192.168.1.9 > 192.168.1.130: icmp: echo request (id:0f60 seq:0) (ttl 64, id 3368, bad cksum 0!)
Feb 16 08:31:36.303255 rule 9/0(match): pass out on le2: 192.168.1.9 > 192.168.1.130: icmp: echo request (id:0f60 seq:0) (ttl 64, id 3368)
Feb 16 08:31:36.304040 rule 9/0(match): pass in on le2: 192.168.1.130 > 192.168.1.9 : icmp: echo reply (id:0f60 seq:0) (ttl 64, id 48246, bad cksum 0!)
Feb 16 08:31:36.304080 rule 5/0(match): pass out on le0: 192.168.1.130 > 192.168.1.9 : icmp: echo reply (id:0f60 seq:0) (ttl 64, id 48246)




WLAN machine pings router:

Passes in on le2 (correct), reply passes out on le0 (incorrect)

This behaves just as in the simple case before.

Again, ALL of the 'tagged' rules plainly fail.  I have no
explanation for this.  Those rules should be exhaustive.  Unless the
bridge is not tagging the frames at all.

The ICMP request should have been tagged t_wan by the bridge, even
if the reply doesn't get tagged as anything by the bridge.

Or, even if my suspicions are correct about bridge rules needing
addresses to apply, they ought to have at least been tagged t_lan.

Feb 16 08:31:49.863045 rule 11/0(match): pass in on le2: 192.168.1.130 > 192.168.1.1 : icmp: echo request (id:028d seq:0) (ttl 64, id 48254)
Feb 16 08:31:49.863258 rule 11/0(match): pass out on le0: 192.168.1.1 > 192.168.1.130: icmp: echo reply (id:028d seq:0) (ttl 255, id 10230)




WLAN machine pings LAN machine:

Similar to above, where LAN pings WLAN, in that the interfaces for
in and out are correct for the request.  Presume, again, that the
reply is passed by rules 11 & 12 because the request was passed by
them.

Again, none of the 'tagged' rules are used.

Feb 16 08:31:59.793930 rule 11/0(match): pass in on le2: 192.168.1.130 > 192.168.1.9 : icmp: echo request (id:028e seq:0) (ttl 64, id 48262, bad cksum 0!)
Feb 16 08:31:59.794069 rule 12/0(match): pass out on le0: 192.168.1.130 > 192.168.1.9 : icmp: echo request (id:028e seq:0) (ttl 64, id 48262)
Feb 16 08:31:59.794677 rule 12/0(match): pass in on le0: 192.168.1.9 > 192.168.1.130: icmp: echo reply (id:028e seq:0) (ttl 64, id 3410, bad cksum 0!)
Feb 16 08:31:59.794715 rule 11/0(match): pass out on le2: 192.168.1.9 > 192.168.1.130: icmp: echo reply (id:028e seq:0) (ttl 64, id 3410)




Router pings LAN machine:

Again, presume 'keep state' is causing the same rule to pass both.

Here, there is nothing at all troubling about the fact that the
request isn't picked up by any 'tagged' rules, because the bridge
rules aren't tagging outbound traffic.

If I were to add bridge rules:

	pass out on le0 tag t_lan
	pass out on le2 tag t_wap

Would the 'tagged' rules pick up this traffic?

I'm not so bothered by the fact that the reply isn't picked up by a
'tagged t_wan' rule, since I'm assuming that it's being passed by
rule 12 due to 'keep state.'

Feb 16 08:32:14.582224 rule 12/0(match): pass out on le0: 192.168.1.1 > 192.168.1.9 : icmp: echo request (id:24ae seq:0) (ttl 255, id 19737)
Feb 16 08:32:14.583016 rule 12/0(match): pass in on le2: 192.168.1.9 > 192.168.1.1 : icmp: echo reply (id:24ae seq:0) (ttl 64, id 3438)




Router pings WLAN machine:

Again, same rule for both ('keep state).
Again, data out is on le0, even though the machine is on le2.
Again, none of the tagged rules pick up any of this, but it's
traffic originating from the router, so the first message wouldn't
be tagged anyway.

Feb 16 22:32:40.087591 rule 12/0(match): pass out on le0: 192.168.1.1 > 192.168.1.130: icmp: echo request (id:5ecf seq:0) (ttl 255, id 16351)
Feb 16 22:32:40.088645 rule 12/0(match): pass in on le2: 192.168.1.130 > 192.168.1.1: icmp: echo reply (id:5ecf seq:0) (ttl 64, id 56062)




I attempt to have the bridge tag traffic arriving on le0 as t_lan,
and arriving on le2 as t_wap.  Even allowing for the notion that
'keep state' means that 'reply' frames are picked up by the rule
that allowed the request frame, I don't understand why ICMP sessions
initiated by LAN and WLAN clients don't get picked up by the tagged
rules, except in one case (the case of clients on opposite sides of
the bridge talking to each other) -- and the tagging there only
works in ONE direction (t_lan tags are used, t_wan tags are not).

So, is the second bridge rule never firing because 'pass in on le0
tag t_lan' catches everything, even frames arriving on le2?  Are
frames arriving on one interface coped to the other interface and
THEN have the rules applied?


Help?

Thanks,
Jim