Transparent ISP proxy problem or PF problem

[Alexander Iliev <alexander_(_dot_)_iliev_(_at_)_gmail_(_dot_)_com>]

Hi there.

First I want to state that I don't claim the problem I'm describing
below to be OpenBSD problem. It looks to me like a problem in the
particular set of setups between me, my ISP and the problem site.

Now, to the problem.

I'm using OpenBSD 3.8-release box as a router between a private network
(192.168.1.0/24) and the internet.

internet <-> (83.148.x.x) [OpenBSD] (192.168.1.1) <-> priv. lan

So far so good - the setup works very well with just one problem. My
ISP passes the traffic for two certain sites through a transparent
proxy. I reached to this conclusion due to the following:

---
$ telnet arenabg.com 80
Trying 82.101.72.23...
Connected to arenabg.com.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.0 503 Service Unavailable
Server: squid/2.5.STABLE5
Mime-Version: 1.0
Date: Thu, 01 Dec 2005 14:22:09 GMT
Content-Type: text/html
Content-Length: 1
Expires: Thu, 01 Dec 2005 14:22:09 GMT
X-Squid-Error: ERR_CONNECT_FAIL 111
X-Cache: MISS from <url>
Connection: close
---

For one of these two sites there is no problem - the traffic passes
through my ISP's transparent proxy and the site works perfectly. The
problem is with the other site. When one tries to open that other site
(arenabg.com), for example with Mozilla Firefox, the browser loads some
data (e.g. displays the title of the page) and continues loading
like forever. I don't this it's a browser problem, since the problem
exists on other browsers/versions too.

I tried to connect the cable for the internet directly to one of the
client machines behind the firewall (Debian GNU/Linux 3.1) and the site
loads perfectly, so I came to the conclusion that my PF rules are
blocking the packets. So, I left a minimal PF setup (pass all
keep state + NAT), but the problem remained.

After some research I found a common problem with very similar symptoms
called Path MTU Discovery problem - the packets sent from the server
are larger than the Path MTU of the route to the client and with DF flag
set, but the server (or some router) blocks the ICMP messages returned
to the server and does not get notified for the fact that it must
descrease the size of the packages.

Unfortunately it seems that this is not the case. Descreasing the MTU
on the router interface should have fixed the problem, but no luck.

Now, what I'm asking here is for some advice on how to proceed from
here in order to diagnose the problem.

Again, I don't claim this to be OpenBSD problem or bug, although I
tried to boot Freesco on the router machine and the problem site
(arenabg.com) did load.

Thanks in advance.

Best regards,
Alexander Iliev


configurations follow:

---------------- dmesg ----------------
OpenBSD 3.8 (GENERIC) #138: Sat Sep 10 15:41:37 MDT 2005
    deraadt_(_at_)_i386_(_dot_)_openbsd_(_dot_)_org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel 486DX (486-class)
real mem  = 33140736 (32364K)
avail mem = 22257664 (21736K)
using 430 buffers containing 1761280 bytes (1720K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(c2) BIOS, date 07/20/94
pcibios at bios0 function 0x1a not configured
bios0: ROM list: 0xc0000/0x8000
cpu0 at mainbus0
isa0 at mainbus0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
vga0 at isa0 port 0x3b0/48 iomem 0xa0000/131072
wsdisplay0 at vga0 mux 1: console (80x25, vt100 emulation), using wskbd0
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
wdc0 at isa0 port 0x1f0/8 irq 14
wd0 at wdc0 channel 0 drive 0: <ST33232A>
wd0: 16-sector PIO, LBA, 3077MB, 6303024 sectors
wd0(wdc0:0:0): using BIOS timings
ep0 at isa0 port 0x300/16 irq 10: address 00:20:af:07:f2:58, utp/aui\
 (default utp)
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom2 at isa0 port 0x3e8/8 irq 5: ns16450, no fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
isapnp0 at isa0 port 0x279: read port 0x203
ne3 at isapnp0 "UMC PLUG & PLAY  Ethernet Chip , UMC9008, PNP80D6, "\
 port 0x200/32 irq 3
ne3: NE2000 Ethernet
ne3: address 40:01:00:00:30:fb
biomask fbd5 netmask ffdd ttymask ffdf
pctr: no performance counters in CPU
dkcsum: wd0 matches BIOS drive 0x80
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302
---------------- /dmesg ----------------

---------------- pf.conf ----------------
# external interface
ext_if = "ne3"

# internal interface
int_if = "ep0"

# Bianor firewall/gateway
work_fw = "212.95.x.x"
webserver = "192.168.1.7"

table <allowed_icmp> persist { $work_fw, $provider }
table <allowed_ssh_to_ws> persist { $work_fw }

# set logging on for ext_if
set loginterface $ext_if

scrub in

altq on $int_if cbq bandwidth 4Mb queue {std,ssh}
  queue std bandwidth 70% cbq(default, borrow)
  queue ssh bandwidth 30% priority 5 cbq(borrow)

# nat on local networks
nat pass on $ext_if from $int_if:network
        to !$int_if:network -> ($ext_if)

# redirect to internal apache and ssh
rdr pass on $ext_if proto tcp from any to ($ext_if)
        port www -> $webserver port www
rdr pass on $ext_if proto tcp from $bianor_fw to ($ext_if)
        port 2222 -> $webserver port ssh

# deny all by default
block all

# allow loopback
pass quick on lo0

# allow local network traffic
pass in  on $int_if from $int_if:network to any keep state
pass out on $int_if proto tcp from any to $int_if:network port ssh
        keep state queue ssh
pass out on $int_if from any to $int_if:network keep state

pass out on $ext_if proto tcp all synproxy state flags S/SA
pass out on $ext_if proto { udp, icmp } all keep state


# allow icmp from bianor
pass in  on $ext_if proto icmp from <allowed_icmp> to ($ext_if)
        keep state

# allow outbound traffic for this machine
pass out on $ext_if from ($ext_if) to any keep state

# allow ssh from bianor fw
pass in on $ext_if proto tcp to ($ext_if) port ssh keep state
pass in on $ext_if proto tcp from <allowed_ssh_to_ws> to ($ext_if)
        port 2222 keep state

---------------- /pf.conf ----------------