PF suddenly and silently drops ack packets for a passive FTP connection

[openbsd_(_at_)_rbaumann_(_dot_)_net]

Hi all

Probably somebody can help with a (at least for me) strange problem. I have a 
redundant OpenBSD based firewall setup with carp and pfsync. But for this 
problem it seems as if this is not the problem. Anyway...

The setup looks as follows:
                  +----------------------+
internet -----fxp0- OpenBSD -em0---- FTP server
                  +----------------------+

I have a 'block in log all' policy and I'm using binat to translate the public 
IP addresses to private ones I use on the FTP server.

Now the problem: I 'm using passive FTP (ProFTPD server on a Linux box, 
configured to use passive FTP with ports in the range 60001-64999). 
Connections work fine, I can also upload files, but suddenly, the upload 
stops. No more packets are passing. When doing a tcpdump on the interfaces on 
the firewall as well as on the Linux box, it looks like the firewall is 
silently dropping (or not passing) some ack packets. The pflog doesn't show an 
entry concerning these packets :(

Does somebody have an idea what could be wrong? Or an idea on how this problem 
can be further evaluated? I attached multiple log and tcpdump outputs, hope 
they help.
Thx a lot
Reto


--------------------- log's and config's ------------------------------
ftpclient.test.ch is used as IP for the connecting FTP client
ftpserver.test.ch for the external IP address for the FTP server
10.0.1.11 is the internal FTP server
firewall.test.ch is the external firewall interface

pf.conf
--------
<snip>
###################
# Macros
# interfaces
ext_if="fxp0"
dmz_if="em0"
# servers
ext_ftp="ftpserver.test.ch"
dmz_ftp="10.0.1.11"

<snip>

###################
# BINAT
binat on $ext_if from $dmz_ftp to any -> $ext_ftp

<snip>

###################     
# DEFAULT POLICY: Block all
block in log all
####
# DMZ traffic
# temporarily allow all outgoing connections from DMZ
pass in quick on $dmz_if inet from any to any keep state
pass out quick on $ext_if from 10.0.1.0/24 to any keep state
# allow incoming FTP
pass in quick on $ext_if inet proto tcp \
        from any to $dmz_ftp port 21 keep state
pass out quick on $dmz_if inet proto tcp \
        from any to $dmz_ftp port 21 keep state
pass in quick on $ext_if inet proto tcp \
        from any to $dmz_ftp port 60000 >< 65000 keep state
pass out quick on $dmz_if inet proto tcp \
        from any to $dmz_ftp port 60000 >< 65000 keep state
<snip>

tcpdump FTP server:
-------------------------
14:23:40.630446 10.0.1.11.64651 > ftpclient.test.ch.30418: . ack 17539073 win 
62780 (DF)
14:23:40.658228 ftpclient.test.ch.30418 > 10.0.1.11.64651: . 17539073:17540533
(1460) ack 1 win 64240
14:23:40.662234 ftpclient.test.ch.30418 > 10.0.1.11.64651: . 17540533:17541993
(1460) ack 1 win 64240
14:23:40.662276 10.0.1.11.64651 > ftpclient.test.ch.30418: . ack 17541993 win 
62780 (DF)
14:23:40.674468 ftpclient.test.ch.30418 > 10.0.1.11.64651: . 17541993:17543453
(1460) ack 1 win 64240
14:23:40.688260 ftpclient.test.ch.30418 > 10.0.1.11.64651: . 17543453:17544913
(1460) ack 1 win 64240
14:23:40.688298 10.0.1.11.64651 > ftpclient.test.ch.30418: . ack 17544913 win 
62780 (DF)
14:23:40.704272 ftpclient.test.ch.30418 > 10.0.1.11.64651: . 17544913:17546373
(1460) ack 1 win 64240
14:23:40.710673 ftpclient.test.ch.30418 > 10.0.1.11.64651: P 17546373:17547265
(892) ack 1 win 64240
14:23:40.710710 10.0.1.11.64651 > ftpclient.test.ch.30418: . ack 17547265 win 
62780 (DF)
14:23:40.730366 ftpclient.test.ch.30418 > 10.0.1.11.64651: . 17547265:17548725
(1460) ack 1 win 64240
14:23:40.740459 ftpclient.test.ch.30418 > 10.0.1.11.64651: . 17548725:17550185
(1460) ack 1 win 64240
14:23:40.740493 10.0.1.11.64651 > ftpclient.test.ch.30418: . ack 17550185 win 
62780 (DF)



tcpdump Firewall external interface:
-------------------------
16:24:13.689509 ftpserver.test.ch.64651 > ftpclient.test.ch.30418: . ack 
17547265 win 62780 (DF)
16:24:13.707398 ftpclient.test.ch.30418 > ftpserver.test.ch.64651: . 
17547265:17548725(1460) ack 1 win 64240 (DF)
16:24:13.717486 ftpclient.test.ch.30418 > ftpserver.test.ch.64651: . 
17548725:17550185(1460) ack 1 win 64240 (DF)
16:24:13.719364 ftpserver.test.ch.64651 > ftpclient.test.ch.30418: . ack 
17550185 win 62780 (DF)
16:24:14.095694 ftpclient.test.ch.30418 > ftpserver.test.ch.64651: . 
17550185:17551645(1460) ack 1 win 64240 (DF)
16:24:14.899489 ftpclient.test.ch.30418 > ftpserver.test.ch.64651: . 
17550185:17551645(1460) ack 1 win 64240 (DF)
16:24:16.513066 ftpclient.test.ch.30418 > ftpserver.test.ch.64651: . 
17550185:17551645(1460) ack 1 win 64240 (DF)
16:24:19.732329 ftpclient.test.ch.30418 > ftpserver.test.ch.64651: . 
17550185:17551645(1460) ack 1 win 64240 (DF)
16:24:26.174905 ftpclient.test.ch.30418 > ftpserver.test.ch.64651: . 
17550185:17551645(1460) ack 1 win 64240 (DF)



tcpdump Firewall internal interface:
-------------------------
16:24:13.667137 10.0.1.11.64651 > ftpclient.test.ch.30418: . ack 17544913 win 
62780 (DF)
16:24:13.681320 ftpclient.test.ch.30418 > 10.0.1.11.64651: . 17544913:17546373
(1460) ack 1 win 64240
16:24:13.688299 ftpclient.test.ch.30418 > 10.0.1.11.64651: P 17546373:17547265
(892) ack 1 win 64240
16:24:13.689497 10.0.1.11.64651 > ftpclient.test.ch.30418: . ack 17547265 win 
62780 (DF)
16:24:13.707411 ftpclient.test.ch.30418 > 10.0.1.11.64651: . 17547265:17548725
(1460) ack 1 win 64240
16:24:13.717500 ftpclient.test.ch.30418 > 10.0.1.11.64651: . 17548725:17550185
(1460) ack 1 win 64240
16:24:13.719351 10.0.1.11.64651 > ftpclient.test.ch.30418: . ack 17550185 win 
62780 (DF)


PF state table (bash-2.05b# pfctl -s state | grep ftpclient.test.ch)
-------------------------
self tcp ftpclient.test.ch:252 -> 10.0.1.11:21       ESTABLISHED:ESTABLISHED
self tcp ftpclient.test.ch:38655 -> 10.0.1.11:61466       
ESTABLISHED:ESTABLISHED
self tcp ftpclient.test.ch:2790 <- 10.0.1.11:60207       
ESTABLISHED:ESTABLISHED
self tcp ftpclient.test.ch:32245 -> 10.0.1.11:64609       
ESTABLISHED:ESTABLISHED
self tcp ftpclient.test.ch:30418 <- 10.0.1.11:64651       
ESTABLISHED:ESTABLISHED
self tcp ftpclient.test.ch:12518 -> 10.0.1.11:61101       
ESTABLISHED:ESTABLISHED
self tcp ftpclient.test.ch:57334 <- 10.0.1.11:60097       
ESTABLISHED:ESTABLISHED
self tcp 10.0.1.11:64651 -> ftpserver.test.ch:64651 -> 
ftpclient.test.ch:30418       ESTABLISHED:ESTABLISHED
self tcp 10.0.1.11:60207 -> ftpserver.test.ch:60207 -> 
ftpclient.test.ch:2790       ESTABLISHED:ESTABLISHED
self tcp 10.0.1.11:61101 <- ftpserver.test.ch:61101 <- 
ftpclient.test.ch:12518       ESTABLISHED:ESTABLISHED
self tcp 10.0.1.11:64609 <- ftpserver.test.ch:64609 <- 
ftpclient.test.ch:32245       ESTABLISHED:ESTABLISHED
self tcp 10.0.1.11:60097 -> ftpserver.test.ch:60097 -> 
ftpclient.test.ch:57334       ESTABLISHED:ESTABLISHED
self tcp 10.0.1.11:21 <- ftpserver.test.ch:21 <- ftpclient.test.ch:252       
ESTABLISHED:ESTABLISHED
self tcp 10.0.1.11:61466 <- ftpserver.test.ch:61466 <- 
ftpclient.test.ch:38655       ESTABLISHED:ESTABLISHED