[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Summary: Cable/ADSL Load Sharing

OK, this e-mail doesn't really have any questions in it.  Specifically,
I only sent it so that this gets put into the archives so someone
wanting to do what I have done can find it and possibly learn
from my mistakes.

A few months ago, I was on this very mailing list asking about advice
for basically shotgunning ADSL and Cable Modem -- not so much for the
speed advantage, but for the bandwidth.  Our house does a LOT of
net traffic a day, and our cable modem provider was getting more than
a little angry at us for it; so we got BOTH (the idea being to send
half of our traffic out each interface).

Actually, this lead to some rather amusing conversations.  The ADSL
repair guy came down the stairs, saw the cable modem and the
64-port patch panel, the 10-high stack of motherboards on the shelf,
and the rather artsy use of P-75s as a tower, and his eyes just
glazed over.

Where we started was a network of this setup:

   |-------|   |----------|        |
---| Cable |---| Router 1 |--------|
   |-------|   |----------|        |

With all of our computers connecting through the gateway (
Note that this required NAT.  ALSO note that we run a domain, so we
had to make sure that incoming connections could get through the
router and in to our network in order to reach the web/mail servers
in our DMZ.  A simple, typical home setup.  Then, with the complaint
of the cable company, things got really complicated pretty darn
quickly.  What follows is a pretty horrendously messy account (hopefully
not missing too much) of what steps I undertook in order to attempt
to get Cable and ADSL to be shared by the computers on our network.
As an aside, please don't condemn me for stupid mistakes -- I am in no
way a networking professional, just a current CS student that likes to
play with networks :)

Alright, then, into the breech.

My first idea was simply to throw another NIC into the existing OpenBSD
router, so we would have:

   |-------|   |----------|
---| ADSL  |---| Router 1 |--------|
   |-------|  /|----------|        |
             /          |     
---| Cable |

Now some of the more experienced network workers might see problems with
this right away, but it took me a while longer to see it wouldn't work.
Setting this up was easy -- after we got the @($#*_(_at_)_$ ADSL company to get us
a working line (they consistently blamed OUR setup, since OBVIOUSLY we
didn't even know how to see if we could get and address from their @($)* DHCP
server... sorry.  It was frustrating).  So after arranging for them to visit
us, at which point they decided that it MIGHT be the port to our house, and
they switched us on to a new one which started to work immediately.


Anyway, I now had a router with two external interfaces and one internal 
(leading to a 10.1.x.x network).  This was fine, except it wasn't splitting
the traffic.  It was all going out the cable modem still.  Which is again
fine, because that was the default route and I expected it.  No problem.
Simply go into the routing setup and add routes FORCING certain subnets
out on whatever external interface you want, right?  Make sure you do
the NATing correctly (i.e. NAT on both external interfaces), and it will
all work.

It won't properly load balance, but if I could split up the
subnets that we see a LOT of traffic from (i.e. newsfeeds and streamload.com),
then half of the traffic would go out one interface and half would go out
the other interface, and voila.  No problem.  Or so I thought.  I instituted
the static routes, and all was well (after a few weeks of keeping track of
where our traffic was going so I knew what subnets to split up, anyway).
Until we started having problems with our domain (all the guys who know
their networking out there are nodding their heads right now, I bet, but
I will explain the problem to those network-clueless [i.e. me]).  See, with
outgoing connections from the network out, there is no problem -- the OpenBSD
router would look at the outgoing address, decide where it should go and send
it through on that interface.  The destination server would get the packet
and start up a connection and everything would be good.  The problem was in
the forming of the INCOMING connections.  For argument's sake, let's say that
I split the network into subnet A, which goes out on cable and subnet B which
goes out on ADSL.  Now remember that with those two external interfaces,
we have two different external IPs.  One of those IPs was associated
with our domain (specifically, the cable modem external IP).  So anyone connecting
to our domain would connect to the cable modem interface.  However, what if the
person was connecting from subnet B, which was supposed to go out on ADSL?  The
router would send the packets out on the ADSL interface, but they would be labelled
with the cable modem external IP.  Our ADSL provider doesn't route packets not
from it's network (an eminently reasonable position), so these packets would get
dropped, and people on subnet B could not connect to our domain.  This was
an untenable position.  So I thought and thought and thought and came up with this
fearsome setup:

   |-------|   |----------|
---| ADSL  |---| Router 1 |--------|
   |-------|   |----------|        |
   |-------|   |----------|        |
---| Cable |---| Router 2 |--------|
   |-------|   |----------|        |
               |----------|        |
               | Router 3 |--------|
               |----------|        |

That third router is probably confusing people right now :)  Oh I should also mention
that since we had two external IPs, we started hosting another domain.  So we
have two domains, and two external IPs.  Both of which had to bypass the set routes
on our bandwidth-splitting box (Router 3).  Basically what happens is this:

All of our internal machines are on the same 10/8 network.  All of the servers
are on 10.1.1/24.  There is one server for each domain that handles http/smtp/pop/ssh.
Each individual domain server (there are two) is set to use it's respective router
as the default gateway.  For example, domain1 is on ADSL.  And the server for domain1
is set to use Router 1 as it's default gateway.  domain2 is on cable, and the server
for domain2 is set to use Router 2 as it's default gateway.  ALL the rest of the computers
on our network (i.e. the ones that never have to worry about accepting incoming
connections, they ONLY have to worry about setting up outgoing ones) have Router 3 setup
as their default gateway.  Now as to the router setup itself.  This is actually
pretty damn simple.  Routers 1 & 2 are simple NAT'ting RDR'ing routers.  Incoming
connections are redirected to whatever the server for that domain is, and outgoing
connections are NAT'ed and sent on their way.  Router 3 has no NAT or RDR or PF
rules.  It doesn't need any.  All that it has is packet forwarding enabled and 
the static routes set up.

Everyone probably sees exactly how this works now.  The domain servers don't even see
the parts of the setup that don't concern them.  They only see their one gateway, and
don't have to worry about the rest of the connection crud.  Also, Router 1/2 only have
one route on them to the outside world, so the problem with incoming connections to one
of the domains is moot.  Router 3 does splitting and sends packets to either Router 1 or
2, which then NAT the packet and send it on it's way.  All of these machines are OpenBSD 3.1.

*whew* Well that's how things stand right now.  Actually right now I am thinking about
making an OpenBSD bootable-CD-firewall, so that we don't have to worry about bad
HDDs, which are the number one (well, only) cause of failure on our network right now.
But that's irrelevant.  If anyone has any suggestions on how I can drop one of the
routers and still keep everything working, I will be hugely happy, but I don't see
how it could work.  The only possibility that I can see is extensive use of PF's
new route-to settings, but I played with those and couldn't get them to do
what I wanted, so I'm happy leaving it this way now.  I fervently hope that
this helps someone down the road somewhere.

Also, note that if you ARE interested in setting up something like this (splitting
traffic between Cable and ADSL), it is possible that your setup will be easier than
mine.  You only have to have separate routers for external lines that you have domains
on (i.e. that you will be ACCEPTING connections) on.  Everything else can just go
into one router that splits the traffic itself.  Basically, if you have RDR statements
in /etc/nat.conf for an interface, then that interface better not be on a machine
that splits traffic using the routing table if you want it to work correctly.
So therefore, if you don't want to accept any incoming connections, you can stuff
all the external interfaces into a box that you want (other than two on the same
subnet, that gets confusing) and set them to route with static routes, and it shouldn't
be any problem.

Also, as to using this approach to increase browsing/overall speed.  Sorry, but not
going to help too much.  Note how I did the 'load-sharing' between the connections.
Static routes (that is to say, if packet is destined for w.x.y.z, then send it out
on THIS gateway) -- this does NOT dynamically shift load around.  You have to
monitor it yourself and set the routes so that things are sent more or less half
on each interface.  Basically, this is a kludge to get the cable company off of 
our back :)  With the proper tools (perl, tcpstat, a database maybe, etc), it would
be possible to set up some sort of 'profiling' tool that would record how much
is transferred to a particular external subnet (be that a class A, B, or C) and
then split them up automatically once a night or once a week based on that
information (redoing the routes on your splitting machine), but frankly
I don't do this for two reasons:

1) I like to keep track of the traffic flow graphs myself.
2) When a script 'automatically' resets the routes to some external subnets, you
   blow away all the active connections to the subnet.  So if you are
   downloading something and your script chooses that moment to change the route of
   that subnet, say bye-bye to your download.

Sorry for the length of this, and for the mistakes that I am sure
that I made in the typing of it. :)

   remi_(_at_)_cs_(_dot_)_ualberta_(_dot_)_ca -{ Remi Broemeling }- remi_(_at_)_penguinscratch_(_dot_)_com
"In theory, theory and practice are the same. In practice, they are
	 - Larry McVoy