[OpenBSDsupport]

OpenBSD 3.7 and PPPoE

A Beginner's Guide to xDSL Firewalling



First off, I'd like to warn everybody that this document should be considered obsolete!  In other words, it is no longer actively maintained, generally ignored, and otherwise neglected.  Why?  Because in version 3.8, an in-kernel PPPoE device was developed under the leadership of Can Acar, and quite simply, it rocks.  It even lets my 486 with ISA cards function successfully, whereas the userland PPPoE would bog down.  The man pages for the new kernel PPPoE are also very self-explanatory, making a document like this redundant.  What you see now is the last touch of the 3.7 guide, and is only here for historical reasons and the screwball user who, for some reason, requires use of the userland PPPoE tools.

So.. you've got yourself a nifty DSL subscription, but after having been p0wn3d by who knows how many script kiddies less than an hour after hooking things up, you've come to realize that a firewall might be a Good Idea(tm).  You thought about software solutions like ZoneAlarm, BlackIce, and the like, but by the time you've finished downloading them, it's already too late.  Browsing the local computer superstore, you were bewildered by the assortment of routers, switches, gateways, firewalls, and other gear, and saying the sales lackey wasn't helpful would be an unfair compliment.  Then you remembered that old pentium box hidden in the bottom drawer of a locked filing cabinet, sitting in a disused lavatory, with a sign on the door reading, "Beware the leopard!"(1)  Maybe that can be your salvation... if only you could get it working.  A little bit of googling, and here you are... the only maintained DSL/PPPoE HOWTO written for OpenBSD that I know of.  Unlike the others I have found, which were sadly out of date, this one is actually current!  (well, it was until 3.8 came out)  Why?  Because too many people keep asking the same dumb questions on the mailing lists, and rather than sit on my butt and whine about it, I'm going to sit on my butt and do something about it!  So without further ado, let's dig in.

Acknowledgements

First, I will acknowledge that my initial attempts would have been complete failures without those out-of-date guides, so if you'd like to take a look at them, here they are...
http://www.realo.ca/BSDinstall.html
http://www.aei.ca/~pmatulis/pub/obsd_pppoe.html
Just because they're somewhat dated doesn't mean they're useless... they just don't include all the nifty things that have come along since to make for better results.  Likewise, a huge tip of the hat to the folks on the misc@ mailing list.  They're the reason I never gave up hope, even when my credit card was only a walk up the stairs.

Background

For starters, read the OpenBSD FAQ.  If you go asking questions without reading this, you are guaranteed to get rude responses (if any at all).  Likewise, go through the PF User's Guide, and the ppp(8) man page.  Reading these now will help you understand what's going on when we build our ruleset.  Finally, for the sake of sanity (and because I don't have access to every last type of hardware on the planet, though I would love a Soekris), I will assume the following...

1.  You have an i386 computer that can handle the traffic.  I've tried in vain to get my 486/33 with ISA cards to work, but the overhead is just too high, even with 3Com 3C509B cards.  Stick to Pentium (or better) processors, and PCI network cards.  As long as it's residential DSL (or even most business T1's), you don't need the latest 4-way Opteron with 4G of RAM.  My 586/166, 96M system has never croaked, even with a roomie saturating the DSL line at the same time I'm pulling a tarball of /var. 
2.  You have a pair of decent PCI network cards to go along with it.  3Com cards are often given the highest marks (the drivers spit out "command not completed" errors which seem ominous, but are usually harmless), though nobody will complain if you use SMC or any of the other name brands.  You will, however, be gently chastised for using $5 el cheapo cards, which are usually based on RealTek 8139 chips.  Search Google if you'd like to know why.  Because of brain-damaged lawyers, Intel is in the doghouse despite making decent kit.  For the purposes of this document, I will assume your external interface is a 3Com, and your internal one of those cheapies.  They are named xl0 and rl0 respectively.  If you're not sure what yours are called, consult the dmesg that appears every time you boot the system.  Likewise, if you have two SMC cards (named we0 and we1), watch the dmesg closely... you'll be able to identify which is which by the MAC address, which is included in the boot probe, and is silkscreened or stickered to the card.
3.  You are the prowd owner of a real DSL modem... that is, one that has absolutely *ZERO* intelligence.  No fancy web interfaces, no cryptic telnet backdoors, no built-in firewall, etc.  It's got one RJ-11 plug, one RJ-45 plug, a wall adapter, and maybe a reset/power button.  There are more "intelligent" modems out there than you can use (and they do work), but they nearly always require some sort of additional configuration.  You might run into them in a small business setting, but getting them running is beyond the scope of this HOWTO, as they often contain a built-in firewall anyways.  I've got a nice SpeedStream 5260, and no, you can't have it!
4.  You have successfully installed the OS (ordering a CD set is easiest, since network installation expects you to have either a static IP or DHCP server handy, neither of which is available when you're working with DSL), read the afterboot man page, and are patiently waiting for guidance.

Getting Started

Ahh.. you're finally sitting there with a brand new install, and that 'login:' prompt is waiting for your happy fingers.  Let's go.  Log in as root (2), and notice how it tells you that logging in as root is a Bad Thing(tm).  We won't worry about that for now.  Read Theo's email (the majority of which is just a listing of the packages available), then make yourself a normal user account using the 'adduser' command.  When it asks you whether or not to invite this user to any extra groups, add yourself to the 'wheel' group.  This will allow you to use the 'su' command to switch from user to root while in a normal login.  Once you've created a user account for yourself, log out of root, then in as yourself, and elevate back to root with the su command.  This may seem like a senseless extra action, but it's a very good UNIX habit to practice, considering that as root, you can literally destroy the entire system, and it won't so much as ask, "Are you sure?"  As a normal user, the damage you can do is pretty much limited to fragging your email folders and web page.  Now, you'll need to make a chage to /etc/sysctl.conf in order to enable packet forwarding.  Specifically, you need to uncomment the line that reads:

#net.inet.ip.forwarding=1

by deleting the # character.  Next, create /etc/ppp/ppp.conf as follows:

default:

 set log Phase Chat IPCP CCP tun command
 set redial 15 0
 set reconnect 15 10000

pppoe:
 set device "!/usr/sbin/pppoe -i xl0"
 disable acfcomp protocomp
 deny acfcomp
 set mtu max 1454
 set speed sync
 enable lqr
 set lqrperiod 5
 set cd 5
 set dial
 set login
 set timeout 0
 set authname xxxxxxx
 set authkey xxxxxx
 add! default HISADDR
 enable dns
 enable mssfixup

Doing that creates...
1. A default setting for all PPP connections is created for logging, initial dialing, and reconnecting.
2. A connection named 'pppoe' is defined which uses the pppoe driver on the xl0 interface.  If your external interface is something else, replace "xl0" with whatever your card is.  Some protocols are disabled, connection characteristics are defined, as are your username and password.  Finally, your ISP's computer is defined as the gateway, and a nifty little feature that lets the firewall interpret strange MTUs on behalf of Windows machines is enabled.  Of particular interest is the MTU setting and LQR.  A value for MTU set too low will result in a non-optimal connection, while one too high will result in *very* strange behavior... webpages coming back with non-sensical errors, establishing connections that never actually work, etc.  1492 is the highest you can ever safely get with xDSL (the 1500 typical of ethernet, minus 8 bytes for the pppoe wrapping), so if you're having trouble, this is the first place to look.  1454, however, is a much tastier option.  If that still doesn't work, continue to reduce this number until it does.  LQR is highly dependent on your service provider.  For some people it's required, for others it's a death warrant.  Some folks experience no difference at all.  The story here, experiment... your ISP probably has a different setup than mine anyways.

ISP/Country Specific Notes:

Bell Canada - According to Damian Gerow, Bell Canada has been phasing out Redback's in favour of ERXes. These do not (and are not planned to) support LQR. Enabling LQR (at least on FreeBSD) caused a disconnect every three seconds or so, as the line would fail the LQR test. This *must* be disabled in order for the line to remain stable.  If your ISP has an FE crossover to Bell Canada (or whichever DSL supplier you've got), the optimal MTU is 1448. This is because of all the tunnelling and packet overhead that's created -- anything greater than 1448 (actually, it's 1452, but that's on on the 8-byte boundary) will cause some sort of fragmentation in the packet between you and your ISP. In periods of heavy traffic, this /may/ result in a slowdown in your traffic, though it would probably only be visible to those paying close attention to speed tests.

France - Benjamin Pineau sends his observation that all the newer DSLAMS in France are also incompatible with LQR.

Continuing onwards...

Now, let's tell /etc/netstart what is going on.  To do this, we need to edit the two /etc/hostname.if files.  Since xl0 is our outward facing card, /etc/hostname.xl0 will have the following in it....

up

That's it.  Just the word 'up'.  This tells the script to bring the interface up, but nothing more.  Something else will be responsible for configuring it.  However, our inward facing card, rl0, is a different story.  Its configuration file, /etc/hostname.rl0, reads thus:

inet 192.168.0.1 255.255.255.0 NONE


This tells netstart to configure rl0 for internet traffic with the IP address 192.168.0.1, broadcast address 255.255.255.0, with no extra media options.(3)  This is a good place to make sure nothing's been broken, so reboot with:

shutdown -r now

While the system reboots, this is a good time to check all your wires, and make sure what needs to be turned on, is.  Everything booted up ok?  No strange errors?  Good.  There shouldn't be, because so far, all you've done is lay the groundwork for what's next.  Log in again, and su yourself to root.  Then issue:

ppp -ddial pppoe

and watch the fun begin.  You will almost certainly see console messages complaining about carrier settings and change route failures.  Don't sweat it.  Wait maybe 10 seconds or so,  then type

ifconfig -a

and see what goes by.  Towards the very bottom should be a line that reads similar to

tun0: flags=8011<UP,POINTOPOINT,MULTICAST> mtu 1492

        inet 68.76.82.109 --> 68.76.83.254 netmask 0xffffffff

If there is, life is good.  This means your computer has successfully logged on, and you are connected to the internet.  You can surf the web with lynx, telnet, rlogin, or SSH into remote machines, ping people, and basically do anything the default install packages allow you to do.  You don't have a firewall yet, so hanging around like this isn't a good idea, but on the bright side, this ain't Windows, either.  If that line isn't there, you've either messed up somewhere, or the login process is just taking a long time.  If a minute goes by and you're still not connected, check your setup.  Did you use quotes around your username or password?  If so, remove them.  Your service provider may only want your username, unlike mine, which wants user@domain.com.  Once you're happy that things work, disconnect by using

pkill ppp

Building the Firewall

Now on to the fun part.  At this point, you should have read the PF User's Guide I mentioned higher up.  If you haven't, you're a bad, bad user.  Naughty, in fact.  Go do that now, because I'm going to be using some of the 'advanced' features.

All read up?  That's better.  Most of this will look very similar to their example ruleset.

# Define interfaces and rooms
int_if = "rl0"
ext_if = "tun0"
north = "192.168.0.15"
south = "192.168.0.10"
center = "192.168.0.20"

Just defining macros to make life easier later.  Since three people live here, two of whom share the same name, it's easier to just name the rooms by their location.

# RFC1918
priv_nets = "{ 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 }"

Same idea, this just saves a lot of typing later.

# Those wonderful scrubbing bubbles
scrub in all

Let's wash the packets of various deformities, mutations, and other nastiness.

# Queueing
# TCP/ACK frames get first dibs, followed by webserver, DNS lookups, and
# the unwashed masses.
altq on $ext_if priq bandwidth 256Kb queue { std_out, web_req, dns_out, web_server, tcp_ack_out }
queue std_out priq(default)
queue web_req priority 3
queue dns_out priority 4
queue web_server priority 5
queue tcp_ack_out priority 6

# The downstream is 1900kb, Everyone gets 600Kb regardless,
# and can get more when nobody else wants theirs.
altq on $int_if cbq bandwidth 100Mb queue { trash, other, north, center, south }
queue trash cbq(default)
queue other bandwidth 6Kb cbq(borrow)
queue north bandwidth 600Kb cbq(borrow)
queue center bandwidth 600Kb cbq(borrow)
queue south bandwidth 600Kb cbq(borrow)

Ahh... queueing.  My line is advertised as being 1500/256(4), but observation has shown it's more like 1900/300.  Whatever.  Outbound, priority queues see to it that empty ACK packets hit the wire first, making the net "feel" faster.  This is followed by the web server, DNS requests, web surfing, and finally general junk traffic (like p2p).  Nothing is worse than one person whining about not being able to websurf because another's getting "Debby does Dresden 22" on Kazaa.  On the inside, each person (North, Center, and South) is guaranteed 1/3 of the total bandwidth, and can borrow any time there's leftovers.  People who come to visit and hook their systems up via DHCP will be placed in the 'other' queue (they don't pay the bills, so why should they get phat pipe?), while packets that don't go to any machine in particular (ARP broadcasts, DHCP requests, etc) end up in the 'trash' queue.(5)

# NAT/RDR directives
nat on $ext_if from $int_if:network to any -> ($ext_if:0) static-port
rdr pass on $int_if proto tcp from any to any port 21 -> 127.0.0.1 port 8021
rdr pass on $ext_if proto { tcp, udp } from any to any port 9980:9989 -> $south port 9980:*
rdr pass on $ext_if proto tcp from any to any port 9990:9999 -> $center port 9990:*

Another important section... Unlike the example, I'm using the ':0' and 'static-port' keywords.  This allows machines on the inside to connect with VPNs elsewhere much more easily than trying to hack together a bunch of special-case rules.  We send all our internal FTP traffic to a proxy on this machine, and create a bunch of holes for various programs to work with.  IM programs, p2p applications, and the like all need a port to transfer files over, and we take care of this here.  Notice the port nnnn:nnnn -> $target port nnnn:* construct... sure beats having to type out 20 rules all at once, doesn't it? :)  The pass keyword here allows these ports to flow through without needed extra rules later.  Also, remember that unless the machines these ports point to are protected with a software firewall, they'll be directly exposed to the big nasty internet through those ports.  If you don't need the holes, don't create them.

# Filtering begins
block drop log all

# Local machine stuff
pass quick on lo0 all

# Clean invalid SRC/DST packets
block in  quick on $ext_if from $priv_nets to any
block out quick on $ext_if from any to $priv_nets

First, we set up a default deny/drop policy to handle packets, and log them so that we can see them with tcpdump+pflogd.  This helps immensely with troubleshooting.  Traffic generated on the local machine goes through immediately (Yes, I'm talking to myself, and answering myself, and I have a feeling that without this rule, the entire PF system could come crashing down around me, but I'm too chicken to try ::grin::), while packets either claiming to come from a private network, or destined for one, are summarily executed.

# Pass in allowed servers
# pass in on $ext_if proto tcp from any to ($ext_if) port ssh flags S/SA keep state
pass in on $ext_if proto tcp from any to ($ext_if) port 80 flags S/SA keep state queue web_server

If you're running any sort of server, this is a good place to allow that traffic.  Notice that SSH has been disabled... it's great when you're setting something up, but once it's in, running, and finished, there's no need to leave it accessable (or even running).

# Pass in FTP proxy
pass in on $ext_if inet proto tcp from any to $ext_if user proxy keep state

Here's the back-connection needed to make ftp-proxy work.

# Out to the 'net
pass out on $ext_if from ($ext_if) to any flags S/SA modulate state queue(std_out, tcp_ack_out)
pass out on $ext_if proto tcp from ($ext_if) to any port 80 modulate state queue (web_req, tcp_ack_out)
pass out on $ext_if proto { udp icmp } from ($ext_if) to any keep state queue std_out
pass out on $ext_if proto { tcp udp } from ($ext_if) to any port domain modulate state queue dns_out
pass out on $ext_if proto tcp from ($ext_if) to any user www modulate state queue web_server

# Internal queueing
pass in on $int_if from $int_if:network to any
pass out quick on $int_if from any to $north keep state queue north
pass out quick on $int_if from any to $south keep state queue south
pass out quick on $int_if from any to $center keep state queue center
pass out on $int_if from any to $int_if:network keep state queue other

Finally, we send traffic through the various queues.  By using the 'quick' keyword on the internal side, packets for residents can be shuffled into their guaranteed pipes without falling into the visitor's 'other' queue.  If you decide not to queue the internal interface, use the first line as-is, and the fifth line replacing "queue other" with "keep state".  Get rid of the rest.

Now, let's actually enable the ftp-proxy program itself.  This is done in /etc/inetd.conf by uncommenting the line that reads:

127.0.0.1:8021 stream tcp       nowait  root    /usr/libexec/ftp-proxy ftp-proxy

Think you're ready to put this online?  Well, you are.  Sort of.  At this moment, you could reconnect to the internet, turn on the firewall, and have a protected system.  But you likely wouldn't be routing packets.  Why?  Because you still have to change your client machine's IP addresses.  If you choose to go the static route (as with my 'guaranteed bandwidth' machines), you can feed it the appropriate information and be on your way.  But if you're not using internal queuing, or expect visitors as I do, setting up DHCPd is the way to go, so let's do that now.

DHCPd....

isn't that hard.  Only two files need alteration... /etc/dhcpd.interfaces, and /etc/dhcpd.conf.  Both are self explanatory, but just in case...  In /etc/dhcpd.interfaces, much like /etc/hostname.rl0, you need only a single item... the name of your internal interface (rl0 in my case).  /etc/dhcpd.conf is a little more involved.  Again, here's mine...

shared-network LOCAL-NET {
        option  domain-name "domain.com";
        option  domain-name-servers 217.115.138.24, 68.15.165.12;

        subnet 192.168.0.0 netmask 255.255.255.0 {
                option routers 192.168.0.1;

                range 192.168.0.32 192.168.0.127;
        }
}
This defines what domain name you're going to pass along to your clients, what name servers are available (I like the OpenNIC tier 2 servers, since they support legacy TLD's as well as a number of independent operators), and basic network configuration parameters needed to make things work.

Testing... Testing... 1... 2... 3...

With all this in place, it's time for your second reboot.

Back already?  Good.  At this point, your OpenBSD box is nearly ready for months of uptime.  Log in, and connect to the internet.  Once you're on, issue the following...

pfctl -e -f /etc/pf.conf


You've just enabled your firewall.  Wander over to a machine on your private network, and try to surf the net.  If it works, congratulations.  You're almost done!  If not, check its IP address.  Is it something that makes sense?  If it's not in that private address block, release it,  then attempt to renew it with DHCP.  Once you get a valid internal address, you should be good to go. 

Final Touches

Once you've confirmed that the internal network can get out, head back to the firewall, and edit /etc/rc.local by adding the lines

# Start PPPoE ahead of ntpd
echo -n ' PPPoE'
ppp -ddial pppoe
sleep 20

just after "echo -n 'starting local daemons'".  This initiates your internet connection as part of the computer's boot-up process, and gives it time to complete handshaking and settling in before continuing.  20 seconds might be excessive, so tweak to fit your circumstances.  Next, create /etc/ppp/ppp.linkup, and in it place

MYADDR:

 !bg sh -c "/sbin/pfctl -e -f /etc/pf.conf"

That will automatically start your firewall upon establishing a connection.  Likewise, should you get disconnected, you want the firewall cleared.  Do this by creating  /etc/ppp/ppp.linkdown, and in it placing

MYADDR:
 !bg /sbin/pfctl -d

Finally, it's nice to clean up after yourself if you have to shut the machine off or reboot it for some reason.  In /etc/rc.shutdown, turn off ppp by including the lines:

# kill the ppp connection to avoid connection timeout at startup
kill `ps -axc | grep -w ppp | awk '{print$1}'`
Or try as well: pkill -x ppp

sleep 5

At this point, your OpenBSD box will automatically log on to the internet when it boots, bring up the firewall when the connection is established, shut it back off should you get disconnected, and by using the -ddial option, will automatically reconnect if (or, more likely, when) that happens.  You have a DHCP server facing inwards, letting anybody get an IP address and surf safely from within your network.  That's it.  You're done.  At this point, you could just as well take the keyboard and screen off, and probably never worry about this machine again.  It would be a good idea to keep up with security/reliability announcements, but unless you also run WWW, FTP, or other servers on this machine, odds are you won't have to worry about it until a hard drive or cooling fan dies.

Notes and diversions.....

(1) Apologies to the late Douglas Adams, but let's face it... where some of our old computer equipment comes from pretty much equates to that!
(2) I hear the howls of the misc@ crowd already...
(3) What IP address you actually give this interface is up to you... to a certain extent.  You could call it 66.35.250.150, but then you'd be conflicting with slashdot.org, which really isn't a good idea if you like your nerd news.  For that reason, RFC1918 defines a couple of network blocks specifically for private use... 10.xxx.xxx.xxx, 172.16.xxx.xxx, and 192.168.xxx.xxx.  Some people like starting with 192.168.1.1.  I'm not one of them.
(4) That is, 1500kbit downloads, 256k uploads
(5) You might be looking at that and saying, "Wait a minute... if this guy's only got a 1.5Mb downstream connection, why's he using the full interface speed???"  Two reasons...  First, telling it the root queue is that large allows me to pull /var/www backups a LOT faster.  Second, in my experience, pf appears to make sure everybody gets their first 600k before handing out the borrows.  With a little bit more effort, it would be possible to define a DSL queue within the root queue, but the filtering/tagging rules would become quite cumbersome.  This could be a feature/bug that changes behavior in future revisions, so watch out!
(6) This is no longer a work in progress.  People besides me (Chris Zakelj) who put their fingerprints on it at some point include Eddy Carroll, Robby Cauwerts, Damian Gerow, Benjamin Pineau, and Stephan Tesch.  Daniel Ouellet provides the hosting at http://www.openbsdsupport.org.  The original was written the evening of 3 Sept 04.  The final update was 11 Jan 06.