Tunnelling Ethernet over IP for FreeBSD
This document is largely based upon the work done by Husni.
It only provides some enhancements regarding automatic start-up and some fine details.
This document explains the configurations and procedures to enable Ethernet over IP tunneling on FreeBSD. I succesfully performed the test on a FreeBSD 4.10-RELEASE. I make no claim that it will work on other releases. I also tested it on FreeBSD 4.7-RELEASE.
Objective
The objective of this set-up is to join two Ethernet segments S1 and S2 through the Internet and bridges, as if S1 and S2 were one single Ethernet segment.
The situation can arrise where both segments are too far away to be connected by a cable (or even optic fiber). Segments need not to be connected to Internet, though an Internet connection should be available at close distance.
The solution consist of encapsulating Ethernet frames into IP packets, those packets being transmitted through Internet and decapsulated at the other end. This kind of implementation is called tunnelling Ethernet over IP.
Tunnelling Ethernet over IP
In this setting, we use a software bridge that runs on FreeBSD.
Configuration Environment
Hardware- i386 (Pentium III)
- 32 MB RAM
- 2 NICs (RealTek, 3Com, etc.)
- generic kernel
- no IP forwarding
- kernel security low
- netgraph
- if_tap
- ng_ether
- ng_bridge
- ng_socket
- vtun (installed from the ports collection)
Installation
In the following part, we will use the following conventions:
- vtun is a client server protocol, we will call the server vtun-server and the client vtun-client
- each box has 2 network interfaces of type 3Com 905
- the interface xl0 is called the outside interface (0 is close to o-utside), it’s the interface that is connected to the Internet
- the interface xl1 is called the inside interface (1 looks like i-nside), it is the interface that connects to the ethernet segment we want to prolongate
- Prepare two FreeBSD boxes, vtun-server and vtun-client.
Assign an IP to the outside interface of the machines. Being the server, vtun-server will probably have a static IP, while vtun-client can do with a dynamic IP.
It is a good idea to install sudo from the security packages (from CD).
For those who like it, install emacs from the editor packages (from CD).
The file /etc/rc.conf will look like:
on vtun-server<br>inetd_enable="NO" <br>sendmail_enable="NONE" <br>sshd_enable="YES" <br>usbd_enable="NO" <br>#kern_securelevel="2" <br>kern_securelevel_enable="NO" <br>syslogd_flags="-ss" # Flags to syslogd (if enabled).<br>ifconfig_xl0="inet 203.159.32.5 netmask 255.255.255.248" <br>defaultrouter="203.159.32.3" # cs router <br>ifconfig_xl1="up" <br>hostname="vtun-server.xxx"
on vtun-client<br>sshd_enable="YES" <br>usbd_enable="NO" sendmail_enable="NONE" <br>inetd_enable="NO" <br>ifconfig_xl0="DHCP" <br>ifconfig_xl1="up" <br>hostname="vtun-client.xxx" <br>#kern_securelevel="2" <br>kern_securelevel_enable="NO" <br>syslogd_flags="-ss" # Flags to syslogd (if enabled).
Note that:- We disabled inetd, sendmail (totally), the USB ports
- We have syslogd running without any socket
- We bring up the inside interface
- Install vtun on both machines, from the ports collection.
cd /usr/ports/net/vtun
make && make install
To save some network bandwidth, it is a good practice to install vtun on one machine first, then to copy /usr/ports/distfiles from that machine to the other before making vtun. That way, the distributions will not be downloaded again from the Internet. - Edit vtun configuration on the machines.
Comment some lines of /usr/local/etc/vtund.conf in the lion session part. The differences would be like below:
The modifications of file /usr/local/etc/vtund.conf<br>vtun-server%diff vtund.conf vtund.conf.orig<br>304c304<br>&lt; # ifconfig "%% 10.1.0.1 netmask 255.255.255.0";<br>—&lt;br&gt;<br>&gt; ifconfig "%% 10.1.0.1 netmask 255.255.255.0";<br>307c307<br>&lt; # route "add -net 10.2.0.0 netmask 255.255.255.0 gw 10.1.0.2";<br>—<br>&gt; route "add -net 10.2.0.0 netmask 255.255.255.0 gw 10.1.0.2";<br>310c310<br>&lt; # firewall "-A forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ";<br>—<br>&gt; firewall "-A forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ";<br>321c321<br>&lt; # firewall "-D forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ";<br>—<br>&gt; firewall "-D forward -s 10.2.0.0/24 -d 0.0.0.0/0 -j MASQ"; On vtun-server modify line 423, change the IP address that will be used by the tunnel end to avoid duplication of IP address. The IP address should read 10.1.0.3
The modifications of file /usr/local/etc/vtund.conf on vtun-server<br>423c423<br>&lt; ifconfig "%% 10.1.0.3 netmask 255.255.255.0";<br>—&lt;br&gt;<br>&gt; ifconfig "%% 10.1.0.2 netmask 255.255.255.0"; - Configure the boot loader to automatically load the necessary modules at boot time.
Once the kernel has been set into a secure mode, some modules will not load anymore, so it is safe to load them since the boot time.
On both machines, edit /boot/loader.conf to look like:
The /boot/loader.conf file<br># — sysinstall generated deltas — #<br>userconfig_script_load="YES"<br>netgraph_load="YES"<br>if_tap_load="YES"<br>ng_ether_load="YES"<br>ng_bridge_load="YES"<br>ng_socket_load="YES" - Install the netgraph bridge on both machines.
cd /etc
cp /usr/share/examples/netgrap/ether.bridge .
chmod u+x ether.bridge
Then edit the file /etc/ether.bridge to include the lines:
The modifications of file /etc/ether.bridge<br>BRIDGE_IFACES="tap1 xl1"<br>LOCAL_IFACE="xl1" - Install the starter for /etc/ether.bridge on both machines.
The netgraph bridge can only be started when the interface tap1 is up. And the interface tap1 comes up only when a vtun connection has been established. To avoid manual starting of netgraph bridge the following script will test for the interface tap1 and then only start the bridge.
Copy that script in /etc/start-bridge, make it executable:
chmod u+x /etc/start-bridge
The script /etc/start-bridge<br>#!/bin/sh<br>res=`/sbin/ifconfig tap1 2&gt;&amp;1`;<br>while [ "$res" = "ifconfig: interface tap1 does not exist" ]; do<br># echo tap1 is down<br> res=`/sbin/ifconfig tap1 2&gt;&amp;1`;<br> sleep 1<br>done<br># echo $res<br>#echo tap1 is up<br>/etc/ether.bridge start<br>/sbin/sysctl kern.securelevel=2 Note that this script will also put the kernel in its higher security mode. It is valid because at that point the tunnel is up and working.
- Edit the /etc/rc.local on both machines.
On vtun-server it will start vtun in server mode and wait for connections, it will also initiate the netgraph bridge:
The file /etc/rc.local on vtun-server<br>/usr/local/sbin/vtund -f /usr/local/etc/vtund.conf -s<br>/etc/start-bridge &amp; On vtun-client it connects the the vtun server mode and initiates the netgraph bridge:
The file /etc/rc.local on vtun-client<br>/usr/local/sbin/vtund -p -f /usr/local/etc/vtund.conf lion 203.159.32.5<br>/etc/start-bridge &amp; Note we use the option -p so vtun client automatically reconnects to the server, whenever the server goes down or the connectivity is lost. This option allows to start the client before the server; the connection will be made as soon as the server is ready and visible.
203.159.32.5 is the IP address of vtun-server. - Reboot both machines.
Once all the cables are connected, the tunnel should be established automatically. - Make sure the tunnel is connected.
Some tests to be sure the tunnel is working<br>vtun-server&lt;on&gt;45: ifconfig tap1<br>tap1: flags=8943&lt;UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST&gt; mtu 1500<br> inet 10.1.0.2 netmask 0xffffff00 broadcast 10.1.0.255<br> inet6 fe80::2bd:85ff:fe02:1%tap1 prefixlen 64 scopeid 0x8 <br> ether 00:bd:85:02:00:01<br> Opened by PID 116<br><br>vtun-server&lt;on&gt;46: netstat -a |grep ESTABLISHED<br>tcp4 0 0 203.159.32.5.commplex- 203.159.31.7.2800 ESTABLISHED<br><br>vtun-server&lt;on&gt;47: grep promiscuous /var/log/messages<br>Aug 6 13:23:46 vtun-server /kernel: tap1: promiscuous mode enabled<br>Aug 6 13:23:46 vtun-server /kernel: xl1: promiscuous mode enabled<br><br>vtun-server&lt;on&gt;48: sysctl -a|grep secure<br>kern.securelevel: 2<br><br>vtun-server&lt;on&gt;49: ps auwx|grep vtun<br>root 116 0.0 2.3 2128 1376 ?? S&lt; 4:54PM 0:15.17 <br> vtund: lion ether tap1 (vtund)<br>root 97 0.0 1.9 2128 1096 ?? Is 4:54PM 0:00.00 <br> vtund: waiting for connections on port 5000 (vtund)<br> Note:
- The command ifconfig shows that the interface tap1 is up and running.
- The command netstat shows that there is an established vtun connection; vtun port is commplex-main or port 5000.
- A look at /var/log/messages shows that the interfaces xl1 and tap1 are in promiscuous mode, netgraph bridge has started.
- In a same way, the kernel has been changed to extreme security mode kern.securelevel=2. This is done only after netgraph bridge has started.
- Lastly, ps shows the vtun server waiting and one vtun connection ongoing. On the vtun-client machine, we will not see vtun waiting for connections.
- Check that packets are crossing the tunnel.
To do that, we attach one end of the tunnel to the segment it should tunnel and we check that we see some traffic at the other end of the tunnel.
In our example (see below) we attached the inside interface of vtun-server to the hub H1, then on vtun-client, we check that we receive packed on the interface xl1:
Check the tunnel with tcpdump on vtun-client<br>vtun-client&lt;on&gt;37: sudo tcpdump -i xl0<br>Password:<br>tcpdump: WARNING: xl0: no IPv4 address assigned<br>tcpdump: listening on xl1<br>13:02:29.936925 sfc-imlsdds.ai3.net.1580 &gt; 224.227.127.211.56952: udp 1408<br>13:02:29.964519 sfc-imlsdds.ai3.net.1580 &gt; 224.227.127.211.56952: udp 1408<br>13:02:29.982184 sfc-imlsdds.ai3.net.1580 &gt; 224.227.127.211.56952: udp 1408<br>13:02:30.013232 sfc-imlsdds.ai3.net.1580 &gt; 224.227.127.211.56952: udp 1408<br>13:02:30.028988 sfc-imlsdds.ai3.net.1580 &gt; 224.227.127.211.56952: udp 1408<br>13:02:30.053034 sfc-imlsdds.ai3.net.1580 &gt; 224.227.127.211.56952: udp 1408<br>13:02:30.078648 sfc-imlsdds.ai3.net.1580 &gt; 224.227.127.211.56952: udp 1408 - Setting up security.
Security was not considered in this framework, but IP firewall should be used to restrict access to the vtun server. Similarily, access to the vtun client should be restricted. Lastly, encryption shold be enabled on vtun tunnel.
Setting for the workshop
Tunnel topology for SOI Asia workshop
The tunnel is established between the hubs H1 and H2. H1 is directly connected to the UDLR receiver Sony Box, while H2 is in the classroom. The machine connected to H2 gets a dynamic IP address so it becomes the vtun client, while the machine connected to H1 gets a static IP and is configured as vtun-server.
Change since Frebruary 2003 setting is with the outside interface of vtun-server. It is not connected to AI3 subnet anymore, but directly to the border of AI3 and AIT AS’s. That way we save one hop (and one IP from AI3 subnet).
