Altough our first field test went well, the network infrastructure has still room for improvements. This article explains some ideas for integrating a Digital Subscriber Line (DSL) based network into field tracks and their evaluation.
Learning from the initial field test
The first test happened on a parking lot. Three mobile treatment center were raised by different units. Participants were tracked during triage and therapy.
To provide wlan at the site, four UniFi AC Mesh AP units were installed. A local controller was used for provisioning and monitoring. Using different channels, the wlan provided more than enough capacity for all JellingStone devices. All units were hooked up to the grid using power-over-ethernet (POE). Cat5 cables were used to create a wired, switched backbone. The deployment went smooth and took about 30 minutes. Still, some shortcomings occurred:
- Not having power cables or batteries in the beginning, the build-up started two hours later than planned initially. By intuition, it appears reasonable to have batteries - but using battery power imposes power constraints on the devices in use: Overprovisioning using powerful devices becomes less attractive.
- Using mesh-links instead of Cat5-cables was not attractive. For power distribution, all AP were hooked up by cables nevertheless and Cat5 is easier to handle than H07RNF-3G2.5. In this non-reflecting environment, mesh-links would have required line-of-sight in addition.
Learning from this event, we decided to extent the network regarding to the devices in use:
- Create a power system using local batteries to exploit mesh links.
- Introduce smaller boxes, requiring less power.
- Potentially use field wire (still common among relief organizations in Germany) for non-line-of-sight situations. This could improve handling and allow larger communication range compared with Cat5.
What do we actually need?
On the one hand, each JellingStone sensor roughly transmits up 10 KByte every 5 seconds - i.e. to 2 KByte/s. Thus 200 devices need 400 Kbyte/s, requiring 3.6 MBit/s approximately. That’s about the uplink capacity of an ADSL2+ Annex M link. Using compression and avoiding fragmentation, we hope reduce the traffic to 10% in the future.
On the other hand, lines of field wire are usually deployed and used with telephones, opening the case for Voice-over-IP (VoIP).
In general, field exercises may take up to few days - batteries should should last for 8-12 hours at least. Replacing batteries 2-3 times per day is fine.
Hardware in use
To validate our idea, we decided to do some benchmarking based on old ADSL hardware and got an old Zyxel IES‑1248‑51A for testing (~ 120 €). It is able to connect up to 48 ADSL2+ boxes.
Being somewhat stingy, we looked for cheap ADSL2+ boxes. Two boxes draw our attention:
Both boxes are available for ~1 € (used; 5 € including shipping) and supported by OpenWRT in principle. But it is a tough choice:
- The ARV752DPW22 consumes three times the power of the 7312. While the 7312 uses 5V/1.6A and could be powered using an USB powerbank, the ARV752DPW22 needs 15V / 1.6A, practically resulting in a lead-battery having a step-up converter.
- On the 7312, the telephone connectors can only be used with AVM’s stock firmware. But when using stock firmware, the wireless cannot do anything interesting (802.11s, P2P, IBSS, etc.).
To keep things simple in the beginning, we went on testing the 7312 using stock firmware and OpenWRT at our hackathon in November.
Network-architecture, roaming and the case for an overlay network
In general, wireless stations should be able to roam within the network while maintaining IP-connectivity. In a very simple form, this can be realized by creating a Layer-2 / ethernet segment bridging all wireless access points.
In principle, this could be realized by running Ethernet-over-ATM (EoA) and bridge the corresponding interface to wireless interfaces running in AP mode. But haven run a similar configuration with Thomson TG605s back-to-back before, we noticed that some DSL devices are not used to update mac address tables when stations roam through the network - the entries turned out to be quite persistent. This is completely fine for PPPoE setups with no mobility at all and I guess, that many boxes were never tested to update any address due to a roaming device.
Another point is, that AVM does not expose the EoA interface to the kernel. Thus it is impossible to bridge it with the wireless lan interface using the linux command line.
Both points can be addressed by bridging an overlay network (VPN without encryption) instead of the EoA interface. Being limited to technologies that are part of Freetz / Fritz OS 6, the two most interesting candidates are:
Stateless L2TPv3 pseudowires are simple point-to-point links. L2TPv3 is implemented in the kernel and can be configured using
ip l2tp for each link individually. Tinc on the other hand is a decentralized and simple VPN program, that can be used like a distributed switch.
Compared with L2TPv3, tinc is running in userland and potentially slow. But it supports fragmentation, path-mtu & peer discovery and passing traffic between peers directly. L2TPv3’s mtu is bound by the underlying link - IPv4 fragmentation is possible, though. In other words: Tinc might be slow, while L2TPv3’s achilles heel is costlier configuration. This raises two questions:
- What is Tinc’s TCP-Goodput compared to L2TP? Is an AVM 7312 running tinc able to switch at 3.5 MBit/s at least?
- Is it possible to create a L2TPv3 pseudowire on AVM 7312 devices? What is the actual effort for configuring and maintaining all L2TPv3 links?
For testing, we decided to give both protocols a try using Freetz - we also tested installing and using OpenWRT.
Working on top of AVM Fritz OS, Freetz enables SSH-access and command line administration, needed to configure l2tp-links. As of today, both Fritz OS 6.33 and Fritz OS 6.55 can be used in combination with freetz’es master branch. In general, build-instructions for freetz are availble at freetz.github.io - keep in mind, that the project is hosted on github nowadays.
/sbin/l2tp in 6.55 initially, 6.30 is used - tinkering with 6.55 is not on the agenda, yet. Tinc on the other hand is included in our freetz build-.config - tun devices drives are included in 6.33 - no futher kernel modules are needed in addition. As of today, running Freetz / FritzOS 6.33 results in linux 2.6.32 and tinc 1.0.35.
After cloning the master branch and entering the menu (
make menuconfig), the tinc package is included in addition. Executing
make creates the image - (for example: 7312_06.30-freetz-HEAD-20191017-828c7d609-dirty.de_20191107-121917.image) Extract and flash:
tar xvf images/7312_06.30-freetz-HEAD-20191017-828c7d609-dirty.de_20191107-121917.image cd var/tmp # change folder ftp 192.168.178.1 # 3-5 seonds after power on; during boot
The Zyxel IES-1248-51A requires the Fritzbox to operate using Annex A. This is done by setting
Transcript of the FTP session. Prompt is
ftp> - username / password: adam2
Connected to 192.168.178.1. 220 ADAM2 FTP Server ready Name (192.168.178.1:jan): adam2 331 Password required for adam2 Password: 230 User adam2 successfully logged in Remote system type is AVM. ftp> quote SETENV kernel_args annex=A 200 SETENV command successful ftp> binary 200 Type set to BINARY ftp> passive Passive mode on. ftp> quote MEDIA FLSH 200 Media set to MEDIA_FLASH ftp> put kernel.image mtd1 local: kernel.image remote: mtd1 227 Entering Passive Mode (192,168,178,1,12,8) 150 Opening BINARY data connection 226 Transfer complete 13660936 bytes sent in 29.32 secs (454.9759 kB/s) ftp> quote REBOOT 221 Thank you for using the FTP service on ADAM2 ftp> quit 221 Goodbye.
Benchmarking Tinc & L2TPv3
For testing, we connected a Lenovo T580 laptop to the AVM 7312 using a 100 MBit/s wired ethernet connection. To avoid limits imposed by the DSL line, the ethernet port (“LAN”) is configured for WAN access - the overlay network is working on top of the wired connection. Bridging the overlay network to the infrastructure wlan on the 7312, the corresponding interface on the laptop becomes effectively part of the LAN of the 7312. At last, another laptop (Lenovo T490) providing an iperf3 service is connected to the infrastructure wlan hosted on the 7312. The network is illustrated in Figure 1:
Including tinc in the Freetz build before, it is possible to setup tinc using the Freetz GUI at Port 81/tcp. But due to a missing inode in the
/dev-filesystem, tinc is unable start. This can be fixed by SSH’ing to the box and creating the inode manually. Doing only preliminary benchmarking, we have not tried creating a reboot-save configuration, yet.
mknod /dev/tun c 10 200
All configuration files are available at https://github.com/FieldTracks/benchmark/tree/master/tinc. The laptop is configured using linux command tools in the distribution (Manjaro Linux 18.1.2, Kernel 4.19.81, Tinc 1.0.35).
$ iperf3 -c 192.168.188.21 Connecting to host 192.168.188.21, port 5201 [ 5] local 192.168.188.200 port 56890 connected to 192.168.188.21 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 864 KBytes 7.08 Mbits/sec 0 73.0 KBytes [ 5] 1.00-2.00 sec 776 KBytes 6.36 Mbits/sec 0 107 KBytes [ 5] 2.00-3.00 sec 545 KBytes 4.47 Mbits/sec 3 89.3 KBytes [ 5] 3.00-4.00 sec 763 KBytes 6.25 Mbits/sec 0 104 KBytes [ 5] 4.00-5.00 sec 592 KBytes 4.85 Mbits/sec 3 78.4 KBytes [ 5] 5.00-6.00 sec 763 KBytes 6.25 Mbits/sec 0 89.3 KBytes [ 5] 6.00-7.00 sec 699 KBytes 5.73 Mbits/sec 3 83.9 KBytes [ 5] 7.00-8.00 sec 699 KBytes 5.73 Mbits/sec 0 78.4 KBytes [ 5] 8.00-9.00 sec 763 KBytes 6.25 Mbits/sec 0 89.3 KBytes [ 5] 9.00-10.00 sec 636 KBytes 5.21 Mbits/sec 0 92.0 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 6.93 MBytes 5.82 Mbits/sec 9 sender [ 5] 0.00-10.15 sec 6.77 MBytes 5.60 Mbits/sec receiver iperf Done.
Looking at the results, the bitrate rate looks volatile, but clearly above 3.5 MBit/s all the time. Assuming, that using DSL instead of ethernet poses no additional CPU load on the 7312, it appears reasonable to assume, that tinc is able to fill the ADSL2+ Annex M uplink completely. For the downlink the situation is different: Being able to serve 27 MBit/s, tinc poses a relevant restriction.
As a result, tinc can potentially be used to transport sensor data generated by JellingStone - but additional IP-traffic (web browsing, etc.) can only be handled to a very limit extend. Tinc’s performance is good enough but has room for improvements.
Unlike tinc, L2TPv3 cannot be configured using the Freetz WebUI. Setting up L2TPv3 requires:
- Creating a virtual interface having the same IPv4 address as used on the WAN-Interface. This is needed because the kernel appears to be unaware of the network configuration done in the driver-blobs.
- Creating a L2TPv3-tunnel according to the IPv4 addresses used in the WAN network.
- Creating a L2TPv3 session on top of the tunnel.
- Adding the corresponding l2tpeth pseudo-device to the existing LAN-bridge
$ iperf3 -c 192.168.188.21 Connecting to host 192.168.188.21, port 5201 [ 5] local 192.168.188.200 port 50020 connected to 192.168.188.21 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 9.22 MBytes 77.3 Mbits/sec 22 451 KBytes [ 5] 1.00-2.00 sec 9.28 MBytes 77.8 Mbits/sec 0 552 KBytes [ 5] 2.00-3.00 sec 8.29 MBytes 69.6 Mbits/sec 0 609 KBytes [ 5] 3.00-4.00 sec 8.17 MBytes 68.5 Mbits/sec 4 453 KBytes [ 5] 4.00-5.00 sec 8.48 MBytes 71.1 Mbits/sec 0 493 KBytes [ 5] 5.00-6.00 sec 8.42 MBytes 70.6 Mbits/sec 0 519 KBytes [ 5] 6.00-7.00 sec 8.29 MBytes 69.6 Mbits/sec 0 533 KBytes [ 5] 7.00-8.00 sec 8.97 MBytes 75.2 Mbits/sec 0 540 KBytes [ 5] 8.00-9.00 sec 8.85 MBytes 74.2 Mbits/sec 0 541 KBytes [ 5] 9.00-10.00 sec 9.03 MBytes 75.8 Mbits/sec 0 541 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 87.0 MBytes 73.0 Mbits/sec 26 sender [ 5] 0.00-10.10 sec 85.4 MBytes 70.9 Mbits/sec receiver iperf Done.
Again, the bitrate looks volatile and is around 70 MBit/s, potentially hitting the capacity of the 2.4 GHz network - we have a rather noisy environment in our local hackerspace.
As a result, the higher bitrate (~70 MBit/s) makes L2TPv3 by far more attractive than tinc (~ 5 MBit/s) .
The AVM 7312 is supported by OpenWRT since version 19.07, which is not released yet. 19.07-rc1 is used for testing.
Unfortunately, the needed DSL driver is not configured in OpenWRT stock builds. By default, it includes ITU 992.1/.3/.5 Annex B (ISDN blanking period, Germany), only. However, the IES-1248-51A needs Annex A (POTS blanking period, rest of the world).
In principle, OpenWRT provides an Annex A firmware, that was made available directly by lantiq a few years ago. Newer versions can only be downloaded using vendor firmware bundles. Thus they cannot be provided by OpenWRT due to legal reasons.
Providing version 18.104.22.168.0.1, OpenWRT’s firmware is rather old and causes stability issues on the 7312. Thus we went on by copying
ar9-A-dsl.bin from a freetz box into the OpenWRT file system and updating the
Flashing can be done using fritz-tools - transcript:
wget https://raw.githubusercontent.com/freifunk-darmstadt/fritz-tools/master/fritzflash.py # Downloading log ommited python3 fritzflash.py --image path/to/openwrt-lantiq-xway-avm_fritz7312-squashfs-sysupgrade.bin
Executing fritzflash.py yields an interactive terminal program guiding through the flash process. Log:
This program will help you installing Gluon, a widely used Firmware for Freifunk networks, onto your AVM device. You can always find the most current version of this script at https://www.github.com/freifunk-darmstadt/fritz-tools It is strongly recommended to only connect your computer to the device you want to flash. Disable all other connections (Ethernet, WiFi/WLAN)! Before we start, make sure you have assigned your PC a static IP Address in the Subnet of the device you want to flash. The following example would be a completely fine option: IP-Address: 192.168.178.2 Subnet: 255.255.255.0 Gateway: 192.168.178.1 DNS Servers: Leave blank Once you're done, press enter, disconnect power from your AVM device and reconnect the power-supply. Trying to autodiscover! Abort via Ctrl-c. FritzBox found at 192.168.178.1 Autodiscovery succesful! -> Device detected at 192.168.178.1. -> Establishing connection to device! --> Try 1 of 10 -> Flash image Writing Gluon image to your AVM device... This process may take a lot of time. First, the device will erase its current Operating System. Next, the device will write the Gluon image to its memory. The red Info LED will illuminate in this step. Don't worry, this is expected behavior. Do *not* turn of the device! We will tell you when your device has finished installing Gluon (this may take a while). -> Image write successful -> Performing reboot == Congratulations! == Your device is now running Gluon. It will restart and in 2-5 minutes you will be able to visit its config-mode. Remember to reconfigure your interface to automatically obtain an IP-address! You can reach config-mode by typing in http://192.168.1.1/ in your preferred Webbrowser. Press any key to exit.
After a while, the box is reachable using ssh at
192.168.1.1. To establish a DSL connection we changed the network configuration according to our DSL setup using SSH
uci set network.dsl.annex='a' uci set network.atm.vpi='0' uci set network.atm.vci='33' uci set network.wan.proto='dhcp' uci del network.wan.username uci del network.wan.password uci commit network reboot
After rebooting and giving the DSL-sync some time,
/etc/init.d/dsl_control status shows the configuration in use.
/etc/init.d/dsl_control status ATU-C Vendor ID: ATU-C System Vendor ID: Chipset: Ifx-AR9 Firmware Version: 22.214.171.124.0.1 API Version: 126.96.36.199 XTSE Capabilities: 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0 Annex: A Line Mode: G.992.5 (ADSL2+) Profile: Line State: UP [0x801: showtime_tc_sync] Forward Error Correction Seconds (FECS): Near: 0 / Far: 0 Errored seconds (ES): Near: 4 / Far: 1 Severely Errored Seconds (SES): Near: 1 / Far: 1 Loss of Signal Seconds (LOSS): Near: 0 / Far: 1 Unavailable Seconds (UAS): Near: 29 / Far: 29 Header Error Code Errors (HEC): Near: 1102 / Far: 16 Non Pre-emtive CRC errors (CRC_P): Near: / Far: Pre-emtive CRC errors (CRCP_P): Near: / Far: Power Management Mode: L0 - Synchronized Latency [Interleave Delay]: 0.25 ms [Fast] 0.50 ms [Fast] Data Rate: Down: 27.446 Mb/s / Up: 1.291 Mb/s Line Attenuation (LATN): Down: 0.0 dB / Up: 6.2 dB Signal Attenuation (SATN): Down: 0.0 dB / Up: 0.0 dB Noise Margin (SNR): Down: 6.0 dB / Up: 6.2 dB Aggregate Transmit Power (ACTATP): Down: 8.1 dB / Up: 6.1 dB Max. Attainable Data Rate (ATTNDR): Down: 27.352 Mb/s / Up: 1.335 Mb/s Line Uptime Seconds: 11 Line Uptime: 11s
As a result, OpenWRT is able to create a stable DSL-connection, while the wireless interface (ath9k) is supporting almost all IEEE 802.11 MAC schemes (IBSS, mesh, etc.).
All tests show promising results. The initial idea of creating a DSL-network using cheap AVM boxes running on USB powerbanks looks surprisingly realistic. OpenWRT can be used as well as stock firmware. Both L2TPv3 and Tinc are able to spawn usable overlay networks, but only L2TPv3 is able to provide the full downlink capacity of an ADSL2+ annex M link.
As suspected before our tests, the choice between OpenWRT and stock firmware boils down to using IBSS / 802.11s versus FXS / DECT. Both configurations appear to be realistic at this point in time. A more powerful but hungry Arcadyan ARV752DPW22 is a - yet untested - alternative.
Future work is probably going to focus on user-experience and field testing:
- Are signal corps of volunteer-driven relief organizations able to set up the network without help from IT-professionals?
- Given, that one can operate either mesh or POTS on a AVM 7312: What is the tactical choice? Is it reasonable to introduce additional complexity by having boxes with different configurations? Are cheap Android SIP Phones a tactical alternative to POTS-ones?
- What are the DSL signal characteristics of field wire (400m, 800m, 1-pair, 2 pairs)? What is the resulting TCP goodput?
That’s it. Happy hacking ;-).