Altough our first field test went well, the network infrastructure has still room for improvements. This article explains some ideas for integrating a Digital Subscriber Line (DSL) based network into field tracks and their evaluation.

Learning from the initial field test

The first test happened on a parking lot. Three mobile treatment center were raised by different units. Participants were tracked during triage and therapy.

To provide wlan at the site, four UniFi AC Mesh AP units were installed. A local controller was used for provisioning and monitoring. Using different channels, the wlan provided more than enough capacity for all JellingStone devices. All units were hooked up to the grid using power-over-ethernet (POE). Cat5 cables were used to create a wired, switched backbone. The deployment went smooth and took about 30 minutes. Still, some shortcomings occurred:

  1. Not having power cables or batteries in the beginning, the build-up started two hours later than planned initially. By intuition, it appears reasonable to have batteries - but using battery power imposes power constraints on the devices in use: Overprovisioning using powerful devices becomes less attractive.
  2. Using mesh-links instead of Cat5-cables was not attractive. For power distribution, all AP were hooked up by cables nevertheless and Cat5 is easier to handle than H07RNF-3G2.5. In this non-reflecting environment, mesh-links would have required line-of-sight in addition.

Learning from this event, we decided to extent the network regarding to the devices in use:

  • Create a power system using local batteries to exploit mesh links.
  • Introduce smaller boxes, requiring less power.
  • Potentially use field wire (still common among relief organizations in Germany) for non-line-of-sight situations. This could improve handling and allow larger communication range compared with Cat5.

What do we actually need?

On the one hand, each JellingStone sensor roughly transmits up 10 KByte every 5 seconds - i.e. to 2 KByte/s. Thus 200 devices need 400 Kbyte/s, requiring 3.6 MBit/s approximately. That’s about the uplink capacity of an ADSL2+ Annex M link. Using compression and avoiding fragmentation, we hope reduce the traffic to 10% in the future.

On the other hand, lines of field wire are usually deployed and used with telephones, opening the case for Voice-over-IP (VoIP).

In general, field exercises may take up to few days - batteries should should last for 8-12 hours at least. Replacing batteries 2-3 times per day is fine.

Hardware in use

To validate our idea, we decided to do some benchmarking based on old ADSL hardware and got an old Zyxel IES‑1248‑51A for testing (~ 120 €). It is able to connect up to 48 ADSL2+ boxes.

Being somewhat stingy, we looked for cheap ADSL2+ boxes. Two boxes draw our attention:

Both boxes are available for ~1 € (used; 5 € including shipping) and supported by OpenWRT in principle. But it is a tough choice:

  • The ARV752DPW22 consumes three times the power of the 7312. While the 7312 uses 5V/1.6A and could be powered using an USB powerbank, the ARV752DPW22 needs 15V / 1.6A, practically resulting in a lead-battery having a step-up converter.
  • On the 7312, the telephone connectors can only be used with AVM’s stock firmware. But when using stock firmware, the wireless cannot do anything interesting (802.11s, P2P, IBSS, etc.).

To keep things simple in the beginning, we went on testing the 7312 using stock firmware and OpenWRT at our hackathon in November.

Network-architecture, roaming and the case for an overlay network

In general, wireless stations should be able to roam within the network while maintaining IP-connectivity. In a very simple form, this can be realized by creating a Layer-2 / ethernet segment bridging all wireless access points.

In principle, this could be realized by running Ethernet-over-ATM (EoA) and bridge the corresponding interface to wireless interfaces running in AP mode. But haven run a similar configuration with Thomson TG605s back-to-back before, we noticed that some DSL devices are not used to update mac address tables when stations roam through the network - the entries turned out to be quite persistent. This is completely fine for PPPoE setups with no mobility at all and I guess, that many boxes were never tested to update any address due to a roaming device.

Another point is, that AVM does not expose the EoA interface to the kernel. Thus it is impossible to bridge it with the wireless lan interface using the linux command line.

Both points can be addressed by bridging an overlay network (VPN without encryption) instead of the EoA interface. Being limited to technologies that are part of Freetz / Fritz OS 6, the two most interesting candidates are:

Stateless L2TPv3 pseudowires are simple point-to-point links. L2TPv3 is implemented in the kernel and can be configured using ip l2tp for each link individually. Tinc on the other hand is a decentralized and simple VPN program, that can be used like a distributed switch.

Compared with L2TPv3, tinc is running in userland and potentially slow. But it supports fragmentation, path-mtu & peer discovery and passing traffic between peers directly. L2TPv3’s mtu is bound by the underlying link - IPv4 fragmentation is possible, though. In other words: Tinc might be slow, while L2TPv3’s achilles heel is costlier configuration. This raises two questions:

  • What is Tinc’s TCP-Goodput compared to L2TP? Is an AVM 7312 running tinc able to switch at 3.5 MBit/s at least?
  • Is it possible to create a L2TPv3 pseudowire on AVM 7312 devices? What is the actual effort for configuring and maintaining all L2TPv3 links?

For testing, we decided to give both protocols a try using Freetz - we also tested installing and using OpenWRT.

Testing freetz

Working on top of AVM Fritz OS, Freetz enables SSH-access and command line administration, needed to configure l2tp-links. As of today, both Fritz OS 6.33 and Fritz OS 6.55 can be used in combination with freetz’es master branch. In general, build-instructions for freetz are availble at freetz.github.io - keep in mind, that the project is hosted on github nowadays.

Not having /sbin/l2tp in 6.55 initially, 6.30 is used - tinkering with 6.55 is not on the agenda, yet. Tinc on the other hand is included in our freetz build-.config - tun devices drives are included in 6.33 - no futher kernel modules are needed in addition. As of today, running Freetz / FritzOS 6.33 results in linux 2.6.32 and tinc 1.0.35.

Installation

After cloning the master branch and entering the menu (make menuconfig), the tinc package is included in addition. Executing make creates the image - (for example: 7312_06.30-freetz-HEAD-20191017-828c7d609-dirty.de_20191107-121917.image) Extract and flash:

tar xvf images/7312_06.30-freetz-HEAD-20191017-828c7d609-dirty.de_20191107-121917.image
cd var/tmp # change folder
ftp 192.168.178.1 # 3-5 seonds after power on; during boot

The Zyxel IES-1248-51A requires the Fritzbox to operate using Annex A. This is done by setting kernel_args accordingly.

Transcript of the FTP session. Prompt is ftp> - username / password: adam2

Connected to 192.168.178.1.
220 ADAM2 FTP Server ready
Name (192.168.178.1:jan): adam2
331 Password required for adam2
Password:
230 User adam2 successfully logged in
Remote system type is AVM.
ftp> quote SETENV kernel_args annex=A
200 SETENV command successful
ftp> binary
200 Type set to BINARY
ftp> passive
Passive mode on.
ftp> quote MEDIA FLSH
200 Media set to MEDIA_FLASH
ftp> put kernel.image mtd1
local: kernel.image remote: mtd1
227 Entering Passive Mode (192,168,178,1,12,8)
150 Opening BINARY data connection
226 Transfer complete
13660936 bytes sent in 29.32 secs (454.9759 kB/s)
ftp> quote REBOOT
221 Thank you for using the FTP service on ADAM2
ftp> quit
221 Goodbye.

Benchmarking Tinc & L2TPv3

For testing, we connected a Lenovo T580 laptop to the AVM 7312 using a 100 MBit/s wired ethernet connection. To avoid limits imposed by the DSL line, the ethernet port (“LAN”) is configured for WAN access - the overlay network is working on top of the wired connection. Bridging the overlay network to the infrastructure wlan on the 7312, the corresponding interface on the laptop becomes effectively part of the LAN of the 7312. At last, another laptop (Lenovo T490) providing an iperf3 service is connected to the infrastructure wlan hosted on the 7312. The network is illustrated in Figure 1:

Benchmark network
Figure 1: Overlay network setup for benchmarking

Tinc

Including tinc in the Freetz build before, it is possible to setup tinc using the Freetz GUI at Port 81/tcp. But due to a missing inode in the /dev-filesystem, tinc is unable start. This can be fixed by SSH’ing to the box and creating the inode manually. Doing only preliminary benchmarking, we have not tried creating a reboot-save configuration, yet.

mknod /dev/tun c 10 200

All configuration files are available at https://github.com/FieldTracks/benchmark/tree/master/tinc. The laptop is configured using linux command tools in the distribution (Manjaro Linux 18.1.2, Kernel 4.19.81, Tinc 1.0.35).

$ iperf3 -c 192.168.188.21

Connecting to host 192.168.188.21, port 5201
[  5] local 192.168.188.200 port 56890 connected to 192.168.188.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   864 KBytes  7.08 Mbits/sec    0   73.0 KBytes       
[  5]   1.00-2.00   sec   776 KBytes  6.36 Mbits/sec    0    107 KBytes       
[  5]   2.00-3.00   sec   545 KBytes  4.47 Mbits/sec    3   89.3 KBytes       
[  5]   3.00-4.00   sec   763 KBytes  6.25 Mbits/sec    0    104 KBytes       
[  5]   4.00-5.00   sec   592 KBytes  4.85 Mbits/sec    3   78.4 KBytes       
[  5]   5.00-6.00   sec   763 KBytes  6.25 Mbits/sec    0   89.3 KBytes       
[  5]   6.00-7.00   sec   699 KBytes  5.73 Mbits/sec    3   83.9 KBytes       
[  5]   7.00-8.00   sec   699 KBytes  5.73 Mbits/sec    0   78.4 KBytes       
[  5]   8.00-9.00   sec   763 KBytes  6.25 Mbits/sec    0   89.3 KBytes       
[  5]   9.00-10.00  sec   636 KBytes  5.21 Mbits/sec    0   92.0 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.93 MBytes  5.82 Mbits/sec    9             sender
[  5]   0.00-10.15  sec  6.77 MBytes  5.60 Mbits/sec                  receiver

iperf Done.

Looking at the results, the bitrate rate looks volatile, but clearly above 3.5 MBit/s all the time. Assuming, that using DSL instead of ethernet poses no additional CPU load on the 7312, it appears reasonable to assume, that tinc is able to fill the ADSL2+ Annex M uplink completely. For the downlink the situation is different: Being able to serve 27 MBit/s, tinc poses a relevant restriction.

As a result, tinc can potentially be used to transport sensor data generated by JellingStone - but additional IP-traffic (web browsing, etc.) can only be handled to a very limit extend. Tinc’s performance is good enough but has room for improvements.

L2TPv3

Unlike tinc, L2TPv3 cannot be configured using the Freetz WebUI. Setting up L2TPv3 requires:

  1. Creating a virtual interface having the same IPv4 address as used on the WAN-Interface. This is needed because the kernel appears to be unaware of the network configuration done in the driver-blobs.
  2. Creating a L2TPv3-tunnel according to the IPv4 addresses used in the WAN network.
  3. Creating a L2TPv3 session on top of the tunnel.
  4. Adding the corresponding l2tpeth pseudo-device to the existing LAN-bridge

For automation, we created a small script, that can be executed during boot, utilizing the rc.custom hook. Running iperf3 in this configuration yields:

$ iperf3 -c 192.168.188.21

Connecting to host 192.168.188.21, port 5201
[  5] local 192.168.188.200 port 50020 connected to 192.168.188.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  9.22 MBytes  77.3 Mbits/sec   22    451 KBytes       
[  5]   1.00-2.00   sec  9.28 MBytes  77.8 Mbits/sec    0    552 KBytes       
[  5]   2.00-3.00   sec  8.29 MBytes  69.6 Mbits/sec    0    609 KBytes       
[  5]   3.00-4.00   sec  8.17 MBytes  68.5 Mbits/sec    4    453 KBytes       
[  5]   4.00-5.00   sec  8.48 MBytes  71.1 Mbits/sec    0    493 KBytes       
[  5]   5.00-6.00   sec  8.42 MBytes  70.6 Mbits/sec    0    519 KBytes       
[  5]   6.00-7.00   sec  8.29 MBytes  69.6 Mbits/sec    0    533 KBytes       
[  5]   7.00-8.00   sec  8.97 MBytes  75.2 Mbits/sec    0    540 KBytes       
[  5]   8.00-9.00   sec  8.85 MBytes  74.2 Mbits/sec    0    541 KBytes       
[  5]   9.00-10.00  sec  9.03 MBytes  75.8 Mbits/sec    0    541 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  87.0 MBytes  73.0 Mbits/sec   26             sender
[  5]   0.00-10.10  sec  85.4 MBytes  70.9 Mbits/sec                  receiver

iperf Done.

Again, the bitrate looks volatile and is around 70 MBit/s, potentially hitting the capacity of the 2.4 GHz network - we have a rather noisy environment in our local hackerspace.

As a result, the higher bitrate (~70 MBit/s) makes L2TPv3 by far more attractive than tinc (~ 5 MBit/s) .

Testing OpenWRT

The AVM 7312 is supported by OpenWRT since version 19.07, which is not released yet. 19.07-rc1 is used for testing.

Unfortunately, the needed DSL driver is not configured in OpenWRT stock builds. By default, it includes ITU 992.1/.3/.5 Annex B (ISDN blanking period, Germany), only. However, the IES-1248-51A needs Annex A (POTS blanking period, rest of the world).

In principle, OpenWRT provides an Annex A firmware, that was made available directly by lantiq a few years ago. Newer versions can only be downloaded using vendor firmware bundles. Thus they cannot be provided by OpenWRT due to legal reasons.

Providing version 4.4.4.0.0.1, OpenWRT’s firmware is rather old and causes stability issues on the 7312. Thus we went on by copying ar9-A-dsl.bin from a freetz box into the OpenWRT file system and updating the /lib/firmware/adsl.bin-symlink accordingly.

Flashing can be done using fritz-tools - transcript:

wget https://raw.githubusercontent.com/freifunk-darmstadt/fritz-tools/master/fritzflash.py
# Downloading log ommited
python3 fritzflash.py --image path/to/openwrt-lantiq-xway-avm_fritz7312-squashfs-sysupgrade.bin

Executing fritzflash.py yields an interactive terminal program guiding through the flash process. Log:

This program will help you installing Gluon, a widely used Firmware for Freifunk networks, onto your AVM device.
You can always find the most current version of this script at https://www.github.com/freifunk-darmstadt/fritz-tools

It is strongly recommended to only connect your computer to the device you want to flash.
Disable all other connections (Ethernet, WiFi/WLAN)!

Before we start, make sure you have assigned your PC a static IP Address in the Subnet of the device you want to flash.
The following example would be a completely fine option:

IP-Address: 192.168.178.2
Subnet: 255.255.255.0
Gateway: 192.168.178.1
DNS Servers: Leave blank

Once you're done, press enter, disconnect power from your AVM device and reconnect the power-supply.

Trying to autodiscover! Abort via Ctrl-c.
FritzBox found at 192.168.178.1

Autodiscovery succesful!
-> Device detected at 192.168.178.1.
-> Establishing connection to device!
--> Try 1 of 10
-> Flash image

Writing Gluon image to your AVM device...
This process may take a lot of time.

First, the device will erase its current Operating System.
Next, the device will write the Gluon image to its memory.
The red Info LED will illuminate in this step. Don't worry, this is expected behavior.

Do *not* turn of the device!

We will tell you when your device has finished installing Gluon (this may take a while).
-> Image write successful
-> Performing reboot

== Congratulations! ==

Your device is now running Gluon.
It will restart and in 2-5 minutes you will be able to visit its config-mode.
Remember to reconfigure your interface to automatically obtain an IP-address!
You can reach config-mode by typing in http://192.168.1.1/ in your preferred Webbrowser.

Press any key to exit.

After a while, the box is reachable using ssh at 192.168.1.1. To establish a DSL connection we changed the network configuration according to our DSL setup using SSH

uci set network.dsl.annex='a'
uci set network.atm.vpi='0'
uci set network.atm.vci='33'
uci set network.wan.proto='dhcp'
uci del network.wan.username
uci del network.wan.password
uci commit network
reboot

After rebooting and giving the DSL-sync some time, /etc/init.d/dsl_control status shows the configuration in use.

/etc/init.d/dsl_control status
ATU-C Vendor ID:                          
ATU-C System Vendor ID:                   
Chipset:                                  Ifx-AR9
Firmware Version:                         4.6.1.7.0.1
API Version:                              3.24.4.4
XTSE Capabilities:                        0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0
Annex:                                    A
Line Mode:                                G.992.5 (ADSL2+)
Profile:                                  
Line State:                               UP [0x801: showtime_tc_sync]
Forward Error Correction Seconds (FECS):  Near: 0 / Far: 0
Errored seconds (ES):                     Near: 4 / Far: 1
Severely Errored Seconds (SES):           Near: 1 / Far: 1
Loss of Signal Seconds (LOSS):            Near: 0 / Far: 1
Unavailable Seconds (UAS):                Near: 29 / Far: 29
Header Error Code Errors (HEC):           Near: 1102 / Far: 16
Non Pre-emtive CRC errors (CRC_P):        Near:  / Far:
Pre-emtive CRC errors (CRCP_P):           Near:  / Far:
Power Management Mode:                    L0 - Synchronized
Latency [Interleave Delay]:               0.25 ms [Fast]   0.50 ms [Fast]
Data Rate:                                Down: 27.446 Mb/s / Up: 1.291 Mb/s
Line Attenuation (LATN):                  Down: 0.0 dB / Up: 6.2 dB
Signal Attenuation (SATN):                Down: 0.0 dB / Up: 0.0 dB
Noise Margin (SNR):                       Down: 6.0 dB / Up: 6.2 dB
Aggregate Transmit Power (ACTATP):        Down: 8.1 dB / Up: 6.1 dB
Max. Attainable Data Rate (ATTNDR):       Down: 27.352 Mb/s / Up: 1.335 Mb/s
Line Uptime Seconds:                      11
Line Uptime:                              11s

As a result, OpenWRT is able to create a stable DSL-connection, while the wireless interface (ath9k) is supporting almost all IEEE 802.11 MAC schemes (IBSS, mesh, etc.).

Conclusion

All tests show promising results. The initial idea of creating a DSL-network using cheap AVM boxes running on USB powerbanks looks surprisingly realistic. OpenWRT can be used as well as stock firmware. Both L2TPv3 and Tinc are able to spawn usable overlay networks, but only L2TPv3 is able to provide the full downlink capacity of an ADSL2+ annex M link.

As suspected before our tests, the choice between OpenWRT and stock firmware boils down to using IBSS / 802.11s versus FXS / DECT. Both configurations appear to be realistic at this point in time. A more powerful but hungry Arcadyan ARV752DPW22 is a - yet untested - alternative.

Future work is probably going to focus on user-experience and field testing:

  • Are signal corps of volunteer-driven relief organizations able to set up the network without help from IT-professionals?
  • Given, that one can operate either mesh or POTS on a AVM 7312: What is the tactical choice? Is it reasonable to introduce additional complexity by having boxes with different configurations? Are cheap Android SIP Phones a tactical alternative to POTS-ones?
  • What are the DSL signal characteristics of field wire (400m, 800m, 1-pair, 2 pairs)? What is the resulting TCP goodput?

That’s it. Happy hacking ;-).