Supernode

[wikipedia-article] i don't like that word

quick'n dirty required software:

1) [fastd] mesh-node-vpn

2) [vpn-key-upload] todo: remove overhead (apache,ruby,sinatra ...), kiss principle] not documented in here (todo)

3) [batman-adv]

4) [tinc] mesh-backbone-vpn

5) OS: debian wheezy (or whatever you like)

load batman-adv kernel-module while booting in /etc/modules.conf add

batman-adv

disable pmtu-discovery in /etc/sysctl.conf add

net.ipv4.ip_no_pmtu_disc=1

fastd in /etc/apt/apt/sources.list add

deb http://repo.universe-factory.net/debian/ sid main

keyimport

 gpg --keyserver pgpkeys.mit.edu --recv-key  AB7A88C5B89033D8
 gpg -a --export AB7A88C5B89033D8 | sudo apt-key add -

install fastd

 apt-get update
 apt-get install fastd

fastd configuration

/etc/fastd/kbu# ls
backbone  fastd.conf peers 


root@fastd4:/etc/fastd/kbu# ls backbone/
fastd1  fastd2  fastd3  fastd4
root@fastd4:/etc/fastd/kbu# cat backbone/*
key "4f856d95bd596ac7724edca73a19e6e9d142b374df27166bb1a78e58785efc59";
remote ipv4 "fastd1.kbu.freifunk.net" port 10000;
key "e1916b66c4f8a795e217877cf72607d952e796463c7024dd9a6a47ae2929bc10";
remote ipv4 "fastd2.kbu.freifunk.net" port 10000;
key "d56181dfe9b5ac7cfe68a94c0ce406322a9924286a751673da0dcb28ad5218b0";
remote ipv4 "fastd3.kbu.freifunk.net" port 10000;
key "9b3f65f99963343e2785c8c4fad65e70b73ee7e1205d63bd84f3e2decb53e621";
remote ipv4 "fastd1.kbu.freifunk.net" port 10000;

[fastd-backbone] for more

fastd.conf

# Log everything to a log file
log to "/var/log/fastd.log" level debug; 
# Set the interface name
interface "fastd";

# Support xsalsa20 and aes128 encryption methods, prefer xsalsa20
method "xsalsa20-poly1305";
method "aes128-gcm";
method "null";  #todo: WTF, encryption is off!?, what do we exchange keys for?

# Bind to a fixed port, IPv4 only
bind 0.0.0.0:10000;

# Secret key generated by `fastd --generate-key`
# in this case you have to ask the  holy guru of that project for the private key'
secret "won't tell my secret here";

# Set the interface MTU for TAP mode with xsalsa20/aes128 over IPv4 with a base MTU of 1492 (PPPoE)
# (see MTU selection documentation)
mtu 1426;
# Include peers from the directory 'peers'
include peers from "/etc/fastd/kbu/backbone";
include peers from "/etc/fastd/kbu/peers";

#ipadress, routing
#on establish  "/root/scripts/fastdup.sh"; #does'not work for me, need to invoke an external script somewhere else

fastd-ifup-script,

#/bin/sh

#bring the interface up
/sbin/ifconfig fastd up

#start bat0 interface
/usr/sbin/batctl if add fastd
#dhcp forward through batman
/usr/sbin/batctl gw_mode server
/sbin/ifconfig bat0 172.27.XX.1 netmask 255.255.192.0 up

#routing 
/sbin/ip rule add from 172.27.0.0/18 table ffkbu
/sbin/ip route add 172.27.0.0/18 dev bat0 table ffkbu
/sbin/ip route flush cache

backbone-net (tinc)

apt-get install tinc

root@fastd4:/etc/tinc/backbone# ls 
bbkeys
hosts -> bbkeys
tinc.conf
tinc-up

public-keys for the tinc-backbone, can be found [here]

add your public key to that repo

tinc.conf

/etc/tinc/backbone# cat tinc.conf 
Name=fastd4
Device=/dev/net/tun
Mode=router
ConnectTo=paula
ConnectTo=paul
#enough links?

tinc-up

 
#!/bin/sh
ifconfig $INTERFACE 172.27.255.X netmask 255.255.255.0 up
#
#set some backbone related routes
ip route add 172.27.255.0/24 dev backbone table ffkbu
ip route add default dev backbone table ffkbu
ip route flush cache

routing

we need to set up policy based routing.

echo "200 ffkbu" >> /etc/iproute2/rt_tables

this is done in the tinc-up and the fastd-up scripts

ip rule ls

ip rule ls
0:      from all lookup local 
32765:  from 172.27.0.0/18 lookup ffkbu 
32766:  from all lookup main 
32767:  from all lookup default

ip route list table ffkbu

ip route list table ffkbu
default dev backbone  scope link 
172.27.0.0/18 dev bat0  scope link 
172.27.255.0/24 dev backbone  scope link 

dhcpd.conf

subnet 172.27.0.0 netmask 255.255.192.0 {
  #if guru talks about subnets,
  #just the range is meant! 
  #all nodes,supernodes,and clients 
  #remain in the net 172.27.0.0/18
  range 172.27.XXX.XXX 172.27.XXX.XXX;
  option domain-name-servers 172.27.XXX.1;
  option domain-name "kbu.freifunk.net";
  option routers 172.27.XXX.1;
  option broadcast-address 172.27.63.255;
  default-lease-time 600;
  max-lease-time 7200;
}

you need to install a bind9

fastd

UDP-Queue Größe

Durch Broadcasts können können auf Supernodes kurzzeitig hohe Lastspitzen enstehen. In der aktuellen batman-adv-Version wird jedes Broadcast-Paket zudem 3x auf jedem Link versendet. Die vergleichweise hohe Datenrate von 100MBit/s zwischen Supernodes bewirkt darüber hinaus, dass broadcast-Pakete mit 100MBit/s eingehen können, während sie gleichzeitig an alle Nodes gesendet werden müssen.

Linux-Distributionen sehen per Default Queue-Größen im Bereich von 128 KB vor (http://www.cyberciti.biz/faq/linux-tcp-tuning/). Supernodes sind somit nicht in der Lage entsprechende Lastspitzen zu puffern und während Idle-Times zu versenden. Läuft die UDP-Queue über, so loggt fastd:

2013-03-30 12:32:01 +0100 --- Warning: sendmsg: Resource temporarily unavailable
2013-03-30 12:32:01 +0100 --- Warning: sendmsg: Resource temporarily unavailable
2013-03-30 12:32:01 +0100 --- Warning: sendmsg: Resource temporarily unavailable
2013-03-30 12:32:01 +0100 --- Warning: sendmsg: Resource temporarily unavailable
2013-03-30 12:32:01 +0100 --- Warning: sendmsg: Resource temporarily unavailable

Jeder Log-Eintrag entspricht einem Paket, dass nicht enqueued werden konnte und damit verworfen wurde. Die Queue-Größen können via systctl angepasst werden (http://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php). Auf fastd2 wird zur Zeit verwendet:

#/etc/sysctl.conf
net.core.rmem_max=83886080
net.core.wmem_max=83886080
net.core.rmem_default=83886080
net.core.wmem_default=83886080

Das System verfügt dadurch über 80MB Speicher für Queues. Per default stehen ebenfalls 80MB zur Verfügung. Die konfigurierten 80MB reichen aus, um die 100MBit/s Verbindung über mehrere Sekunden auszulasten und Lastspitzen abzufangen.