Linux Networking: Bonded Interfaces and Vlans
I have recently been doing a lot of work with
asylum and during testing have been using
/sbin/ip command from the
iproutes2 package more and more, in
place of more traditional tools such as
looking around the internet i found that usage of
/bin/ip for creating
VLANs and bridges was not well documented. As I find the syntax better than
/sbin/ifconfig command and the unification of multiple tools much
easier to work with I thought I would document it for others to find.
If you find any mistakes please let me know by sending an email to 'code' at this domain. or contact me direct via XMPP
As mentioned above, only
iproute2 commands are used, if you are following
along you will need to ensure the following packages are installed (package
names should be the same on Debian and Redhat based systems):
Please note that all the commands below will require root privileges to run
(as indicated by the line starting with "$" and as such you may need to
sudo -s to root or use the
Linux over the years has gathered an interesting blend of networking and virtual networking features that help Linux in its role as a cheap way to set up a high performance router. With the introduction of virtualisation these capabilities have been extended further still to cope with the new demands imposed by various Virtualisation solution.
In this series we will start off with the traditional Virtual Networking features in Linux that people are familiar with and give a brief rundown of what they are, How to set them up and How they fit into the bigger picture and move on to the newer, more exotic interfaces as we get to the end. I will mainly be keeping the articles focused on Linux networking and try and avoid talking about the network on the 'switch' side of an interface
At some point i would like to include an article or two on the 'bigger picture' that mixes these solutions together to show you the types of networks that can be built with these primitives and their advantages and disadvantages.
And without any Further delay, Bonded interfaces and VLAN:
Bonded interfaces (also known as 'teaming', 'LACP' '802.3ad' 'link aggregation' or 'channel bonding') allow multiple links to behave like one single link. The common reason to do this is to aggregate the bandwidth of multiple links together or to simply provide fault tolerance against the failure of a single link.
There are currently 2 implementations of 'channel bonding'/'channel teaming' in Linux, called 'bonding' and 'teaming' respectfully. The 'bonding' driver is the traditional Linux bonding interface while the 'teaming' driver is a newer implementation that requires a teamd daemon to set things such as packet balancing policy.
If connecting multiple interfaces to the same switch for aggregating bandwidth then you will need to ensure that your switch has explicit '802.3ad' or 'LACP' support and that this is enabled for the ports you are plugging into. If you do not have this support on your switch or are using a simple 'dumb' switch then you may need to use one of the alternate bonding policies to spread packets across the interfaces.
- Provide redundant links to protect against the failure of a single link
- Provide redundant links to different upstream switches to prevent failure of a single upstream switch making the host unavailable.
- Provide a higher bandwidth link made up from the aggregate of the links in the bonded interface.
- Assist in physical migration of hosts (see notes)
As there are 2 ways of doing channel bonding under linux i have provided both methods. If you are just playing around i would recomend giving the 'Teaming' implementation a go, however for anything that requires stability such as a productiin system i can only recomend that the older 'Bonding' implementation be used.
The following command will create a 'teamed' Ethernet interface under Linux:
$ /sbin/ip link add link team0 type team
To add the physical interface eth0 to the team team0 use the following command:
$ /sbin/ip link set eth0 master team0
To remove a physical interface from a team, use the following command:
$ /sbin/ip link set eth0 nomaster
First ensure the 'bonding' module is loaded:
$ /sbin/modprobe bonding
To create a bonded interface called bond0 and assign an ip address:
$ /sbin/ip li set bond0 up
IP addresses can now be added to this interface.
To add the eth0 interface to the bond bond0, ensure that no IP address is assigned to the eth0 interface and use the following command:
$ /sbin/ip li set eth0 master bond0
And repeat for each interface that needs to be added to the bond.
Interfaces can also be removed from the bond using the same method as described for the 'teaming' mode:
$ /sbin/ip li set eth0 nomaster
To set the policy for bond0, first ensure the link is down, then echo the policy to the file below (replacing bond0 with your name of choice):
$ echo "balance-rr" > /sys/class/net/bond0/bonding
The old 'Bonding' interface can operate in several modes as listed below. All modes provide some level of protection against link failure, while some provide aggregation of outgoing bandwidth as well. With the exception of 802.3ad, none of the modes below provide a way to fully utilize all the incoming bandwidth of the links in a bond. However some may work better than others. This is due to the MAC address bouncing between ports, effectively 'time-slicing' which port the switch will send traffic down instead of all links being active in parallel.
Switch topology may also have an effect on incoming bandwidth. Connecting to multiple separate switches as opposed to connecting all links to the same switch should give a boost in bandwidth to loads where the traffic is coming from multiple sources and is evenly distributed over both switches. this is beyond this guide however and really one of the reasons why there is such a position as 'Network Administrator'
- balance-rr: Send packets out each interface sequentially. While this does give fault tolerance, if using only one switch it will only give Aggregate bandwidth for packets leaving via this interface. All incoming packets will be limited to the speed of the slowest link in the bond. Also keep in mind that this will cause the ARP entries for the machine to bounce between interfaces rapidly in the single switch situation. Connecting to multiple upstream switches (one for each link in the bond) will not suffer from these affects and should receive maximum bandwidth on both egressing and ingressing traffic.
- active-backup: Only one link in the bond will be used at a time with the
primary being chosen by the name of the interface in
/sys/class/net/<interface>/bonding/primary, updating the interface in this file will change the primary (active) interface. This mode only provides fault tolerance.
- balance-xor: Send packets out based on a configurable policy of XORing important filed in the packet before transmission. The exact policy can be set with the xmit_hash_policy (consult the Linux documentation for more info). One advantage of this mode is that packets to the same host will be sent out the same interface which may produce more predictable traffic patterns. Provides Outgoing aggregate bandwidth and Fault Tolerance.
- broadcast: All traffic going out is broadcast on all links, most useful in situations where you have 2 mirrors of your network that are not connected to each other (fully isolated). This mode provides fault tolerance.
- 802.3ad: Full LACP support (requires support from the upstream switch), provides both fault tolerance and full aggregate bandwidth. See the Linux documentation on how to set the transmit policy to load balance traffic going out on this interface. Requires all links are connected to the same switch.
- balance-tlb: Attempts to balance outgoing traffic evenly over all interfaces and requires no special switch support. Provides Load balancing and Aggregate bandwidth.
- balance-alb: Same as balance-tlb but also actively updates the MAC address table of the switch at the other end by using ARP requests to attempt to load balance incoming traffic (The difference between the two is that in balance-tlb, the MAC entries are passively updated by packets egressing the interface while this transmits packets to actively update them).
Where possible 802.3ad is recommended, its going to give you the least amount of issues. otherwise balance-[alb,tlb] is a good choice, followed by balance-rr if connected to a single switch or balance-xor if connected to multiple. Active-backup is a good choice if you are not concerned about bandwidth but more about link failure and finally broadcast is very situation specific and only recommended if you know you need it.
- Using 'teaming' mode requires a 'teamd' daemon that can be obtained from the links below
- When attaching interfaces to a bond, the interface to be attached to the bond device must have no IP addresses bound to it
- If using
tcpdumpor other packet capture utilities you will need to ensure that you listen on the bonded interface and not the individual ports unless you know what you are doing or you will miss packets that came in via the other interfaces in the bond
- Having a fault tolerant interface allows for some flexibility when physically moving hosts or moving interfaces between switches. dues to the fault tolerance, an interface can be disconnected and connected to its new location. this can be repeated for each link in turn until the device is migrated with no loss of traffic.
- Wikipedia Article
- Libteam Info: More info on libteam
- Libteam Website
- Documentation/networking/bonding.txt has more info on manually setting things via /sys and the available modes as well as more background on other flags you can set. Highly recommended reading if you wish to use bonding
VLANs are easiest to think of as a virtual switch. They allow you to cut up a switch and all its ports into multiple smaller switches. This is normally done for policy, security or performance and is mainly a 'switch' side feature and not specifically about Linux. Switches with VLAN support typically allow a port to be set in a 'trunking' mode that allows traffic for multiple VLANs to be sent over a link without the traffic mixing. This is typically done by 'tagging' the Ethernet packets with a header to indicate which virtual network they belong to (the VLAN id) and facilitate extending a VLAN over multiple physical switches.
As Linux may need to talk to multiple VLANs and may only have a single connection to the upstream switch it has a VLAN interface that can be used to communicate over a 'trunked' port and communicate with these VLANs. It does this by creating a virtual interface that is bound to a master (trunked) interface. When packets go out via the VLAN interface, The packet is prepended with the correct VLAN header and the VLAN id (which specifies which VLAN you wish to communicate with). While packets out the Master interface have no header prepended.
- Typically used to isolate servers and computers by their 'use', marketing gets put on the 'marketing' VLAN, webservers on the 'webserver' VLAN
- Helps to force all inter subnet traffic to go via a router where policies can be enforced (eg DMZ's). As hosts in different VLANs cannot directly communicate under normal circumstances.
- Useful in router situations where you only have one free port left but need to communicate to multiple networks (eg to route between them). Just set The port to 'trunked' on the switch and bring up the virtual interfaces in Linux.
- Limit the broadcast domain. Some traffic such as ARP resolution (for mapping MAC addresses to IP addresses, The IP equivalent of the game 'Marco Polo') is 'broadcast', Meaning it is sent out all connected interfaces. As hosts are added the bandwidth requirements go up exponentially and as such limiting the size of the 'broadcast' domain can be a good way to ensure the performance of a network. Typically you want to restrict a VLAN to no more than 1000 hosts or a /22 in CIDR terms.
To enable VLAN 20 on eth0 the following command can be used:
$ /sbin/ip li add link eth0 vlan0 type vlan id 20
This will create an interface called vlan0 that is bound to the eth0
ip command will list this as vlan0@eth0, However you
may interact with the link (eg adding or deleting IP addresses) with vlan0
instead of the displayed name.
To remove the interface, the
del command works as normal:
$ /sbin/ip li del vlan0
If you need to register the vlan on the network with GVRP, the following command will enable this when creating the interface:
$ /sbin/ip li add link eth0 vlan0 type vlan id 20 gvrp on
Switches on the network should then cooperate and forward traffic on that VLAN along their interfaces and to the port you are connected to. Note: this requires switch support, Please contact your Network Administrator for more info.
- Sending traffic out the master of a truncked interface may still cause the switch to prepend a VLAN header. This normally happens in VLAN inside VLAN situations.
- VLANs are not a complete security solution, It is possible to break out of a VLAN and send traffic to other VLANs depending on the topology of your network.
- MTUs may need to be adjusted if you have network issues. The VLAN header is 26 bytes. If you notice that things such as an ssh login works fine until you transfer large amount of texts or files then this is a good sign that the MTU needs to be reduced on the VLAN link to accommodate for the extra overhead (the VLAN MTU plus the header should add up to less than the master link MTU).
- While the MTU for most switches is stated as 1500, Many devices actually have a MTU of 1526 bytes to account for a VLAN header. However this should not be relied upon and MTU adjustment is a much safer method.
- The VLAN ID field is 12bit, Limiting the ammount of Virtual LANs to 4096, some VLANs are reserved on some devices and other devices may place limits on the ammount of ative VLANs, preventing use of the full 4096 VLANs.
- GVRP can be used to 'register' a VLAN on a network of switches and have them cooperate in forwarding VLANs to the switches participating in that VLAN automatically. This feature requires switch support and a network set up to use this feature.