import site.body

Linux Networking: MAC VLANs and Virtual Ethernets

Updates

  • Hairpin mode for MAC VLAN has been updated to reflect that Default L2 switch behavior and not STP is responsible for preventing a packet flowing back from the port it came from

This time we only play with Linux specific Virtual interfaces. MAC VLAN and Virtual Ethernet Pipes. These are mainly used in containerization situations but can be useful for other things such as emulating network topologies in conjunction with bridges.

MAC VLAN

A MAC VLAN can be thought of as a reverse VLAN under Linux. instead of taking a single interface on the OS side of a network interface and mapping it to multiple virtual networks on the Network side of the interface (one to many). A MAC VLAN takes a single Network interface and creates multiple virtual ones with different MAC addresses (many to one).

While under Linux an interface can already have multiple addresses, a MAC VLAN allows further isolation on what traffic can be seen on such an interface. A MAC VLAN will only be able to see traffic that has a MAC address that matches the interface, Preventing processes on other interfaces from listening in on traffic destined for another MAC VLAN.

Usage

  • Replace bridge setups with a more lite-weight solution in situations where the bridge is just providing egress to the normal network and is not set up as a private internal network between Guests on the host.
  • for use with MAC Address filtering where some devices are put onto a separate network based on their MAC address (eg in the case of a VoIP phone, bring up a MAC VLAN that is recognized as a VoIP phone by the switch at the other end and put on a voice VLAN, then set it up with static or DHCP IP address and force your VoIP softphone to listen on that interface, while your normal traffic goes out onto the normal traffic VLAN).
  • When consolidating multiple hosts together, a MAC VLAN can be created on the new host for every public interface on the old hosts with the same MAC address as the old host's interface. As DHCP can only assign a single IP address per interface, doing the above will allow you to run multiple DHCPs with one per interface and have the old services available at the old addresses allocated by DHCP, Minimizing the amount of configuration changes needed to consolidate hosts.

Setup

To create a MAC VLAN interface and auto generate a MAC address use the following command:

$ /sbin/sudo ip li add link eth0 mac0 type macvlan

This will create a new interface called mac0@eth0 that can have an IP address assigned to it or use DHCP to assign an address. Note that to bring the interface up, The main interface the virtual link is bound to must also be in the 'UP' state.

You do not need to specify the full mac0@eth0 to make changes to the mac0 interface, the '@eth0' is for display purposes only.

if you need to create a MAC VLAN with a specific mac address use the form:

$ /sbin/ip li add link eth0 mac1 address 56:61:4f:7c:77:db type macvlan

This will create a new interface called mac1@eth0 with a preset MAC address (56:61:4f:7c:77:db, generated from script below). the MAC address may be a randomly generated MAC address or can be cloned from an existing interface (eg when consolidating hosts).

Notes

  • The Second bit of the most significant Byte indicates if the MAC address is Locally or Universally administered. when generating MAC addresses, this bit should be set to '1', indicating it is Locally administered.
  • Deleting the main interface the link is bound to (eg in the case of a bridge) will cause the MAC VLAN interfaces bound to that interface to also be deleted.

Modes

a MAC VLAN can be created in one of 4 modes: private, vepa, bridge or passthru and is documented in the git commit 618e1b7482f7a8a4c6c6e8ccbe140e4c331df4e9 (linked below) for the Linux kernel.

Which mode you choose depends on your networking hardware and security requirements. In most cases passthru should 'just work' however if you experience issues with 2 MAC VLANs being able to communicate consider the 'bridge' mode below to have Linux handle switching of the packets between interfaces.

  • Private: Filter all incoming packets so that no MAC VLAN bound to an interface can communicate with each other (drop all packets ingressing over the interface that have a source MAC address that matches one of the MAC VLAN interfaces).
  • VEPA: This mode expects a VEPA/802.1Qbg capable switch at the other end of the interface. The switch on the other end acts as a reflective relay, As such we forward all traffic out to the switch even if it is destined for us and rely on the switch at the other end to send it back (aka 'Hairpin' mode). Useful when there are network level policies being enforced by the switches (eg DHCP filtering).
  • Bridge: Provide a pseudo bridge to allow MAC VLANs bound to the same interface to communicate without the packet being sent out the network interface.
  • Passthru: Just pass the packet to the network. The default behavior of a L2 switch is to not send a packet back down the port it egressed from to prevent loops forming. This has the effect of making a MAC VLAN in this mode with a standard switch behave as if it was in Private mode and prevent cross MAC VLAN communication

In both VEPA and Passthru modes traffic destined to a different MAC VLAN on the same interface will transit the physical interface twice, Once to egress the interface where it is switched and sent back and ingresses via the same interface. As this can affect available physical bandwidth and also restricts inter MAC VLAN traffic to the speed of the physical connection.

In Private mode, No node may communicate with each other. This can help prevent discovery of other MAC VLANs and may be useful in a multi-tenant environment in conjunction with a switch that sends all traffic to a router.

In Bridge mode all traffic between MAC VLANs will be switched in memory, this can lead to higher network speeds between interfaces than VEPA or Passthru due to memory typically being faster than network interfaces. this may incur some additional cpu overhead to handle each packet.

MAC Address Generation

To generate a new random mac address with all the bits set corectly use the following script:

python -c "import random; mac = [random.choice(range(256)) for i in range(6)]; mac[0] |= 0x02; mac[0] &= 0xfe; print ':'.join('%02x' % m for m in mac)"

Links

  • git entry explaining the diffrent modes (Alternativly, run git log 618e1b7482f7a8a4c6c6e8ccbe140e4c331df4e9 in a checkout of the Linux source code)
  • VEPA Support: Notes on VEPA support in the linux kernel (useful for understanding what this feature does)
  • Edge Virtual Bridging: More background on VEPA
  • Transparent Bridging: Flooding behavior for switches

Virtual Ethernet Device

Virtual Ethernet device pairs are a pair of fake Ethernet devices that act as a pipe, Traffic sent via one interface comes out the other. As these are Ethernet devices and not point to point devices you can handle broadcast traffic on these interfaces and use protocols other than IP.

The are most useful in conjunction with containers and are the main way to get traffic into and out of a container with a separate network namespace. Where the host can then route the traffic or attach the interface to a bridge.

Usage

  • Creating virtual networks between containers
  • Providing a routed link for a container
  • Providing an endpoint that can be bound to a bridge device for a container
  • Emulating high latency internet links (see links section below)
  • Emulate Networks in conjunction with bridges

Setup

To create a virtual ethernet pipe with one end called veth0 and the other called veth1, use the following command:

$ /sbin/ip li add veth0 type veth peer name veth1

You can also set values for either end of the link in the same command using the normal syntax for the left hand side link and by adding the values after the peer statement for the right hand side of the link.

the above command uses this to explicitly set the name of the right hand side of the link, which if not specified will be set to the same as the left hand side and fail with the following error:

Error: argument "peer" is wrong: "name" too long

Notes

  • The pair of interfaces are identical and act as a dumb pipe, there is no master or slave end
  • Deleting either end will cause both interfaces to be deleted
  • The pair of interfaces implement carrier detection and can tell when one side of the link is in the 'DOWN' state. if the other link is in the 'DOWN' state it will indicate 'NO-CARRIER' until the other end is brought up
  • If you wish to have the same interface name in both a guest container and the host, i would recommend using the following command: /sbin/ip li add veth0-master type veth peer name veth0 And then renaming the left hand side interface (veth0-master) to veth0 after executing the following command to place the network interface in the guest container: /sbin/ip li set veth0 netns <guest pid>

Links

  • Previous article that uses veth devices and network namespaces to emulate a high latency link