Traffic mirroring is a useful tool for debugging protocol level issues. It’s particularly useful for network elements which may have limited logging or other tooling that could be used for troubleshooting. Traditionally, traffic mirroring was performed by a technique called Switched Port Analyzer (SPAN) at the physical switch. In this case, the switch copies network traffic from certain ports or VLANs to another port where the traffic can be inspected. However, as network equipment evolves into software running on cloud hosts, i.e. network function virtualization, we are faced with a challenge: how do we mirror traffic when we have limited control over the physical network infrastructure? The answer, of course, is to move the traffic mirroring function into software too.

Traffic mirroring is also known as port mirroring, tapping, sniffing, or snooping. The choice of terminology largely depends on the motives of the person performing the traffic mirroring, which leads me to the disclaimer that these techniques should not be used to do anything illegal.

By moving the traffic mirroring to software, we do add some complexity to our systems and suffer some performance overhead. But at the same time we gain the ability to control which traffic is monitored by applying filters. Furthermore, we can use network tunnels to ship our mirrored traffic to one or more generic physical or virtual Linux hosts. This does away with the need for costly monolithic monitoring appliances plugged directly into switches and gives us a lot more flexibility to create and destroy network taps as required.

If the traffic that needs to be mirrored is generated by a virtual machine, there are two places to do it: either within the virtual machine itself or on the underlying hypervisor. If a virtual machine is supplied by a network equipment vendor, they might have some reservations about installing any additional software or features which may affect the performance of the machine. In this case, we are limited to capturing traffic at the hypervisor level.

In my previous article I covered how to set up layer 2 and layer 3 GRE tunnels. A layer 2 GRE tunnel is a great option for traffic mirroring and here I will demonstrate using Linux tc to copy traffic onto a GRE tunnel where it can be shipped to another host for further processing. If you don’t already have a layer 2 GRE tunnel established, go ahead and create one as below where remote is the host to which the mirrored packets will delivered and dev is the interface that the tunnel will use.

ubuntu@host-1:~$ sudo ip link add tun0 type gretap remote 10.131.73.16 local 10.131.73.9 dev eth1ubuntu@host-1:~$ sudo ip addr add 172.18.0.1/24 dev tun0ubuntu@host-1:~$ sudo ip link set tun0 up

Note that we’re assigning an IP address to the tun0 interface. This is not strictly necessary because simply writing a packet to the tunnel interface will cause it to get sent to the far end of the tunnel. However, it can be useful for debugging purposes.

Typically, traffic from one network interface, let’s call it the capture interface, would be mirrored to a tunnel running over a different interface. However, you may wish to mirror traffic to a tunnel on the same network interface. In this case, special care must be taken to avoid a traffic loop, whereby the tunnel traffic is recursively mirrored back to the tunnel.

Apart from the danger of loops, there is another serious drawback to using the capture interface to host the tunnel: you will not be able to increase the MTU of the tunnel interface independently of the capture interface. Therefore, any packets that are close to the MTU size cannot be mirrored because once the GRE and encapsulating IP headers are added, the tunneled packet will exceed the MTU and it will be dropped. Therefore, it is highly recommend having a second network interface to host the tunnel with an MTU that exceeds that of the capture interface by at least 38 bytes to cater for the tunnel overhead.

Linux Tc

Modern Linux kernels have built-in traffic control and shaping features. The iproute2 package includes a command called tc to control these features from the command line. Tc works well to mirror packets to a tunnel but the command syntax can be a little tricky.

A Queuing Discipline (qdisc) is an algorithm that manages the queues of a network device. Qdiscs can be used to shape traffic using different queuing algorithms. Given that only egress traffic can be shaped (you can’t control when others will send you packets), most of the functionality and documentation of qdiscs is centered around egress qdiscs. Egress qdiscs can be configured in a hierarchical tree starting with the root qdisc, whereas there is only one ingress qdisc. I only mentioned this because, as you will see, the syntax for creating the ingress and egress qdiscs is slightly different.

Let’s first create our ingress qdisc on our capture interface eth0:

ubuntu@host-1:~$ sudo tc qdisc add dev eth0 handle ffff: ingress

Confirm that the ingress qdisc was created. Ignore the first qdisc for now which is the default egress qdisc:

ubuntu@host-1:~$ tc -s qdisc ls dev eth0
qdisc fq_codel 0: root refcnt 2 limit 10240p flows 1024 quantum 8990 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
 Sent 600 bytes 4 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
 new_flows_len 0 old_flows_len 0
qdisc ingress ffff: parent ffff:fff1 — — — — — — — — 
 Sent 267532 bytes 3914 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

The next step is to add filters to our qdiscs that will mirror the traffic to our tunnel. There are several tc filters and the syntax can be a little complicated. It is possible to filter on any part of the packet but in this article I’m just going to show a few examples using the u32 filter, such as this one:

ubuntu@host-1:~$ sudo tc filter add dev eth0 parent ffff: protocol ip u32 match ip protocol 1 0xff action mirred egress mirror dev tun0

The above command sets up a filter that matches IP protocol 1 (ICMP) exactly, then mirror the traffic to the tun0 device. 0xff is the protocol mask and you can use it like a netmask to ignore certain bits when making a bitwise comparison. Note that the action mentions ‘mirred egress’. Mirred is the tc method of mirroring packets. Technically it’s possible to mirror packets so they appear on the ingress queue of an interface but for our purposes we want the mirrored packets to be transmitted so we always specify ‘egress’ here.

Most likely you want to mirror all incoming traffic to the tunnel. We can do so with the shorthand ‘match u32 0 0’:

sudo tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress mirror dev tun0

Let’s confirm our filter is set up correctly:

ubuntu@host-1:~$ tc -s -p filter ls dev eth0 parent ffff:
filter protocol all pref 49152 u32 chain 0
filter protocol all pref 49152 u32 chain 0 fh 800: ht divisor 1
filter protocol all pref 49152 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid ??? not_in_hw
 match 00000000/00000000 at 0
 action order 1: mirred (Egress Mirror to device tun0) pipe
 index 1 ref 1 bind 1 installed 494 sec
 Action statistics:
 Sent 20397 bytes 342 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

Let’s say you only want to mirror only SSH traffic (TCP port 22). You can do so with two selectors in the same statement:

ubuntu@host-1:~$ sudo tc filter add dev eth0 parent ffff: protocol ip u32 match ip protocol 6 0xff ip dport 22 0xffff action mirred egress mirror dev tun0

You can delete the filters like this:

ubuntu@host-1:~$ sudo tc filter del dev eth0 parent ffff:

If you are mirroring to a tunnel on the same interface as the capture interface then be sure to exclude tunnel packets to avoid a loop. Unfortunately the filter syntax does not support ‘not’ so you will need to explicitly state what you do want mirrored to the tunnel. You can add several filters to achieve this, e.g.

ubuntu@host-1:~$ sudo tc filter add dev eth0 parent ffff: protocol ip u32 match ip protocol 1 0xff action mirred egress mirror dev tun0ubuntu@host-1:~$ sudo tc filter add dev eth0 parent ffff: protocol ip u32 match ip protocol 6 0xff action mirred egress mirror dev tun0ubuntu@host-1:~$ sudo tc filter add dev eth0 parent ffff: protocol ip u32 match ip protocol 17 0xff action mirred egress mirror dev tun0

The above commands will mirror all ICMP, TCP, and UDP to the tunnel. This will exclude any GRE packets and prevent a loop.

We need to create an additional qdisc and filters if we also want to mirror our egress packets. Recall that there can be many egress qdiscs. We will simply replace the default root egress qdisc on our capture interface eth0. It is customary to name the root qdisc ‘1:’.

Egress qdiscs use queuing algorithms to shape traffic. We will use the PRIO algorithm because it does not shape the traffic in any way — we certainly don’t want to disrupt our host traffic when we mirror it.

ubuntu@host-1:~$ sudo tc qdisc add dev eth0 handle 1: root prio

Verify that the egress qdisc was created. Note that it has replaced the default egress qdisc that we saw earlier:

ubuntu@host-1:~$ tc -s qdisc ls dev eth0
qdisc prio 1: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 600 bytes 4 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc ingress ffff: parent ffff:fff1 — — — — — — — — 
 Sent 274976 bytes 4021 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

And create a filter on the egress qdisc to mirror all outgoing traffic to our tunnel interface:

ubuntu@host-1:~$ sudo tc filter add dev eth0 parent 1: protocol all u32 match u32 0 0 action mirred egress mirror dev tun0

Now check your filter:

ubuntu@host-1:~$ tc -s -p filter ls dev eth0 parent 1:
filter protocol all pref 49152 u32 chain 0
filter protocol all pref 49152 u32 chain 0 fh 800: ht divisor 1
filter protocol all pref 49152 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid ??? not_in_hw
 match 00000000/00000000 at 0
 action order 1: mirred (Egress Mirror to device tun0) pipe
 index 2 ref 1 bind 1 installed 456 sec
 Action statistics:
 Sent 4578 bytes 46 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

Again, remember to filter out your tunnel packets if you are using your capture interface to host your tunnel.

There is a suspected bug with the Linux kernel whereby packets copied directly from the egress queue of the network interface to the tunnel on that same interface causes the operating system to lock-up. The workaround I devised is to first copy the packets to the host loopback interface and then copy the packets to the GRE tunnel. In this case you must also create an ingress qdisc on the loopback interface:

ubuntu@host-1:~$ sudo tc qdisc add dev lo handle ffff: ingress

Now setup a filter on the capture interface egress qdisc to mirror packets to the loopback interface and a filter on the loopback ingress qdisc to mirror packets to the tunnel:

ubuntu@host-1:~$ sudo tc filter add dev eth0 parent 1: protocol all u32 match u32 0 0 action mirred egress mirror dev lo
ubuntu@host-1:~$ sudo tc filter add dev lo parent ffff: protocol all u32 match u32 0 0 action mirred egress mirror dev tun0

Let’s do a quick test to ensure our packet mirror solution is working as designed. I’ve set up ingress and egress qdiscs on my target host ‘host-1’, and I’ve added filters to mirror only ICMP traffic to a tunnel, which will then ship the traffic to a monitoring host ‘host-2’.

ubuntu@host-1:~$ tc -s -p filter ls dev eth0 parent 1:
filter protocol ip pref 49152 u32 chain 0
filter protocol ip pref 49152 u32 chain 0 fh 800: ht divisor 1
filter protocol ip pref 49152 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid ??? not_in_hw
 match IP protocol 1
 action order 1: mirred (Egress Mirror to device tun0) pipe
 index 2 ref 1 bind 1 installed 1748 sec used 1343 sec
 Action statistics:
 Sent 2058 bytes 21 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0ubuntu@host-1:~$ tc -s -p filter ls dev eth0 parent ffff:
filter protocol ip pref 49152 u32 chain 0
filter protocol ip pref 49152 u32 chain 0 fh 800: ht divisor 1
filter protocol ip pref 49152 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid ??? not_in_hw
 match IP protocol 1
 action order 1: mirred (Egress Mirror to device tun0) pipe
 index 1 ref 1 bind 1 installed 1763 sec used 1349 sec
 Action statistics:
 Sent 1764 bytes 21 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

The target host pings its network gateway:

ubuntu@host-1:~$ ping -c3 10.131.73.1
PING 10.131.73.1 (10.131.73.1) 56(84) bytes of data.
64 bytes from 10.131.73.1: icmp_seq=1 ttl=64 time=0.370 ms
64 bytes from 10.131.73.1: icmp_seq=2 ttl=64 time=0.408 ms
64 bytes from 10.131.73.1: icmp_seq=3 ttl=64 time=0.411 ms

Meanwhile on the monitoring host, a tcpdump session is running on the underlying network interface of the layer 2 GRE tunnel:

ubuntu@host-2:~$ sudo tcpdump -n -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
19:08:44.830729 IP 192.168.1.7 > 192.168.1.10: GREv0, length 102: IP 10.131.73.9 > 10.131.73.1: ICMP echo request, id 6168, seq 1, length 64
19:08:44.830914 IP 192.168.1.7 > 192.168.1.10: GREv0, length 102: IP 10.131.73.1 > 10.131.73.9: ICMP echo reply, id 6168, seq 1, length 64
19:08:45.855400 IP 192.168.1.7 > 192.168.1.10: GREv0, length 102: IP 10.131.73.9 > 10.131.73.1: ICMP echo request, id 6168, seq 2, length 64
19:08:45.855674 IP 192.168.1.7 > 192.168.1.10: GREv0, length 102: IP 10.131.73.1 > 10.131.73.9: ICMP echo reply, id 6168, seq 2, length 64
19:08:46.878746 IP 192.168.1.7 > 192.168.1.10: GREv0, length 102: IP 10.131.73.9 > 10.131.73.1: ICMP echo request, id 6168, seq 3, length 64
19:08:46.878976 IP 192.168.1.7 > 192.168.1.10: GREv0, length 102: IP 10.131.73.1 > 10.131.73.9: ICMP echo reply, id 6168, seq 3, length 64

Everything is working as designed. The monitoring host is receiving both the ingress and egress ICMP packets from the target host encapsulated in GRE packets.