Linux Interface Latency Testing

NETWORK_LATENCY_TESTING.md

Linux Interface Latency Testing

This is a general reference for introducing artificial network latency on Linux when testing how a protocol behaves as latency increases.

The standard tool is tc with the netem qdisc.

Overview

• Egress delay on an interface is straightforward with tc netem.
• Ingress delay is usually handled by redirecting ingress traffic to an ifb device and applying netem there.
• If you apply the same one-way delay to both ingress and egress on one host, the added round-trip time is approximately 2 x delay.
• These examples require root privileges.
• These examples overwrite existing tc qdisc state on the target interface.

Basic Egress Delay

Apply a fixed delay to packets leaving eth0:

sudo tc qdisc add dev eth0 root netem delay 50ms

Change it later:

sudo tc qdisc change dev eth0 root netem delay 100ms

Remove it:

sudo tc qdisc del dev eth0 root

Basic Ingress Delay With ifb

Linux does not directly delay ingress in the same way as egress. A common pattern is:

1. Create an ifb device.
2. Redirect ingress traffic from the real interface to the ifb.
3. Apply netem to the ifb.

Example:

sudo modprobe ifb
sudo ip link add ifb0 type ifb
sudo ip link set ifb0 up

sudo tc qdisc add dev eth0 handle ffff: ingress
sudo tc filter add dev eth0 parent ffff: protocol all u32 \
  match u32 0 0 action mirred egress redirect dev ifb0

sudo tc qdisc add dev ifb0 root netem delay 50ms

Remove it:

sudo tc qdisc del dev eth0 ingress
sudo tc qdisc del dev ifb0 root
sudo ip link delete ifb0

If you have a peer machine available, another option is to apply egress delay on the peer instead of emulating ingress locally.

Command Breakdown

These examples use Linux traffic control terminology that is not very intuitive at first glance.

• tc: traffic control, the Linux command used to inspect and modify packet scheduling and shaping.
• qdisc: queueing discipline, the packet queue/shaper attached to a network interface.
• netem: the qdisc used to emulate network effects such as delay, loss, duplication, corruption, and reordering.
• dev eth0: apply the rule to interface eth0.
• root: attach the qdisc to the main outbound transmit path of the interface.
• ingress: attach a special receive-side hook for packets arriving on the interface.
• ifb: Intermediate Functional Block, a virtual interface used to redirect ingress traffic so it can be shaped like egress traffic.

Example:

sudo tc qdisc add dev eth0 root netem delay 50ms

Meaning:

• qdisc add: create and attach a new queueing discipline.
• dev eth0: attach it to eth0.
• root: place it on the normal outbound path.
• netem: use the network emulator.
• delay 50ms: hold packets for 50 milliseconds before sending.

Ingress needs a few more parts:

sudo tc qdisc add dev eth0 handle ffff: ingress
sudo tc filter add dev eth0 parent ffff: protocol all u32 \
  match u32 0 0 action mirred egress redirect dev ifb0
sudo tc qdisc add dev ifb0 root netem delay 50ms

Meaning:

• handle ffff:: give the ingress qdisc an identifier so other tc rules can refer to it.
• parent ffff:: attach the filter to that ingress qdisc.
• protocol all: match all traffic.
• u32 match u32 0 0: a simple match-all filter expression.
• action mirred egress redirect dev ifb0: redirect packets to ifb0 as if they were leaving that interface.
• qdisc add dev ifb0 root netem delay 50ms: apply delay on the virtual interface after redirection.

The important idea is that Linux shapes outbound traffic naturally, so ingress traffic is redirected to an ifb device and then shaped there.

Simulating Packet Loss

tc netem can also simulate packet loss in addition to delay.

Apply packet loss on egress:

sudo tc qdisc add dev eth0 root netem loss 1%

Apply both delay and packet loss on egress:

sudo tc qdisc add dev eth0 root netem delay 100ms loss 2%

Change an existing rule:

sudo tc qdisc change dev eth0 root netem delay 100ms loss 5%

Remove it:

sudo tc qdisc del dev eth0 root

For ingress, use the same ifb setup described above and apply loss on the ifb device:

sudo tc qdisc add dev ifb0 root netem loss 2%

You can also combine multiple impairments:

• loss 2%
• duplicate 1%
• corrupt 0.5%
• reorder 10% 50%

Example:

sudo tc qdisc add dev eth0 root netem delay 80ms loss 3% duplicate 0.2%

Script: Ramp Delay Over Time

This script gradually increases one-way delay on both ingress and egress for a specific interface.

#!/usr/bin/env bash
set -euo pipefail

IFACE="${1:-eth0}"
START_MS="${2:-0}"
END_MS="${3:-500}"
STEP_MS="${4:-50}"
HOLD_SEC="${5:-10}"
IFB="${6:-ifb0}"

if [[ $EUID -ne 0 ]]; then
  echo "Run as root, e.g. sudo $0 $*" >&2
  exit 1
fi

cleanup() {
  tc qdisc del dev "$IFACE" root 2>/dev/null || true
  tc qdisc del dev "$IFACE" ingress 2>/dev/null || true
  tc qdisc del dev "$IFB" root 2>/dev/null || true
  ip link set "$IFB" down 2>/dev/null || true
  ip link delete "$IFB" type ifb 2>/dev/null || true
}

trap cleanup EXIT INT TERM

modprobe ifb
modprobe sch_netem 2>/dev/null || true

ip link add "$IFB" type ifb 2>/dev/null || true
ip link set "$IFB" up

tc qdisc del dev "$IFACE" root 2>/dev/null || true
tc qdisc del dev "$IFACE" ingress 2>/dev/null || true
tc qdisc del dev "$IFB" root 2>/dev/null || true

tc qdisc add dev "$IFACE" root netem delay "${START_MS}ms"
tc qdisc add dev "$IFACE" handle ffff: ingress
tc filter add dev "$IFACE" parent ffff: protocol all u32 \
  match u32 0 0 action mirred egress redirect dev "$IFB"
tc qdisc add dev "$IFB" root netem delay "${START_MS}ms"

for ((ms=START_MS; ms<=END_MS; ms+=STEP_MS)); do
  echo "Applying ${ms}ms one-way delay on $IFACE ingress+egress (~$((ms * 2))ms added RTT)"
  tc qdisc replace dev "$IFACE" root netem delay "${ms}ms"
  tc qdisc replace dev "$IFB" root netem delay "${ms}ms"
  sleep "$HOLD_SEC"
done

Example:

sudo ./latency-ramp.sh eth0 0 500 50 15

Argument meanings:

• eth0: target interface
• 0: starting delay in milliseconds
• 500: ending delay in milliseconds
• 50: step size in milliseconds
• 15: hold each step for 15 seconds

Script: Set A Constant Delay

This version applies a fixed one-way delay to both ingress and egress on a given interface and leaves it in place until cleared.

#!/usr/bin/env bash
set -euo pipefail

usage() {
  cat <<'EOF'
Usage:
  sudo ./latency.sh set <iface> <delay_ms> [ifb]
  sudo ./latency.sh clear <iface> [ifb]

Examples:
  sudo ./latency.sh set eth0 75
  sudo ./latency.sh clear eth0

Notes:
  - Delay is applied to both ingress and egress.
  - 75ms here means about 150ms of added RTT.
  - This overwrites any existing tc qdisc on the target interface.
EOF
}

cleanup_latency() {
  local iface="$1"
  local ifb="$2"

  tc qdisc del dev "$iface" root 2>/dev/null || true
  tc qdisc del dev "$iface" ingress 2>/dev/null || true
  tc qdisc del dev "$ifb" root 2>/dev/null || true
  ip link set "$ifb" down 2>/dev/null || true
  ip link delete "$ifb" type ifb 2>/dev/null || true
}

if [[ $EUID -ne 0 ]]; then
  echo "Run as root." >&2
  usage
  exit 1
fi

action="${1:-}"

case "$action" in
  set)
    iface="${2:-}"
    delay_ms="${3:-}"
    ifb="${4:-ifb0}"

    if [[ -z "$iface" || -z "$delay_ms" ]]; then
      usage
      exit 1
    fi

    modprobe ifb
    modprobe sch_netem 2>/dev/null || true

    cleanup_latency "$iface" "$ifb"

    ip link add "$ifb" type ifb
    ip link set "$ifb" up

    tc qdisc add dev "$iface" root netem delay "${delay_ms}ms"
    tc qdisc add dev "$iface" handle ffff: ingress
    tc filter add dev "$iface" parent ffff: protocol all u32 \
      match u32 0 0 action mirred egress redirect dev "$ifb"
    tc qdisc add dev "$ifb" root netem delay "${delay_ms}ms"

    echo "Applied ${delay_ms}ms one-way delay to ingress+egress on ${iface}."
    echo "Approximate added RTT: $((delay_ms * 2))ms"
    ;;

  clear)
    iface="${2:-}"
    ifb="${3:-ifb0}"

    if [[ -z "$iface" ]]; then
      usage
      exit 1
    fi

    cleanup_latency "$iface" "$ifb"
    echo "Cleared latency shaping on ${iface}."
    ;;

  *)
    usage
    exit 1
    ;;
esac

Example:

sudo ./latency.sh set eth0 100
sudo ./latency.sh clear eth0

Notes

• modprobe ifb may be required if the ifb module is not already loaded.
• modprobe sch_netem may be required on systems where the netem scheduler is not already available.
• The example scripts in this document only automate delay. Packet loss can be added by extending the netem arguments in the same places where delay is configured.
• If you need to target only specific IPs or ports instead of all interface traffic, tc filters can be used to narrow the scope.