Or: How I Learned to Stop Worrying and Love Packet Marking
- Introduction: Why Are We Doing This to Ourselves?
- Policy-Based Routing: Because Normal Routing is Too Mainstream
- Configuration Variables: The Boring But Necessary Stuff
- Address Lists: Teaching Your Router to Play Favorites
- Routing Tables: Now We're Getting Fancy
- Full VRF: When VRF-Lite Isn't Extra Enough
- Route Configuration: Where Packets Go to Party
- Mangle Rules: The Dark Arts of Packet Manipulation
- NAT: Because Nothing Can Ever Be Simple
- Performance: Making It Go Brrrrr
- Common Pitfalls: Learn From My Pain
- Testing: Trust But Verify
Welcome, brave network administrator! You've decided that regular routing—where packets go from A to B like normal, well-adjusted data—is simply too boring. You want policy-based routing, where you get to micromanage every packet's life choices.
Imagine you're a control freak (if you're reading this, you probably are) who wants to:
- Send your streaming traffic through a VPN because geo-restrictions are annoying
- Keep your work traffic on the boring-but-reliable connection
- Make your router's life complicated because... why not?
flowchart LR
A[Your Computer] -->|Normal Traffic| B[Regular ISP]
A -->|Special Traffic| C[VPN/Alt Route]
B --> D[The Internet]
C --> D
style A fill:#800,stroke:#333,stroke-width:2px
style C fill:#008,stroke:#333,stroke-width:2px
Policy-Based Routing: Because Normal Routing is Too Mainstream {#understanding-policy-based-routing}
Let's start with a truth bomb: traditional routing is like a GPS that only knows one route to each destination. It's simple, reliable, and boring as watching paint dry.
graph TD
subgraph "Traditional Routing (Boring)"
A1[Packet arrives] --> B1{Destination?}
B1 -->|8.8.8.8| C1[Always use ISP1]
B1 -->|1.1.1.1| C1
B1 -->|Anywhere| C1
end
subgraph "Policy-Based Routing (Fun!)"
A2[Packet arrives] --> B2{What kind?}
B2 -->|Streaming| C2[Use VPN]
B2 -->|Work| D2[Use ISP1]
B2 -->|Gaming| E2[Use ISP2]
end
Traditional Routing thinks:
- "Oh, you want to go to Netflix? Same route as everything else, buddy!"
Policy-Based Routing thinks:
- "Netflix, you say? Let me check my list... Ah yes, you get the VPN treatment!"
- "Work email? Straight through the corporate connection!"
- "Random browsing? Dealer's choice!"
Good question! Here are some totally reasonable scenarios:
- The "I'm Totally Not Pirating" Scenario: Route certain Linux ISOs through a VPN
- The "My Boss Thinks I'm Working" Setup: Work traffic goes through reliable connection
- The "I Have Two ISPs Because I'm Fancy" Configuration: Spread the load, feel important
- The "This One Service Hates My ISP" Workaround: Route problem children differently
:local mainGatewayIPv4 "YOUR_MAIN_GATEWAY_IPv4"
:local mainGatewayIPv6 "YOUR_MAIN_GATEWAY_IPv6"
:local altGatewayIPv4 "YOUR_ALT_GATEWAY_IPv4"
:local altGatewayIPv6 "YOUR_ALT_GATEWAY_IPv6"
:local altGatewayInterface "YOUR_ALT_INTERFACE"
Look, I know variables are about as exciting as watching grass grow, but here's the thing: without them, you'll be playing find-and-replace with IP addresses at 3 AM when something breaks. And trust me, something will break.
- mainGateway: Your ISP's router (the boring, reliable one)
- altGateway: Your VPN/secondary connection (the fun, mysterious one)
- altGatewayInterface: The actual port/connection name (ether2, pppoe-out1, or whatever creative name you gave it at 2 AM)
Name your interfaces something meaningful. Future-you will thank present-you when you're not staring at "ether2" wondering if that's the VPN or your IoT network.
/ip firewall address-list
add list=putio address=IPv4_ADDRESS_1 comment="Put.io - Streaming Service"
add list=putio address=IPv4_SUBNET/24 comment="Put.io - Entire subnet because I'm lazy"
Address lists are like your router's little black book—except instead of phone numbers, it's IP addresses, and instead of calling them, it routes packets differently.
flowchart TD
subgraph "Address List: 'putio'"
A[151.80.40.155]
B[151.80.40.0/24]
C[192.241.141.0/24]
D[put.io - DNS entry]
end
E[Incoming Packet] -->|Is destination in list?| F{Match?}
F -->|Yes| G[Special Treatment]
F -->|No| H[Normal Routing]
style G fill:#4a4,stroke:#333,stroke-width:2px
style H fill:#a22,stroke:#333,stroke-width:2px
The Good:
- Change IPs once, rules update everywhere
- Named lists are self-documenting (if you're not lazy with names)
- Can mix IPs, subnets, and even DNS names
The Bad:
- DNS entries update on a schedule (not instantly)
- Large lists eat RAM like Chrome eats... well, RAM
The Ugly:
- Forgetting you have overlapping subnets and wondering why your counters are weird
# The "I'll deal with this later" entry
add list=putio address=1.2.3.4 timeout=1h comment="Temporary - Bob's test"
# The "Follow the DNS" entry (updates automatically!)
add list=putio address=sketchy-streaming-site.com comment="Follows DNS changes"
# The "Import from file because I have 500 IPs" method
/import file=my-massive-list.txt
DNS-based entries are like that friend who's always "5 minutes away"—they update when they feel like it, not when you need them to. Plan accordingly.
/routing table
add name=putio-route fib comment="Where streaming dreams come true"
Ah, routing tables—where we segregate our packets like it's network apartheid. But in a good way! Think of it as giving your special traffic its own VIP lane.
graph LR
subgraph "RouterOS Brain"
subgraph "Main Routing Table"
A[Google: -> ISP1]
B[Facebook: -> ISP1]
C[Everything: -> ISP1]
end
subgraph "putio-route Table"
D[Put.io: -> VPN]
E[Everything else: -> VPN]
end
end
F[Marked Packet] -->|Routing Mark| G{Which table?}
G -->|main| A
G -->|putio-route| D
Because sometimes you want to live in two different networking realities simultaneously. It's like having two GPS systems: one for your daily commute and one for your secret midnight food runs.
- FIB: Forwarding Information Base (fancy words for "where packets actually go")
- Isolation: Each table is its own little universe
- Memory: Yes, each table eats some. No, you probably won't notice unless you go crazy
So you've been using our simple routing table approach (VRF-lite), but now you want the full VRF experience? You absolute madlad. Let's do this.
graph TD
subgraph "VRF-Lite (What we did)"
A[Shared Interfaces]
B[Separate Routing Tables]
C[Packet Marking]
A --> B
B --> C
end
subgraph "Full VRF (Going Nuclear)"
D[Dedicated Interfaces]
E[Isolated Routing]
F[Separate Everything]
G[No Cross-Contamination]
D --> E
E --> F
F --> G
end
Ready to go full network isolation? Here's how to implement proper VRF:
# Step 1: Create the VRF instance
/ip vrf
add name=streaming-vrf comment="My streaming paradise"
# Step 2: Assign interfaces to VRF (this is where it gets serious)
/interface ethernet
set ether3 vrf=streaming-vrf comment="Dedicated to VPN connection"
# Step 3: VRF-specific addresses
/ip address
add address=10.99.0.1/24 interface=ether3
# Step 4: VRF-specific DHCP (because why not complicate things further)
/ip dhcp-server
add name=streaming-dhcp interface=ether3 address-pool=streaming-pool
# Step 5: VRF-specific routes
/ip route
add dst-address=0.0.0.0/0 gateway=10.99.0.254 vrf=streaming-vrf
# Step 6: Inter-VRF routing (when you need them to talk)
/ip route
add dst-address=192.168.1.0/24 gateway=streaming-vrf@main
Isolation Level: Maximum
- Interfaces are imprisoned in their VRF
- No accidental packet leakage
- Each VRF is like its own mini-router
Complexity Level: Also Maximum
- Need separate everything (DHCP, DNS, firewall rules)
- Inter-VRF communication requires explicit routes
- Troubleshooting becomes "fun"
flowchart TD
subgraph "Main VRF"
A[LAN: 192.168.1.0/24]
B[WAN: ISP1]
C[Normal Services]
end
subgraph "Streaming VRF"
D[LAN: 10.99.0.0/24]
E[WAN: VPN]
F[Streaming Services]
end
G[Inter-VRF Route] -.->|Explicit Permission| A
G -.-> D
style G fill:#660,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5
Use VRF-Lite (our original approach) when:
- You just want some traffic to go differently
- You trust your firewall rules
- You value your sanity
Use Full VRF when:
- You need actual security isolation
- You're running multi-tenant environments
- You enjoy complexity
- Your therapist says you need more stress in your life
-
The "Why Can't I Ping?" Classic
- Services in different VRFs can't see each other
- Yes, this includes your management access
- Hope you have console access!
-
The "My DNS Broke" Surprise
- Each VRF needs its own DNS config
- Or explicit routes to shared DNS
-
The "Where Did My Route Go?" Mystery
- Routes are VRF-specific
- Check you're in the right VRF context
/ip route
add dst-address=0.0.0.0/0 gateway=$mainGatewayIPv4 distance=1 comment="The boring route"
add dst-address=0.0.0.0/0 gateway=$altGatewayIPv4 distance=1 routing-table=putio-route comment="The fun route"
Routes are like giving directions to a very literal friend. You need to be specific, or they'll end up in Saskatchewan when you meant San Francisco.
graph TD
A[I need to go to 8.8.8.8] --> B{Check Routing Table}
B -->|Main Table| C[Gateway: 192.168.1.1 The Reliable ISP]
B -->|putio-route Table| D[Gateway: 10.0.0.1 The VPN]
C --> E[Destination]
D --> E
style C fill:#060
style D fill:#006
# The "I know the IP" gateway
add gateway=192.168.1.1 comment="Simple and direct"
# The "Just use that interface" gateway
add gateway=ether1 comment="For PPPoE and other weird stuff"
# The "I'm very specific" gateway
add gateway=192.168.1.1%ether1 comment="IP + Interface, because paranoia"
# The "I'll figure it out later" gateway
add gateway=pppoe-out1 comment="Dynamic interfaces"
Distance in routing is like popularity in high school—lower is better, and it's all about preference:
add dst-address=0.0.0.0/0 gateway=192.168.1.1 distance=1 comment="Primary (favorite child)"
add dst-address=0.0.0.0/0 gateway=10.0.0.1 distance=2 comment="Backup (second favorite)"
add dst-address=0.0.0.0/0 gateway=172.16.0.1 distance=10 comment="Last resort (red-headed stepchild)"
Welcome to the mangle table, where we perform unspeakable acts on innocent packets. It's like packet surgery, but without the medical degree.
/ip firewall mangle
# Step 1: Mark the connection (like tagging cattle, but for data)
add chain=prerouting dst-address-list=putio action=mark-connection \
new-connection-mark=putio-conn passthrough=yes comment="Tag, you're it!"
# Step 2: Mark the routing (the actual magic)
add chain=prerouting connection-mark=putio-conn action=mark-routing \
new-routing-mark=putio-route passthrough=no comment="Now go that way!"
sequenceDiagram
participant P as Packet
participant CM as Connection Marker
participant RM as Routing Marker
participant RT as Routing Table
P->>CM: First packet arrives
CM->>CM: Check dst-address-list
CM->>P: Mark connection "putio-conn"
Note over CM: Connection remembered!
P->>RM: Continue to next rule
RM->>RM: See connection mark
RM->>P: Mark routing "putio-route"
Note over RM: Stop processing (passthrough=no)
P->>RT: Use alternate routing table
Note over P: Future packets skip address check!
Packet #1: "Hi, I'm going to Put.io!" Rule 1: "Ah, Put.io! Let me mark your entire connection. There, you're tagged!" Rule 2: "I see you're tagged. Here's your VIP routing mark. Off you go!"
Packet #2 (same connection): "Hi, I'm also going to Put.io!" Rule 1: "No need to check the address list, your connection is already marked!" Rule 2: "Tagged connection? VIP routing for you too!"
This is why we love connection marking—it's lazy in the best way possible.
flowchart LR
A[Packet enters] --> B{Rule 1<br/>passthrough=yes}
B -->|Green light| C{Rule 2<br/>passthrough=no}
C -->|Red light| D[Stop here]
C -.->|Would continue if yes| E{Rule 3}
style C fill:#060
style D fill:#c00
style E fill:#c00,stroke-dasharray: 5 5
- passthrough=yes: "Mark it and keep going, there might be more rules!"
- passthrough=no: "Mark it and STOP. We're done here."
- The "Wrong Chain" Classic
# WRONG - Forward chain can't do routing marks
add chain=forward action=mark-routing # Won't work!
# RIGHT - Prerouting for routing marks
add chain=prerouting action=mark-routing # Works!
- The "Order Matters" Disaster
# WRONG ORDER - Routing mark before connection mark
add chain=prerouting action=mark-routing # How do you know what to mark?
add chain=prerouting action=mark-connection # Too late!
- The "Forgotten Return Traffic" Special
# INCOMPLETE - Only marks destination
add chain=prerouting dst-address-list=putio action=mark-connection
# COMPLETE - Marks both directions
add chain=prerouting dst-address-list=putio action=mark-connection
add chain=prerouting src-address-list=putio action=mark-connection
/ip firewall nat
add chain=srcnat out-interface=$altGatewayInterface action=masquerade \
comment="Hide behind the VPN like a network ninja"
NAT is like wearing a disguise to a party. Sometimes you need it, sometimes you don't, but when you need it and don't have it, you're not getting in.
sequenceDiagram
participant L as LAN Device<br/>192.168.1.100
participant R as Router
participant V as VPN<br/>10.0.0.1
participant I as Internet
L->>R: From: 192.168.1.100<br/>To: Put.io
Note over R: NAT Translation
R->>V: From: 10.0.0.5<br/>To: Put.io
V->>I: Forwarded
I->>V: Reply to: 10.0.0.5
V->>R: Reply to: 10.0.0.5
Note over R: NAT Reverse Translation
R->>L: Reply to: 192.168.1.100
- VPN Connections: They expect traffic from their assigned IP, not your LAN
- Mobile Internet: Your 4G provider doesn't care about your 192.168.x.x
- That One ISP: Who refuses to route your LAN without paying extra
- Cloud VPS: When you're bouncing through a remote server
# Masquerade: The "I'm flexible" approach
action=masquerade # Uses whatever IP the interface has
# SNAT: The "I know what I want" approach
action=src-nat to-addresses=203.0.113.1 # Always use this IP
Choose Masquerade when:
- Your IP changes (DHCP, PPPoE)
- You're lazy (it's okay, we all are)
- You enjoy the mystery
Choose SNAT when:
- You have a static IP
- You need that 0.01% performance boost
- You enjoy typing more
graph TD
A[Common NAT Problems] --> B[Double NAT]
A --> C[NAT Hairpinning]
A --> D[Breaking Protocols]
B --> E[Performance Dies]
C --> F[LAN Can't Access Services]
D --> G[FTP/SIP/Others Cry]
style E fill:#c00
style F fill:#c00
style G fill:#c00
/ip firewall raw
add chain=prerouting dst-address-list=putio action=notrack \
comment="Bypass connection tracking because SPEED"
Performance optimization is like tuning a car—you can make it faster, but you might lose your air conditioning.
flowchart TD
subgraph "Normal Flow (Slow but Safe)"
A1[Packet] --> B1[Connection Tracking]
B1 --> C1[Mangle]
C1 --> D1[Routing]
D1 --> E1[Filter]
E1 --> F1[Forward]
end
subgraph "Notrack Flow (Fast but Reckless)"
A2[Packet] --> B2[Raw: NOTRACK]
B2 --> C2[Mangle]
C2 --> D2[Routing]
D2 --> F2[Forward]
end
style B1 fill:#c00
style B2 fill:#060
Perfect for:
- Netflix binges (unidirectional, high bandwidth)
- That Linux ISO collection (definitely legitimate)
- Any bulk transfer where you trust both ends
Terrible for:
- Anything needing NAT
- Stateful firewall rules
- Connection limits
- Accounting (unless you like mystery)
graph LR
A[Slowest] --> B[Connection Tracking]
B --> C[FastTrack]
C --> D[Hardware Offload]
D --> E[Notrack]
E --> F[Fastest]
style A fill:#c00
style F fill:#060
Choose your fighter based on your needs:
- Full Tracking: Maximum features, maximum overhead
- FastTrack: Best of both worlds for established connections
- Hardware Offload: If your hardware supports it (lucky you!)
- Notrack: YOLO mode
graph LR
A[Your PC] -->|Upload via VPN| B[VPN Server]
C[Put.io] -->|Download via ISP| D[Main ISP]
B --> C
D --> A
style B fill:#060
style D fill:#c00
The Problem: Your uploads go through VPN, downloads come back through regular ISP. TCP has a mental breakdown.
The Fix: Connection marking saves the day by remembering the path.
The Scene: Everything's routing perfectly, except... nothing works. The Plot Twist: DNS queries are using the wrong path.
# The "Oh right, DNS exists" fix
add chain=output protocol=udp dst-port=53 dst-address-list=putio \
action=mark-routing new-routing-mark=putio-route \
comment="Because DNS is important, apparently"
You: "The routing works perfectly!" Also you: "But ping doesn't work..." ICMP: "Did everyone forget about me?"
# The "ICMP needs love too" solution
add chain=prerouting protocol=icmp dst-address-list=putio \
action=mark-routing new-routing-mark=putio-route \
comment="ICMP: The forgotten protocol"
sequenceDiagram
participant U as User
participant R as Router
participant V as VPN (Dead)
participant I as Internet
U->>R: Send packet to Put.io
R->>V: Route via VPN
Note over V: VPN is down!
R->>V: Still trying...
R->>V: Still trying...
U->>U: Why is nothing working?
The Fix: Check-gateway to the rescue!
add dst-address=0.0.0.0/0 gateway=$altGatewayIPv4 \
routing-table=putio-route check-gateway=ping \
comment="Actually check if the gateway is alive"
Because "it should work" is not a valid troubleshooting step.
# 1. The "Is my address list working?" check
/ip firewall address-list print where list=putio
# 2. The "Are packets getting marked?" investigation
/ip firewall mangle print stats where action=mark-routing
# 3. The "Where are my packets going?" tracker
/tool traceroute put.io routing-table=putio-route
# 4. The "Show me everything" debug mode
/ip firewall mangle
add chain=prerouting dst-address-list=putio \
action=log log-prefix="PUTIO-DEBUG:" \
comment="Temporary debugging - REMOVE ME"
flowchart TD
A[Routing Not Working] --> B{Packets Marked?}
B -->|No| C[Check Mangle Rules]
B -->|Yes| D{Route Exists?}
C --> E[Check Address List]
C --> F[Check Rule Order]
D -->|No| G[Check Routing Table]
D -->|Yes| H{Gateway Up?}
H -->|No| I[Check Gateway/Interface]
H -->|Yes| J[Check NAT/Firewall]
style A fill:#c00
style J fill:#060
- Start Simple: Test with one IP before adding entire subnets
- Use Logging: But remember to remove it (your disk will thank you)
- Test Both Directions: Upload and download, ping and pong
- Document What Works: Future you will appreciate it
-
Thou Shalt Comment Everything
# BAD add list=putio address=1.2.3.4 # GOOD add list=putio address=1.2.3.4 comment="Put.io CDN - US East servers" -
Thou Shalt Not Overlap Subnets Unnecessarily
- Having both 10.0.0.0/8 and 10.0.0.0/24 in the same list is just wasteful
-
Thou Shalt Test Before Production
- Your users don't appreciate being your guinea pigs
-
Thou Shalt Have a Rollback Plan
- Export your config before making changes
- Know how to access your router when you break routing
-
Thou Shalt Monitor Thy Creation
- Set up logging during testing
- Remove excessive logging in production
- Keep an eye on connection counts
- Start Simple: Get basic routing working before adding complexity
- Document Everything: Your successor (probably you in 6 months) will thank you
- Test Incrementally: Add one feature at a time
- Have Console Access: Because you will lock yourself out eventually
Congratulations! You've just learned how to make your router's life significantly more complicated. You now have the power to route packets based on their hopes, dreams, and destination addresses.
Remember:
- With great routing power comes great troubleshooting responsibility
- Every rule you add is one more thing that can break
- But hey, at least your streaming works through the VPN now!
Now go forth and route selectively. May your packets find their way, and may your connection marks never expire unexpectedly.
P.S. - If everything breaks, you can always reset to defaults and pretend this never happened. We won't judge.