Skip to content

Policy routing for VPC-like network environments

Noah Meyerhans requested to merge noahm/debian-cloud-images:policy-routes into master

This is a first stab at addressing https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963826

In their default configuration, AWS VPC networks, and other networks with similar anti-spoofing protection, enforce that traffic with a given source IP must egress via the network interface associated with that address. This violates standard routing assumptions that any interface is valid as long as the destination address is reachable via that interface, regardless of the source address chosen. In order to work around this, we deploy policy routing.

The change is implemented by extending our existing ifupdown helper and providing dhclient hook fragments. For secondary interfaces, we create a per-interface routing table and define a routing rule to select that table based on the source address of an outbound packet. For the primary interface, which we define as being the one with IFINDEX==2 according to udev, we do not configure an alternate table or rule, and instead use the default table.

For example, on an EC2 instance with two interfaces, each of which has an IPv4 and IPv6 address, we see the following:

Addresses

admin@ip-10-0-0-129:~$ ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    altname enp0s5
    inet 10.0.0.129/24 brd 10.0.0.255 scope global dynamic ens5
       valid_lft 3371sec preferred_lft 3371sec
3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    altname enp0s6
    inet 10.0.0.204/24 brd 10.0.0.255 scope global dynamic ens6
       valid_lft 1884sec preferred_lft 1884sec

default route table

ip -4 route
admin@ip-10-0-0-129:~$ ip -4 ro
default via 10.0.0.1 dev ens5
default via 10.0.0.1 dev ens6 metric 3
10.0.0.0/24 dev ens5 proto kernel scope link src 10.0.0.129
10.0.0.0/24 dev ens6 proto kernel scope link src 10.0.0.204

policy rules matching traffic with a source address of ens6

admin@ip-10-0-0-129:~$ ip -4 rule
0:      from all lookup local
10003:  from 10.0.0.204 lookup 10003
32766:  from all lookup main
32767:  from all lookup default

route table 10003

admin@ip-10-0-0-129:~$ ip -4 ro sh table 10003
default via 10.0.0.1 dev ens6 src 10.0.0.204
10.0.0.0/24 dev ens6 scope link src 10.0.0.204

And the IPv6 case

Addresses

admin@ip-10-0-0-129:~$ ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 state UP qlen 1000
    inet6 2600:1f14:eeb:2203:454a:fcbd:485d:73ef/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::b9:63ff:fe80:7d53/64 scope link
       valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 state UP qlen 1000
    inet6 2600:1f14:eeb:2203:ac24:6708:2ab9:75a3/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::ad:c7ff:fe1a:7486/64 scope link
       valid_lft forever preferred_lft forever

default route table

admin@ip-10-0-0-129:~$ ip -6 ro
::1 dev lo proto kernel metric 256 pref medium
2600:1f14:eeb:2203:454a:fcbd:485d:73ef dev ens5 proto kernel metric 256 pref medium
2600:1f14:eeb:2203:ac24:6708:2ab9:75a3 dev ens6 proto kernel metric 256 pref medium
2600:1f14:eeb:2203::/64 dev ens5 proto kernel metric 256 pref medium
2600:1f14:eeb:2203::/64 dev ens6 proto kernel metric 256 pref medium
fe80::/64 dev ens6 proto kernel metric 256 pref medium
fe80::/64 dev ens5 proto kernel metric 256 pref medium
default via fe80::8f:53ff:fe6f:64ef dev ens5 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium
default via fe80::8f:53ff:fe6f:64ef dev ens6 proto ra metric 1024 expires 1790sec hoplimit 64 pref medium

policy rules

admin@ip-10-0-0-129:~$ ip -6 rule
0:      from all lookup local
10003:  from 2600:1f14:eeb:2203:ac24:6708:2ab9:75a3 lookup 10003
32766:  from all lookup main

route table 10003

admin@ip-10-0-0-129:~$ ip -6 ro sh table 10003
default via fe80::8f:53ff:fe6f:64ef dev ens6 metric 1024 pref medium

I've tested various scenarios including attach/detach/reboot, etc.

The main drawback with this approach is that it couples us more tightly to dhclient.

If we do decide to use this implementation, it is worth discussing whether we want this maintained in this repository or if it'd be better in a package. Updating it here is a little easier for us, but a package is likely easy for our users, and allows the use of this change outside our images (assuming we make the ifupdown helper available in a package, too)

Merge request reports