Hello. Recently I encountered a strange routing issue.
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP group default qlen 1000
link/ether 00:1e:42:19:31:51 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-lan state DOWN group default qlen 1000
link/ether 00:1e:42:19:31:52 brd ff:ff:ff:ff:ff:ff
4: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default
link/tunnel6 :: brd ::
5: ifb0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
link/ether 02:4e:89:11:56:7a brd ff:ff:ff:ff:ff:ff
6: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
link/ether de:36:7c:42:b9:80 brd ff:ff:ff:ff:ff:ff
7: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default
link/gre 0.0.0.0 brd 0.0.0.0
8: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: ip6gre0@NONE: <NOARP> mtu 1448 qdisc noop state DOWN group default
link/[823] 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
11: wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
link/ether 5a:55:60:60:b3:39 brd ff:ff:ff:ff:ff:ff
inet 172.28.0.3/29 brd 172.28.0.7 scope global wwan0
valid_lft forever preferred_lft forever
inet6 fe80::5855:60ff:fe60:b339/64 scope link
valid_lft forever preferred_lft forever
12: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 00:1e:42:19:31:51 brd ff:ff:ff:ff:ff:ff
inet 172.21.21.1/24 brd 172.21.21.255 scope global br-lan
valid_lft forever preferred_lft forever
inet 172.21.21.254/24 brd 172.21.21.255 scope global secondary br-lan
valid_lft forever preferred_lft forever
13: tun_c_a: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
link/none
inet 192.168.21.3 peer 192.168.21.1/32 scope global tun_c_a
valid_lft forever preferred_lft forever
# ip r
default via 172.28.0.4 dev wwan0 proto static src 172.28.0.3
default via 172.21.21.253 dev br-lan proto static metric 10
10.0.0.0/8 via 192.168.21.1 dev tun_c_a
172.16.0.0/12 via 192.168.21.1 dev tun_c_a
172.21.21.0/24 dev br-lan proto kernel scope link src 172.21.21.1
172.28.0.0/29 dev wwan0 proto kernel scope link src 172.28.0.3
172.28.0.4 dev wwan0 proto static scope link src 172.28.0.3
192.168.0.0/16 via 192.168.21.1 dev tun_c_a
192.168.5.0/24 via 172.21.21.253 dev br-lan proto static
192.168.21.0/24 via 192.168.21.1 dev tun_c_a
192.168.21.1 dev tun_c_a proto kernel scope link src 192.168.21.3
# traceroute -n 10.10.0.111
traceroute to 10.10.0.111 (10.10.0.111), 30 hops max, 38 byte packets
1 10.126.2.1 701.872 ms 179.483 ms 179.861 ms
2 192.168.10.206 249.848 ms 229.637 ms 289.779 ms
3 192.168.10.205 159.924 ms 239.324 ms 219.833 ms
4 192.168.22.77 439.599 ms 239.680 ms 249.756 ms
5 192.168.22.78 249.748 ms 209.680 ms 229.558 ms
6 10.10.0.111 249.965 ms * 198.709 ms
Although a route to 10.0.0.0/8 exists specifically thru vpn connection 192.168.21.1, the kernel uses default route, which is strange.
I can reboot the system and the correct working will start, but as this is a production environment which should work 24/24 without problem, my question is to solve the issue so it will not happen again.
Thanks.