klaster/docs/lte-failover-design.md

LTE Failover Design

Reference documentation of the as-built LTE failover design. For day-to-day network overview see network.md; for BM806C modem firmware workarounds see wwan-bm806c-qmi-workaround.md.

Summary

Property	Value
Failover signalling	Symmetric iBGP between D-Link (BIRD2) and CRS (RouterOS)
BGP AS	65000 (iBGP; CRS acts as route reflector for D-Link)
LTE transit path	D-Link wwan ← VLAN 6 (192.168.6.0/24) ← CRS
D-Link default route source	Learned from CRS via BGP (no static default gateway)
CRS LTE route source	Learned from D-Link via BGP at distance 200
Announcement trigger	wwan interface up/down tracked by BIRD2 device protocol
Scope	All internet-capable VLANs (vlan2, vlan4, vlan5, vlan6)
IPv4 NAT	CRS masquerades on pppoe-gpon only; D-Link masquerades on wwan
IPv6 NAT	D-Link masquerades IPv6 on wwan (no inbound on LTE; outbound only)
wwan bringup	Triggered by /etc/init.d/wwan-bringup after USB re-auth (BM806C wedge fix)

Route exchange

CRS announces to D-Link

Prefix	Source	Withdrawn when
0.0.0.0/0	output.default-originate: if-installed (active default in main table)	GPON drops or pppoe-gpon route inactive
2000::/3	output.redistribute: static (HE tunnel default)	sit1 interface down / HE route inactive
VLAN subnets (192.168.0.0/24, 192.168.1.0/24, etc.)	output.redistribute: connected	never (CRS always reachable on vlan6)
100.64.0.0/10 (Tailscale)	output.redistribute: static	never
172.17.0.0/16 (dockers bridge)	output.redistribute: connected	never
10.42.0.0/16, 10.43.0.0/16, 10.44.0.0/16 (k8s)	reflected via iBGP RR	when k8s BGP session drops
pod/service/LB IPv6 ranges	reflected via iBGP RR	when k8s BGP session drops

Internal prefixes are announced regardless of GPON state. They remain reachable via 192.168.6.1 (directly connected on vlan6) even when GPON fails, so D-Link-originated traffic to internal subnets always routes to CRS rather than incorrectly exiting via wwan.

The CRS route reflector role (local.role: ibgp-rr on the dlink-lte connection) allows it to reflect routes learned from the k8s peer (bgp1) to D-Link without violating iBGP split-horizon. RFC 4456 ORIGINATOR_ID loop prevention is handled automatically by RouterOS — no output filter needed.

nexthop-choice: force-self ensures CRS advertises 192.168.6.1 as the next-hop for all prefixes, rather than the original route's next-hop (which may be unreachable from D-Link, e.g. k8s peer 2001:470:61a3:100::3).

D-Link announces to CRS

Prefix	Source	Withdrawn when
0.0.0.0/0	BIRD2 static lte_default via wwan0	wwan0 down (device protocol detects)
2000::/3	BIRD2 static lte_default6 via wwan0	wwan0 down

BIRD2's protocol device tracks wwan0 via netlink in real time; when the interface goes down the static routes become unreachable and BGP withdraws the announcements immediately.

The BIRD2 static routes use preference 50 (below the BGP default of 100) so the BGP-learned routes from CRS are preferred for kernel installation on D-Link itself — D-Link's own outbound traffic uses the CRS path when GPON is up. The static routes only exist as triggers for BGP export.

D-Link kernel routing table

Destination	Source	Kernel metric	Active when
Internal prefixes (VLANs, k8s, Tailscale)	BGP from CRS, via 192.168.6.1	10 (IPv4) / 32 (IPv6)	always (CRS reachable)
0.0.0.0/0	BGP from CRS	10	GPON up
0.0.0.0/0	wwan QMI-assigned (qmi.sh)	100	wwan up
default via wwan IPv6 GW (non-source-specific)	wwan-bringup script	1024	wwan up
default from <wwan prefix>/64 via wwan IPv6 GW (source-specific)	qmi.sh	100	wwan up

D-Link's own outbound traffic prefers the BGP route (metric 10) over wwan (metric 100). The non-source-specific IPv6 default at metric 1024 exists because qmi.sh only installs a source-specific IPv6 default (constrained to the wwan-assigned /64 prefix); forwarded traffic from internal subnets would fail routing lookup with "net unreachable" without it.

CRS routing table

Destination	Source	Distance	Active when
0.0.0.0/0	static via pppoe-gpon	1	GPON up
0.0.0.0/0	BGP from D-Link via 192.168.6.2	200	wwan up on D-Link
2000::/3	static via sit1 (HE tunnel)	1	sit1 active (HE tunnel works)
2000::/3	BGP from D-Link via 2001:470:61a3:600::2	200	wwan up on D-Link

RouterOS distance comparison is straightforward: distance 1 always wins over distance 200. BGP-learned routes activate automatically when the static route becomes inactive (e.g. GPON down → pppoe-gpon route inactive → BGP route at distance 200 becomes active).

Traffic paths

Normal (GPON up)

LAN/SRV/IoT  →  CRS  →  pppoe-gpon  →  ISP
D-Link own   →  uplink  →  CRS  →  pppoe-gpon  →  ISP
                (via BGP-learned default at kernel metric 10)

wwan is connected and D-Link announces the LTE default to CRS, but CRS ignores it (distance 200 loses to distance 1). D-Link uses the CRS-announced default (metric 10) for its own traffic, not wwan (metric 100).

Failover (GPON down)

LAN/SRV/IoT  →  CRS  →  vlan6 (→192.168.6.2)  →  D-Link  →  wwan  →  Orange LTE
D-Link own   →  wwan  →  Orange LTE

CRS distance-1 routes go inactive → distance-200 BGP routes from D-Link activate. D-Link receives forwarded traffic on uplink, routes it via the non-source-specific wwan default (metric 1024), fw4 masquerades the source, packet exits via wwan. Return traffic reverses through masquerade state and forwards back to CRS via the established connection-tracking entry.

When CRS withdraws its BGP-announced default to D-Link (because GPON is down and CRS has no default to announce), D-Link's kernel default at metric 10 is removed, leaving the wwan default at metric 100 as the preferred route for D-Link's own traffic.

Failure detection

• D-Link crashes / power loss → BGP session drops after hold-time: 30s → CRS withdraws all D-Link-learned routes → internet unavailable if GPON also down (acceptable single-point-of-failure for home network)
• wwan modem goes down → BIRD2 device protocol detects wwan0 down → static lte_default / lte_default6 become unreachable → BGP withdraws announcements → CRS removes BGP-learned default
• GPON drops → pppoe-gpon interface down → CRS distance-1 default route inactive → distance-200 BGP route activates → CRS withdraws its default-originate announcement to D-Link (since no default is installed any more) → D-Link's kernel default-via-CRS is removed → D-Link uses wwan kernel default → traffic flows from CRS via vlan6 → D-Link → wwan

All transitions are automatic and driven by interface state. No active probing (Netwatch / mwan3), no scripts toggling routes.

NAT rules

NAT rules are always active, matched by output interface. No failover-triggered toggling needed.

CRS (RouterOS)

• IPv4 masquerade on srcnat chain with out-interface: pppoe-gpon. Only the GPON public interface gets masqueraded — vlan6 is internal and never natted, sit1 (IPv6) has its own dedicated src-nat for the Tailscale prefix.
• IPv6 src-nat tailnet to internet on srcnat chain for Tailscale prefix (fd7a:115c:a1e0::/48) to 2001:470:61a3:600::/64, applied on out-interface-list: wan. Fires regardless of whether the egress is sit1 or vlan6.

D-Link (OpenWrt fw4)

• wwan zone has option masq '1' and option masq6 '1'. All traffic exiting via wwan (own outbound + forwarded from uplink) is source-NAT'd, IPv4 to the wwan-assigned CG-NAT IP, IPv6 to the wwan-assigned /128 from the Orange-assigned /64 prefix.
• Forwarding rule uplink → wwan allows MikroTik-routed traffic to egress via wwan during failover. Default forward policy on the wwan zone stays REJECT.

BGP / route reflection details

CRS connection config

/routing/bgp/connection set dlink-lte \
  remote.address=192.168.6.2/32 \
  local.role=ibgp-rr \
  nexthop-choice=force-self \
  output.redistribute=connected,static \
  output.default-originate=if-installed \
  hold-time=30s keepalive-time=10s

output.default-originate=if-installed is required for the 0.0.0.0/0 advertisement because RouterOS does not advertise interface-gateway static routes (gateway=pppoe-gpon) via plain output.redistribute=static. default-originate advertises a synthetic default whenever any active default exists in the routing table, regardless of how it was installed.

IPv6 Extended Next Hop workaround

RouterOS uses BGP Extended Next Hop Encoding (RFC 5549 / RFC 8950) for IPv6 routes on this iBGP session, advertising them with an IPv4-mapped next-hop (::ffff:192.168.6.1). The Linux kernel does not support installing IPv6 routes with IPv4 next-hops, so BIRD2 cannot push them directly to the kernel.

There is no way to disable ENHE on RouterOS — local.address, nexthop-choice: force-self, and output set gw filters all fail to override it. The workaround is on the BIRD2 side: an import filter on the BGP IPv6 channel rewrites gw to CRS's native IPv6 address (2001:470:61a3:600::1) before the route is exported to the kernel.

ipv6 {
  extended next hop yes;
  import filter {
    gw = 2001:470:61a3:600::1;
    accept;
  };
  ...
};

The reverse direction (D-Link → CRS) was solved cleanly via BIRD2 export filter setting bgp_next_hop = 2001:470:61a3:600::2, since BGP-level attribute manipulation isn't constrained by kernel limitations.

Direct protocol on D-Link

BIRD2 needs to know about the directly connected 192.168.6.0/24 and 2001:470:61a3:600::/64 subnets on eth0.6 to resolve BGP next-hops. The protocol direct { interface "eth0.6"; } declaration provides this; without it BIRD2 marks all CRS-learned routes as unreachable.

BM806C modem cold-boot wedge

The BM806C firmware enters a permanently broken state on cold boot: /dev/cdc-wdm0 exists, kernel driver attaches, but uqmi commands return "Failed to connect to service" indefinitely. UIM (SIM) QMI service specifically never comes up.

Recovery requires a USB device re-enumeration. The /etc/init.d/wwan-bringup service writes 0 then 1 to /sys/bus/usb/devices/1-1/authorized on boot, then triggers ifup wwan. After re-auth the modem completes its QMI initialization within ~1 second.

Full investigation: see wwan-bm806c-qmi-workaround.md.

Implementation files

File	Role
ansible/roles/routeros/tasks/base.yml	vlan6 in wan interface list
ansible/roles/routeros/tasks/routing.yml	BGP instance, template, dlink-lte connection
ansible/roles/routeros/tasks/firewall.yml	IPv4 masquerade narrowed to pppoe-gpon; BGP input rules for vlan6
ansible/roles/openwrt/tasks/network.yml	wwan interface (no auto bringup); uplink with no static gateway
ansible/roles/openwrt/tasks/firewall.yml	wwan zone with masq '1' / masq6 '1'; uplink → wwan forwarding
ansible/roles/openwrt/tasks/bird.yml	BIRD2 install + config
ansible/roles/openwrt/tasks/wwan.yml	qmi.sh patches, BM806C profiles, wwan-bringup init script
ansible/roles/openwrt/defaults/main.yml	bird2 in openwrt_packages