256 lines
11 KiB
Markdown
256 lines
11 KiB
Markdown
# LTE Failover Design
|
|
|
|
Reference documentation of the as-built LTE failover design. For day-to-day
|
|
network overview see [network.md](./network.md); for BM806C modem firmware
|
|
workarounds see [wwan-bm806c-qmi-workaround.md](./wwan-bm806c-qmi-workaround.md).
|
|
|
|
## Summary
|
|
|
|
| Property | Value |
|
|
|---|---|
|
|
| Failover signalling | Symmetric iBGP between D-Link (BIRD2) and CRS (RouterOS) |
|
|
| BGP AS | 65000 (iBGP; CRS acts as route reflector for D-Link) |
|
|
| LTE transit path | D-Link wwan ← VLAN 6 (192.168.6.0/24) ← CRS |
|
|
| D-Link default route source | Learned from CRS via BGP (no static default gateway) |
|
|
| CRS LTE route source | Learned from D-Link via BGP at distance 200 |
|
|
| Announcement trigger | wwan interface up/down tracked by BIRD2 device protocol |
|
|
| Scope | All internet-capable VLANs (vlan2, vlan4, vlan5, vlan6) |
|
|
| IPv4 NAT | CRS masquerades on `pppoe-gpon` only; D-Link masquerades on `wwan` |
|
|
| IPv6 NAT | D-Link masquerades IPv6 on `wwan` (no inbound on LTE; outbound only) |
|
|
| wwan bringup | Triggered by `/etc/init.d/wwan-bringup` after USB re-auth (BM806C wedge fix) |
|
|
|
|
## Route exchange
|
|
|
|
### CRS announces to D-Link
|
|
|
|
| Prefix | Source | Withdrawn when |
|
|
|---|---|---|
|
|
| `0.0.0.0/0` | `output.default-originate: if-installed` (active default in main table) | GPON drops or `pppoe-gpon` route inactive |
|
|
| `2000::/3` | `output.redistribute: static` (HE tunnel default) | `sit1` interface down / HE route inactive |
|
|
| VLAN subnets (`192.168.0.0/24`, `192.168.1.0/24`, etc.) | `output.redistribute: connected` | never (CRS always reachable on vlan6) |
|
|
| `100.64.0.0/10` (Tailscale) | `output.redistribute: static` | never |
|
|
| `172.17.0.0/16` (dockers bridge) | `output.redistribute: connected` | never |
|
|
| `10.42.0.0/16`, `10.43.0.0/16`, `10.44.0.0/16` (k8s) | reflected via iBGP RR | when k8s BGP session drops |
|
|
| pod/service/LB IPv6 ranges | reflected via iBGP RR | when k8s BGP session drops |
|
|
|
|
Internal prefixes are announced regardless of GPON state. They remain
|
|
reachable via `192.168.6.1` (directly connected on vlan6) even when GPON
|
|
fails, so D-Link-originated traffic to internal subnets always routes to
|
|
CRS rather than incorrectly exiting via wwan.
|
|
|
|
The CRS route reflector role (`local.role: ibgp-rr` on the `dlink-lte`
|
|
connection) allows it to reflect routes learned from the k8s peer (`bgp1`)
|
|
to D-Link without violating iBGP split-horizon. RFC 4456 `ORIGINATOR_ID`
|
|
loop prevention is handled automatically by RouterOS — no output filter
|
|
needed.
|
|
|
|
`nexthop-choice: force-self` ensures CRS advertises `192.168.6.1` as the
|
|
next-hop for all prefixes, rather than the original route's next-hop
|
|
(which may be unreachable from D-Link, e.g. k8s peer `2001:470:61a3:100::3`).
|
|
|
|
### D-Link announces to CRS
|
|
|
|
| Prefix | Source | Withdrawn when |
|
|
|---|---|---|
|
|
| `0.0.0.0/0` | BIRD2 static `lte_default` via `wwan0` | wwan0 down (device protocol detects) |
|
|
| `2000::/3` | BIRD2 static `lte_default6` via `wwan0` | wwan0 down |
|
|
|
|
BIRD2's `protocol device` tracks wwan0 via netlink in real time; when the
|
|
interface goes down the static routes become unreachable and BGP withdraws
|
|
the announcements immediately.
|
|
|
|
The BIRD2 static routes use `preference 50` (below the BGP default of 100)
|
|
so the BGP-learned routes from CRS are preferred for kernel installation
|
|
on D-Link itself — D-Link's own outbound traffic uses the CRS path when
|
|
GPON is up. The static routes only exist as triggers for BGP export.
|
|
|
|
### D-Link kernel routing table
|
|
|
|
| Destination | Source | Kernel metric | Active when |
|
|
|---|---|---|---|
|
|
| Internal prefixes (VLANs, k8s, Tailscale) | BGP from CRS, via `192.168.6.1` | 10 (IPv4) / 32 (IPv6) | always (CRS reachable) |
|
|
| `0.0.0.0/0` | BGP from CRS | 10 | GPON up |
|
|
| `0.0.0.0/0` | wwan QMI-assigned (qmi.sh) | 100 | wwan up |
|
|
| `default via wwan IPv6 GW` (non-source-specific) | wwan-bringup script | 1024 | wwan up |
|
|
| `default from <wwan prefix>/64 via wwan IPv6 GW` (source-specific) | qmi.sh | 100 | wwan up |
|
|
|
|
D-Link's own outbound traffic prefers the BGP route (metric 10) over wwan
|
|
(metric 100). The non-source-specific IPv6 default at metric 1024 exists
|
|
because qmi.sh only installs a source-specific IPv6 default (constrained
|
|
to the wwan-assigned `/64` prefix); forwarded traffic from internal
|
|
subnets would fail routing lookup with "net unreachable" without it.
|
|
|
|
### CRS routing table
|
|
|
|
| Destination | Source | Distance | Active when |
|
|
|---|---|---|---|
|
|
| `0.0.0.0/0` | static via `pppoe-gpon` | 1 | GPON up |
|
|
| `0.0.0.0/0` | BGP from D-Link via `192.168.6.2` | 200 | wwan up on D-Link |
|
|
| `2000::/3` | static via `sit1` (HE tunnel) | 1 | sit1 active (HE tunnel works) |
|
|
| `2000::/3` | BGP from D-Link via `2001:470:61a3:600::2` | 200 | wwan up on D-Link |
|
|
|
|
RouterOS distance comparison is straightforward: distance 1 always wins
|
|
over distance 200. BGP-learned routes activate automatically when the
|
|
static route becomes inactive (e.g. GPON down → `pppoe-gpon` route
|
|
inactive → BGP route at distance 200 becomes active).
|
|
|
|
## Traffic paths
|
|
|
|
### Normal (GPON up)
|
|
|
|
```
|
|
LAN/SRV/IoT → CRS → pppoe-gpon → ISP
|
|
D-Link own → uplink → CRS → pppoe-gpon → ISP
|
|
(via BGP-learned default at kernel metric 10)
|
|
```
|
|
|
|
wwan is connected and D-Link announces the LTE default to CRS, but CRS
|
|
ignores it (distance 200 loses to distance 1). D-Link uses the
|
|
CRS-announced default (metric 10) for its own traffic, not wwan
|
|
(metric 100).
|
|
|
|
### Failover (GPON down)
|
|
|
|
```
|
|
LAN/SRV/IoT → CRS → vlan6 (→192.168.6.2) → D-Link → wwan → Orange LTE
|
|
D-Link own → wwan → Orange LTE
|
|
```
|
|
|
|
CRS distance-1 routes go inactive → distance-200 BGP routes from D-Link
|
|
activate. D-Link receives forwarded traffic on uplink, routes it via the
|
|
non-source-specific wwan default (metric 1024), fw4 masquerades the
|
|
source, packet exits via wwan. Return traffic reverses through masquerade
|
|
state and forwards back to CRS via the established connection-tracking
|
|
entry.
|
|
|
|
When CRS withdraws its BGP-announced default to D-Link (because GPON is
|
|
down and CRS has no default to announce), D-Link's kernel default at
|
|
metric 10 is removed, leaving the wwan default at metric 100 as the
|
|
preferred route for D-Link's own traffic.
|
|
|
|
### Failure detection
|
|
|
|
- **D-Link crashes / power loss** → BGP session drops after `hold-time: 30s`
|
|
→ CRS withdraws all D-Link-learned routes → internet unavailable if
|
|
GPON also down (acceptable single-point-of-failure for home network)
|
|
- **wwan modem goes down** → BIRD2 device protocol detects wwan0 down →
|
|
static `lte_default` / `lte_default6` become unreachable → BGP withdraws
|
|
announcements → CRS removes BGP-learned default
|
|
- **GPON drops** → `pppoe-gpon` interface down → CRS distance-1 default
|
|
route inactive → distance-200 BGP route activates → CRS withdraws its
|
|
default-originate announcement to D-Link (since no default is installed
|
|
any more) → D-Link's kernel default-via-CRS is removed → D-Link uses
|
|
wwan kernel default → traffic flows from CRS via vlan6 → D-Link → wwan
|
|
|
|
All transitions are automatic and driven by interface state. No active
|
|
probing (Netwatch / mwan3), no scripts toggling routes.
|
|
|
|
## NAT rules
|
|
|
|
NAT rules are always active, matched by output interface. No
|
|
failover-triggered toggling needed.
|
|
|
|
### CRS (RouterOS)
|
|
|
|
- IPv4 `masquerade` on `srcnat` chain with `out-interface: pppoe-gpon`.
|
|
Only the GPON public interface gets masqueraded — `vlan6` is internal
|
|
and never natted, `sit1` (IPv6) has its own dedicated src-nat for the
|
|
Tailscale prefix.
|
|
- IPv6 `src-nat tailnet to internet` on `srcnat` chain for Tailscale
|
|
prefix (`fd7a:115c:a1e0::/48`) to `2001:470:61a3:600::/64`, applied
|
|
on `out-interface-list: wan`. Fires regardless of whether the
|
|
egress is `sit1` or `vlan6`.
|
|
|
|
### D-Link (OpenWrt fw4)
|
|
|
|
- `wwan` zone has `option masq '1'` and `option masq6 '1'`. All traffic
|
|
exiting via wwan (own outbound + forwarded from `uplink`) is
|
|
source-NAT'd, IPv4 to the wwan-assigned CG-NAT IP, IPv6 to the
|
|
wwan-assigned `/128` from the Orange-assigned `/64` prefix.
|
|
- Forwarding rule `uplink → wwan` allows MikroTik-routed traffic to
|
|
egress via wwan during failover. Default forward policy on the wwan
|
|
zone stays REJECT.
|
|
|
|
## BGP / route reflection details
|
|
|
|
### CRS connection config
|
|
|
|
```
|
|
/routing/bgp/connection set dlink-lte \
|
|
remote.address=192.168.6.2/32 \
|
|
local.role=ibgp-rr \
|
|
nexthop-choice=force-self \
|
|
output.redistribute=connected,static \
|
|
output.default-originate=if-installed \
|
|
hold-time=30s keepalive-time=10s
|
|
```
|
|
|
|
`output.default-originate=if-installed` is required for the `0.0.0.0/0`
|
|
advertisement because RouterOS does not advertise interface-gateway
|
|
static routes (gateway=`pppoe-gpon`) via plain `output.redistribute=static`.
|
|
`default-originate` advertises a synthetic default whenever any active
|
|
default exists in the routing table, regardless of how it was installed.
|
|
|
|
### IPv6 Extended Next Hop workaround
|
|
|
|
RouterOS uses BGP Extended Next Hop Encoding (RFC 5549 / RFC 8950) for
|
|
IPv6 routes on this iBGP session, advertising them with an IPv4-mapped
|
|
next-hop (`::ffff:192.168.6.1`). The Linux kernel does not support
|
|
installing IPv6 routes with IPv4 next-hops, so BIRD2 cannot push them
|
|
directly to the kernel.
|
|
|
|
There is no way to disable ENHE on RouterOS — `local.address`,
|
|
`nexthop-choice: force-self`, and output `set gw` filters all fail to
|
|
override it. The workaround is on the BIRD2 side: an import filter on
|
|
the BGP IPv6 channel rewrites `gw` to CRS's native IPv6 address
|
|
(`2001:470:61a3:600::1`) before the route is exported to the kernel.
|
|
|
|
```
|
|
ipv6 {
|
|
extended next hop yes;
|
|
import filter {
|
|
gw = 2001:470:61a3:600::1;
|
|
accept;
|
|
};
|
|
...
|
|
};
|
|
```
|
|
|
|
The reverse direction (D-Link → CRS) was solved cleanly via BIRD2 export
|
|
filter setting `bgp_next_hop = 2001:470:61a3:600::2`, since BGP-level
|
|
attribute manipulation isn't constrained by kernel limitations.
|
|
|
|
### Direct protocol on D-Link
|
|
|
|
BIRD2 needs to know about the directly connected `192.168.6.0/24` and
|
|
`2001:470:61a3:600::/64` subnets on `eth0.6` to resolve BGP next-hops.
|
|
The `protocol direct { interface "eth0.6"; }` declaration provides this;
|
|
without it BIRD2 marks all CRS-learned routes as unreachable.
|
|
|
|
## BM806C modem cold-boot wedge
|
|
|
|
The BM806C firmware enters a permanently broken state on cold boot:
|
|
`/dev/cdc-wdm0` exists, kernel driver attaches, but uqmi commands return
|
|
`"Failed to connect to service"` indefinitely. UIM (SIM) QMI service
|
|
specifically never comes up.
|
|
|
|
Recovery requires a USB device re-enumeration. The `/etc/init.d/wwan-bringup`
|
|
service writes `0` then `1` to `/sys/bus/usb/devices/1-1/authorized` on
|
|
boot, then triggers `ifup wwan`. After re-auth the modem completes its
|
|
QMI initialization within ~1 second.
|
|
|
|
Full investigation: see [wwan-bm806c-qmi-workaround.md](./wwan-bm806c-qmi-workaround.md).
|
|
|
|
## Implementation files
|
|
|
|
| File | Role |
|
|
|---|---|
|
|
| `ansible/roles/routeros/tasks/base.yml` | `vlan6` in `wan` interface list |
|
|
| `ansible/roles/routeros/tasks/routing.yml` | BGP instance, template, `dlink-lte` connection |
|
|
| `ansible/roles/routeros/tasks/firewall.yml` | IPv4 masquerade narrowed to `pppoe-gpon`; BGP input rules for `vlan6` |
|
|
| `ansible/roles/openwrt/tasks/network.yml` | `wwan` interface (no auto bringup); `uplink` with no static gateway |
|
|
| `ansible/roles/openwrt/tasks/firewall.yml` | `wwan` zone with `masq '1'` / `masq6 '1'`; `uplink → wwan` forwarding |
|
|
| `ansible/roles/openwrt/tasks/bird.yml` | BIRD2 install + config |
|
|
| `ansible/roles/openwrt/tasks/wwan.yml` | qmi.sh patches, BM806C profiles, wwan-bringup init script |
|
|
| `ansible/roles/openwrt/defaults/main.yml` | `bird2` in `openwrt_packages` |
|