# LTE Failover Design

Reference documentation of the as-built LTE failover design. For day-to-day
network overview see [network.md](./network.md); for BM806C modem firmware
workarounds see [wwan-bm806c-qmi-workaround.md](./wwan-bm806c-qmi-workaround.md).

## Summary

| Property | Value |
|---|---|
| Failover signalling | Symmetric iBGP between D-Link (BIRD2) and CRS (RouterOS) |
| BGP AS | 65000 (iBGP; CRS acts as route reflector for D-Link) |
| LTE transit path | D-Link wwan ← VLAN 6 (192.168.6.0/24) ← CRS |
| D-Link default route source | Learned from CRS via BGP (no static default gateway) |
| CRS LTE route source | Learned from D-Link via BGP at distance 200 |
| Announcement trigger | wwan interface up/down tracked by BIRD2 device protocol |
| Scope | All internet-capable VLANs (vlan2, vlan4, vlan5, vlan6) |
| IPv4 NAT | CRS masquerades on `pppoe-gpon` only; D-Link masquerades on `wwan` |
| IPv6 NAT | D-Link masquerades IPv6 on `wwan` (no inbound on LTE; outbound only) |
| wwan bringup | Triggered by `/etc/init.d/wwan-bringup` after USB re-auth (BM806C wedge fix) |

## Route exchange

### CRS announces to D-Link

| Prefix | Source | Withdrawn when |
|---|---|---|
| `0.0.0.0/0` | `output.default-originate: if-installed` (active default in main table) | GPON drops or `pppoe-gpon` route inactive |
| `2000::/3` | `output.redistribute: static` (HE tunnel default) | `sit1` interface down / HE route inactive |
| VLAN subnets (`192.168.0.0/24`, `192.168.1.0/24`, etc.) | `output.redistribute: connected` | never (CRS always reachable on vlan6) |
| `100.64.0.0/10` (Tailscale) | `output.redistribute: static` | never |
| `172.17.0.0/16` (dockers bridge) | `output.redistribute: connected` | never |
| `10.42.0.0/16`, `10.43.0.0/16`, `10.44.0.0/16` (k8s) | reflected via iBGP RR | when k8s BGP session drops |
| pod/service/LB IPv6 ranges | reflected via iBGP RR | when k8s BGP session drops |

Internal prefixes are announced regardless of GPON state. They remain
reachable via `192.168.6.1` (directly connected on vlan6) even when GPON
fails, so D-Link-originated traffic to internal subnets always routes to
CRS rather than incorrectly exiting via wwan.

The CRS route reflector role (`local.role: ibgp-rr` on the `dlink-lte`
connection) allows it to reflect routes learned from the k8s peer (`bgp1`)
to D-Link without violating iBGP split-horizon. RFC 4456 `ORIGINATOR_ID`
loop prevention is handled automatically by RouterOS — no output filter
needed.

`nexthop-choice: force-self` ensures CRS advertises `192.168.6.1` as the
next-hop for all prefixes, rather than the original route's next-hop
(which may be unreachable from D-Link, e.g. k8s peer `2001:470:61a3:100::3`).

### D-Link announces to CRS

| Prefix | Source | Withdrawn when |
|---|---|---|
| `0.0.0.0/0` | BIRD2 static `lte_default` via `wwan0` | wwan0 down (device protocol detects) |
| `2000::/3` | BIRD2 static `lte_default6` via `wwan0` | wwan0 down |

BIRD2's `protocol device` tracks wwan0 via netlink in real time; when the
interface goes down the static routes become unreachable and BGP withdraws
the announcements immediately.

The BIRD2 static routes use `preference 50` (below the BGP default of 100)
so the BGP-learned routes from CRS are preferred for kernel installation
on D-Link itself — D-Link's own outbound traffic uses the CRS path when
GPON is up. The static routes only exist as triggers for BGP export.

### D-Link kernel routing table

| Destination | Source | Kernel metric | Active when |
|---|---|---|---|
| Internal prefixes (VLANs, k8s, Tailscale) | BGP from CRS, via `192.168.6.1` | 10 (IPv4) / 32 (IPv6) | always (CRS reachable) |
| `0.0.0.0/0` | BGP from CRS | 10 | GPON up |
| `0.0.0.0/0` | wwan QMI-assigned (qmi.sh) | 100 | wwan up |
| `default via wwan IPv6 GW` (non-source-specific) | wwan-bringup script | 1024 | wwan up |
| `default from <wwan prefix>/64 via wwan IPv6 GW` (source-specific) | qmi.sh | 100 | wwan up |

D-Link's own outbound traffic prefers the BGP route (metric 10) over wwan
(metric 100). The non-source-specific IPv6 default at metric 1024 exists
because qmi.sh only installs a source-specific IPv6 default (constrained
to the wwan-assigned `/64` prefix); forwarded traffic from internal
subnets would fail routing lookup with "net unreachable" without it.

### CRS routing table

| Destination | Source | Distance | Active when |
|---|---|---|---|
| `0.0.0.0/0` | static via `pppoe-gpon` | 1 | GPON up |
| `0.0.0.0/0` | BGP from D-Link via `192.168.6.2` | 200 | wwan up on D-Link |
| `2000::/3` | static via `sit1` (HE tunnel) | 1 | sit1 active (HE tunnel works) |
| `2000::/3` | BGP from D-Link via `2001:470:61a3:600::2` | 200 | wwan up on D-Link |

RouterOS distance comparison is straightforward: distance 1 always wins
over distance 200. BGP-learned routes activate automatically when the
static route becomes inactive (e.g. GPON down → `pppoe-gpon` route
inactive → BGP route at distance 200 becomes active).

## Traffic paths

### Normal (GPON up)

```
LAN/SRV/IoT  →  CRS  →  pppoe-gpon  →  ISP
D-Link own   →  uplink  →  CRS  →  pppoe-gpon  →  ISP
                (via BGP-learned default at kernel metric 10)
```

wwan is connected and D-Link announces the LTE default to CRS, but CRS
ignores it (distance 200 loses to distance 1). D-Link uses the
CRS-announced default (metric 10) for its own traffic, not wwan
(metric 100).

### Failover (GPON down)

```
LAN/SRV/IoT  →  CRS  →  vlan6 (→192.168.6.2)  →  D-Link  →  wwan  →  Orange LTE
D-Link own   →  wwan  →  Orange LTE
```

CRS distance-1 routes go inactive → distance-200 BGP routes from D-Link
activate. D-Link receives forwarded traffic on uplink, routes it via the
non-source-specific wwan default (metric 1024), fw4 masquerades the
source, packet exits via wwan. Return traffic reverses through masquerade
state and forwards back to CRS via the established connection-tracking
entry.

When CRS withdraws its BGP-announced default to D-Link (because GPON is
down and CRS has no default to announce), D-Link's kernel default at
metric 10 is removed, leaving the wwan default at metric 100 as the
preferred route for D-Link's own traffic.

### Failure detection

- **D-Link crashes / power loss** → BGP session drops after `hold-time: 30s`
  → CRS withdraws all D-Link-learned routes → internet unavailable if
  GPON also down (acceptable single-point-of-failure for home network)
- **wwan modem goes down** → BIRD2 device protocol detects wwan0 down →
  static `lte_default` / `lte_default6` become unreachable → BGP withdraws
  announcements → CRS removes BGP-learned default
- **GPON drops** → `pppoe-gpon` interface down → CRS distance-1 default
  route inactive → distance-200 BGP route activates → CRS withdraws its
  default-originate announcement to D-Link (since no default is installed
  any more) → D-Link's kernel default-via-CRS is removed → D-Link uses
  wwan kernel default → traffic flows from CRS via vlan6 → D-Link → wwan

All transitions are automatic and driven by interface state. No active
probing (Netwatch / mwan3), no scripts toggling routes.

## NAT rules

NAT rules are always active, matched by output interface. No
failover-triggered toggling needed.

### CRS (RouterOS)

- IPv4 `masquerade` on `srcnat` chain with `out-interface: pppoe-gpon`.
  Only the GPON public interface gets masqueraded — `vlan6` is internal
  and never natted, `sit1` (IPv6) has its own dedicated src-nat for the
  Tailscale prefix.
- IPv6 `src-nat tailnet to internet` on `srcnat` chain for Tailscale
  prefix (`fd7a:115c:a1e0::/48`) to `2001:470:61a3:600::/64`, applied
  on `out-interface-list: wan`. Fires regardless of whether the
  egress is `sit1` or `vlan6`.

### D-Link (OpenWrt fw4)

- `wwan` zone has `option masq '1'` and `option masq6 '1'`. All traffic
  exiting via wwan (own outbound + forwarded from `uplink`) is
  source-NAT'd, IPv4 to the wwan-assigned CG-NAT IP, IPv6 to the
  wwan-assigned `/128` from the Orange-assigned `/64` prefix.
- Forwarding rule `uplink → wwan` allows MikroTik-routed traffic to
  egress via wwan during failover. Default forward policy on the wwan
  zone stays REJECT.

## BGP / route reflection details

### CRS connection config

```
/routing/bgp/connection set dlink-lte \
  remote.address=192.168.6.2/32 \
  local.role=ibgp-rr \
  nexthop-choice=force-self \
  output.redistribute=connected,static \
  output.default-originate=if-installed \
  hold-time=30s keepalive-time=10s
```

`output.default-originate=if-installed` is required for the `0.0.0.0/0`
advertisement because RouterOS does not advertise interface-gateway
static routes (gateway=`pppoe-gpon`) via plain `output.redistribute=static`.
`default-originate` advertises a synthetic default whenever any active
default exists in the routing table, regardless of how it was installed.

### IPv6 Extended Next Hop workaround

RouterOS uses BGP Extended Next Hop Encoding (RFC 5549 / RFC 8950) for
IPv6 routes on this iBGP session, advertising them with an IPv4-mapped
next-hop (`::ffff:192.168.6.1`). The Linux kernel does not support
installing IPv6 routes with IPv4 next-hops, so BIRD2 cannot push them
directly to the kernel.

There is no way to disable ENHE on RouterOS — `local.address`,
`nexthop-choice: force-self`, and output `set gw` filters all fail to
override it. The workaround is on the BIRD2 side: an import filter on
the BGP IPv6 channel rewrites `gw` to CRS's native IPv6 address
(`2001:470:61a3:600::1`) before the route is exported to the kernel.

```
ipv6 {
  extended next hop yes;
  import filter {
    gw = 2001:470:61a3:600::1;
    accept;
  };
  ...
};
```

The reverse direction (D-Link → CRS) was solved cleanly via BIRD2 export
filter setting `bgp_next_hop = 2001:470:61a3:600::2`, since BGP-level
attribute manipulation isn't constrained by kernel limitations.

### Direct protocol on D-Link

BIRD2 needs to know about the directly connected `192.168.6.0/24` and
`2001:470:61a3:600::/64` subnets on `eth0.6` to resolve BGP next-hops.
The `protocol direct { interface "eth0.6"; }` declaration provides this;
without it BIRD2 marks all CRS-learned routes as unreachable.

## BM806C modem cold-boot wedge

The BM806C firmware enters a permanently broken state on cold boot:
`/dev/cdc-wdm0` exists, kernel driver attaches, but uqmi commands return
`"Failed to connect to service"` indefinitely. UIM (SIM) QMI service
specifically never comes up.

Recovery requires a USB device re-enumeration. The `/etc/init.d/wwan-bringup`
service writes `0` then `1` to `/sys/bus/usb/devices/1-1/authorized` on
boot, then triggers `ifup wwan`. After re-auth the modem completes its
QMI initialization within ~1 second.

Full investigation: see [wwan-bm806c-qmi-workaround.md](./wwan-bm806c-qmi-workaround.md).

## Implementation files

| File | Role |
|---|---|
| `ansible/roles/routeros/tasks/base.yml` | `vlan6` in `wan` interface list |
| `ansible/roles/routeros/tasks/routing.yml` | BGP instance, template, `dlink-lte` connection |
| `ansible/roles/routeros/tasks/firewall.yml` | IPv4 masquerade narrowed to `pppoe-gpon`; BGP input rules for `vlan6` |
| `ansible/roles/openwrt/tasks/network.yml` | `wwan` interface (no auto bringup); `uplink` with no static gateway |
| `ansible/roles/openwrt/tasks/firewall.yml` | `wwan` zone with `masq '1'` / `masq6 '1'`; `uplink → wwan` forwarding |
| `ansible/roles/openwrt/tasks/bird.yml` | BIRD2 install + config |
| `ansible/roles/openwrt/tasks/wwan.yml` | qmi.sh patches, BM806C profiles, wwan-bringup init script |
| `ansible/roles/openwrt/defaults/main.yml` | `bird2` in `openwrt_packages` |