Lumpiasty/klaster
0
0
Fork 0
Code Issues Pull Requests 2 Packages Projects Releases Wiki Activity

lte failover
ci/woodpecker/push/flux-reconcile-source Pipeline was successful
Details
ci/woodpecker/cron/renovate Pipeline was successful
Details

Browse Source
This commit is contained in:
Lumpiasty
2026-05-27 23:40:33 +02:00
parent 754c8952bc
commit 5b026593ce
15 changed files with 761 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/network.md
+7 -2
View File
@@ -115,8 +115,13 @@ One of quirks of the ISP is that it doesn't allow incoming port 53/DNS connectio
The ISP does not provide any IPv6 connectivity at all. For that purpose I'm using [tunnel broker from Hurricane Electric](https://tunnelbroker.net/), which gives /48 routed prefix that I divided to /64 networks.
There used to be backup internet link using USB LTE modem connected to CRS, which was exposing NDIS interface, but when installing D-Link I decided to remove the modem and move SIM card to it to reduce clutter in rack and have direct access to fully fledged modem, not just web interface management. Configuration of lte1 modem is yet to be removed from the CRS configuration. Modem in D-Link requires workaround to work due to firmware bug, described in detail in [LTE failover (BroadMobi BM806C / D-Link DWR-921 C1) — QMI data-plane workaround](./wwan-bm806c-qmi-workaround.md). It is currently partially configured, with internet working on OpenWRT router when enabled, but failover functionality of internet gateway on CRS is yet to be designed and implemented.
SIM card allows for IPv4 and IPv6 connectivity via separate APNs. Network hands out globally routable IPv6 prefix, but there are no incoming IPv6 connections, which is most likely network carrier enforced firewall. Network works when using two different APNs at once, but when using the card in Android phone, there's no need to configure two separate APNs, IPv6 alone is sufficient. Whether the network announces NAT64 and Android phone is doing CLAT or how is that working exactly and if we can utilize it in our network to simplify connection is yet to be figured out.
The backup internet link is an LTE connection via the embedded BroadMobi BM806C modem in the D-Link router (Orange Poland, dual-stack). The SIM was previously in a USB modem attached directly to the CRS; it was moved to the D-Link to reduce rack clutter and gain access to a proper modem interface. The modem requires firmware-level workarounds — QMI data-plane bugs, a cold-boot UIM wedge that needs USB re-enumeration — documented in [LTE failover (BroadMobi BM806C / D-Link DWR-921 C1) — QMI data-plane workaround](./wwan-bm806c-qmi-workaround.md).
Failover is implemented using iBGP between the D-Link (BIRD2, AS 65000) and the CRS (`local.role: ibgp-rr` so CRS acts as route reflector for D-Link). The D-Link announces `0.0.0.0/0` and `2000::/3` to the CRS whenever its `wwan` interface is up. The CRS installs these at BGP distance 200 — below the GPON static default at distance 1 — so they only become active when GPON fails. The CRS in turn announces all its connected and static routes (VLAN subnets, Tailscale, k8s pod/service/LB prefixes via RR reflection) to the D-Link so it always has explicit routes to internal subnets regardless of WAN state. The D-Link's own default route also comes from this BGP session (no static gateway on the uplink interface); when the CRS withdraws the default on GPON failure, the D-Link falls back to its wwan kernel route at metric 100.
For full design rationale, route exchange tables, and implementation notes including the BGP Extended Next Hop workarounds, see [LTE failover design](./lte-failover-design.md).
During LTE failover, all VLANs route through `vlan6` to the D-Link, which forwards traffic out `wwan` and masquerades it (IPv4 and IPv6 via fw4 `masq`/`masq6`). IPv6 is outbound-only — the carrier enforces an inbound firewall, and there is no routed prefix large enough to cover all internal subnets without NAT.
## Configuration management