Compare commits

...

14 Commits

Author SHA1 Message Date
Lumpiasty 2e2ccb4f3e Merge pull request 'add peer api server to remedy DNS' (#33) from fix/add-peerapiserver into main
ci/woodpecker/push/release-tag Pipeline was successful
ci/woodpecker/push/pr-build Pipeline was successful
ci/woodpecker/tag/release Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
Reviewed-on: #33
2026-06-16 22:34:43 +00:00
Lumpiasty e009040cb4 add peer api server to remedy DNS
ci/woodpecker/pr/pr-build Pipeline was successful
2026-06-17 00:30:46 +02:00
Lumpiasty 492230a746 Merge pull request 'Enable netstack to hopefully fix DNS' (#31) from fix/add-gvisor-netstack into main
ci/woodpecker/push/release-tag Pipeline was successful
ci/woodpecker/push/pr-build Pipeline was successful
ci/woodpecker/tag/release Pipeline was successful
Reviewed-on: #31
2026-06-16 21:41:22 +00:00
Lumpiasty 0a8a40fdb8 add documentation on netstack decision
ci/woodpecker/pr/pr-build Pipeline was successful
2026-06-16 23:37:53 +02:00
Lumpiasty 7482ddb832 Enable netstack to hopefully fix DNS
ci/woodpecker/pr/pr-build Pipeline was successful
2026-06-16 23:32:03 +02:00
Lumpiasty da2b3b5d3a Merge pull request 'Remove docker build cache' (#32) from fix/remove-docker-build-cache into main
ci/woodpecker/push/release-tag Pipeline is pending
ci/woodpecker/push/pr-build Pipeline failed
Reviewed-on: #32
2026-06-16 21:31:26 +00:00
Lumpiasty d03c7d3da7 Remove docker build cache
ci/woodpecker/pr/pr-build Pipeline was successful
2026-06-16 23:18:18 +02:00
Renovate 85f522bce1 Merge pull request 'chore(deps): update golang:1.26.4-alpine docker digest to f1ddd9f' (#30) from renovate/golang-1.26.4-alpine into main
ci/woodpecker/push/pr-build Pipeline was successful
ci/woodpecker/push/release-tag Pipeline was successful
2026-06-16 02:12:34 +00:00
Renovate 509762c1b4 chore(deps): update golang:1.26.4-alpine docker digest to f1ddd9f
ci/woodpecker/pr/pr-build Pipeline was successful
2026-06-16 02:01:02 +00:00
Lumpiasty 06083dcf58 Merge pull request 'Speed up build pipeline' (#29) from feat/busybox-crosscompile into main
ci/woodpecker/push/release-tag Pipeline was successful
ci/woodpecker/push/pr-build Pipeline was successful
ci/woodpecker/tag/release Pipeline was successful
ci/woodpecker/cron/renovate Pipeline was successful
Reviewed-on: #29
2026-06-16 00:12:58 +00:00
Lumpiasty ff60452758 Empty commit to trigger CI
ci/woodpecker/pr/pr-build Pipeline was successful
2026-06-16 01:59:47 +02:00
Lumpiasty 524b83d911 Docker build caching 2026-06-16 01:57:20 +02:00
Lumpiasty 8fee49bf09 cross compile busybox instead of emulation
ci/woodpecker/pr/pr-build Pipeline was successful
2026-06-16 01:50:50 +02:00
Lumpiasty b8dd344a93 Merge pull request 'Add workaround for panic with ts_omit_netstack' (#28) from fix/invertgsochecksum into main
ci/woodpecker/push/release-tag Pipeline was successful
ci/woodpecker/push/pr-build Pipeline was successful
Reviewed-on: #28
2026-06-15 23:24:55 +00:00
4 changed files with 339 additions and 76 deletions
+1 -2
View File
@@ -33,8 +33,7 @@ steps:
image: woodpeckerci/plugin-docker-buildx:6.1.0 image: woodpeckerci/plugin-docker-buildx:6.1.0
privileged: true privileged: true
settings: settings:
repo: mikrotik-tailscale
platforms: linux/amd64,linux/arm64,linux/arm/v7 platforms: linux/amd64,linux/arm64,linux/arm/v7
dry-run: true dry_run: true
build_args: build_args:
- OCI_VERSION=ci-${CI_COMMIT_SHA} - OCI_VERSION=ci-${CI_COMMIT_SHA}
+164 -58
View File
@@ -12,14 +12,27 @@
# it would need a glibc (Debian) base and produces a much larger image. See # it would need a glibc (Debian) base and produces a much larger image. See
# README for details if you need it. # README for details if you need it.
# #
# The Go builder cross-compiles, so it always runs NATIVELY on the build host # Both the Go (Tailscale) stage and the C (busybox) stage cross-compile: they
# ($BUILDPLATFORM) for speed; only the busybox stage and the final image run on # always run NATIVELY on the build host ($BUILDPLATFORM) and produce binaries
# the target platform. # for $TARGETPLATFORM. This eliminates QEMU emulation entirely from the build,
# which is the main source of slowness in multi-arch builds. Only the final
# scratch stage pulls in the target-arch-specific layers (CA certs, busybox
# rootfs) which are just file copies with no emulated execution.
#
# Cross-compilation for C (busybox) is provided by tonistiigi/xx, which
# configures clang+lld as a cross-compiler and installs musl headers for the
# target arch via xx-apk.
# =============================================================================
# xx: Dockerfile cross-compilation helpers (provides xx-clang, xx-apk, etc.)
# =============================================================================
# renovate: datasource=docker depName=tonistiigi/xx versioning=docker
FROM --platform=$BUILDPLATFORM tonistiigi/xx:1.9.0@sha256:c64defb9ed5a91eacb37f96ccc3d4cd72521c4bd18d5442905b95e2226b0e707 AS xx
# ============================================================================= # =============================================================================
# Stage 1: Build Tailscale combined binary (cross-compiled, runs natively) # Stage 1: Build Tailscale combined binary (cross-compiled, runs natively)
# ============================================================================= # =============================================================================
FROM --platform=$BUILDPLATFORM golang:1.26.4-alpine@sha256:7a3e50096189ad57c9f9f865e7e4aa8585ed1585248513dc5cda498e2f41812c AS builder FROM --platform=$BUILDPLATFORM golang:1.26.4-alpine@sha256:f1ddd9fe14fffc091dd98cb4bfa999f32c5fc77d2f2305ea9f0e2595c5437c14 AS builder
# renovate: datasource=github-releases depName=tailscale packageName=tailscale/tailscale versioning=semver # renovate: datasource=github-releases depName=tailscale packageName=tailscale/tailscale versioning=semver
ARG TAILSCALE_VERSION=v1.98.5 ARG TAILSCALE_VERSION=v1.98.5
@@ -57,49 +70,6 @@ WORKDIR /src/tailscale
# disables the filter at runtime for debugging — no rebuild needed. # disables the filter at runtime for debugging — no rebuild needed.
COPY patches/stderr_verbosity_filter.go cmd/tailscaled/ COPY patches/stderr_verbosity_filter.go cmd/tailscaled/
# Patch net/tstun/wrap.go: fix panic("unreachable") in invertGSOChecksum for
# ts_omit_netstack builds.
#
# invertGSOChecksum is a gVisor/GSO helper that inverts a transport-layer
# checksum before/after SNAT when gVisor hands us a segment with a partial
# checksum (NeedsCsum=true). It is only meaningful when netstack (gVisor) is
# compiled in (HasNetstack=true).
#
# The function correctly guards its body with:
# if !buildfeatures.HasNetstack { panic("unreachable") }
#
# When built with ts_omit_netstack, HasNetstack is a const false, so that guard
# evaluates to `if true { panic(...) }` — the function always panics.
#
# The problem: invertGSOChecksum is called unconditionally from injectedRead()
# (twice, around pc.snat()), even for the res.data path where res.packet==nil
# and gso is a zero-value netstack_GSO (NeedsCsum=false). The HasNetstack
# guard in the res.packet branch does NOT protect these calls.
#
# As a result, any code path that injects an outbound packet via InjectOutbound()
# — which happens when enabling exit-node use (Tailscale sends TSMP messages
# and synthesizes packets through the TUN injection path) — hits injectedRead
# with res.data!=nil, calls invertGSOChecksum, and crashes with:
# panic: unreachable
# tailscale.com/net/tstun.invertGSOChecksum(...)
# tailscale.com/net/tstun.(*Wrapper).injectedRead(...) wrap.go:1077
#
# Fix: replace the `panic("unreachable")` with a `return` in invertGSOChecksum.
# When HasNetstack=false (ts_omit_netstack), a zero-value netstack_GSO always
# has NeedsCsum=false, so the function is correctly a no-op anyway. This matches
# what the function would do if the rest of its body ran: NeedsCsum=false → return.
#
# The sed expression targets the function precisely: it matches the three-line
# sequence that opens invertGSOChecksum's HasNetstack guard, and replaces only
# the panic line with return. The pattern is stable across minor reformats
# because it anchors on the literal function comment and the specific panic string.
#
# See tailscale/tailscale issue for context (no upstream fix as of v1.98.5):
# panic happens when using exit-node via a ts_omit_netstack build.
RUN sed -i \
-e '/func invertGSOChecksum/,/^}/ s/\t\tpanic("unreachable")/\t\treturn/' \
net/tstun/wrap.go
# Build a minimal combined binary (tailscale CLI + tailscaled daemon in one file). # Build a minimal combined binary (tailscale CLI + tailscaled daemon in one file).
# #
# Tag strategy — ALLOWLIST, not blocklist: # Tag strategy — ALLOWLIST, not blocklist:
@@ -135,6 +105,59 @@ RUN sed -i \
# waiting for completion") WITHOUT printing the auth URL # waiting for completion") WITHOUT printing the auth URL
# or confirming success. Including it makes interactive # or confirming success. Including it makes interactive
# 'up' behave normally (blocks, prints login URL). # 'up' behave normally (blocks, prints login URL).
# netstack — gVisor userspace network stack. Counter-intuitively
# REQUIRED even though the router uses a real kernel TUN
# (NOT --tun=userspace-networking). In v1.98.5 the
# 100.100.100.100:53 MagicDNS listener is served ONLY by
# netstack's handleLocalPackets, installed via
# PreFilterPacketOutboundToWireGuardNetstackIntercept.
# The non-netstack "engine" interceptor that the wrap.go
# comments claim handles quad-100 "if netstack is not
# installed" does NOT actually do so on Linux (its body
# only reflects loopback on darwin/ios/plan9, else
# Accept). So with ts_omit_netstack, NOTHING absorbs
# packets to 100.100.100.100: queries fall through to
# WireGuard, no peer owns that IP, and even tailnet-name
# resolution (and 'ping host.tailnet.ts.net') times out.
# The 'dns' tag links the resolver but nothing routes
# packets to it without netstack — the two tags are
# independent (dns has no Dep on netstack). Omitting
# netstack ALSO triggered a panic("unreachable") in
# net/tstun.invertGSOChecksum on the exit-node inject
# path (HasNetstack=const false made the guard always
# panic); enabling netstack makes that guard dead code,
# fixing the crash as a side effect. Cost (arm64, vs a
# netstack-omitted build): ~+0.5 MB extracted on flash
# and ~+2.3 MB resident RAM after UPX decompression —
# measured, acceptable for a 16 MB-flash router.
# gro — Generic Receive Offload (perf). Depends on netstack;
# pulled in with it. Small, and improves throughput on
# the netstack DNS/inject path.
# peerapiserver — REQUIRED to be a functional exit node. In v1.98.5
# 'advertiseexitnode' DECLARES a dependency on
# peerapiserver (featuretags.go Deps, "to run the ExitDNS
# server"), but this build's allowlist works by stripping
# individual ts_omit_ tags and does NOT re-resolve Deps —
# so featuretags --min still emitted ts_omit_peerapiserver
# and our advertiseexitnode opt-in alone left it omitted.
# peerapiserver gates the entire PeerAPI HTTP server,
# including the /dns-query DoH endpoint (peerapi.go,
# guarded by buildfeatures.HasPeerAPIServer). Without it
# initPeerAPIListenerLocked() returns early: the node
# never advertises the PeerAPIDNS service, so exit-node
# CLIENTS' exitNodeCanProxyDNS(thisNode) returns false.
# With no tailnet global nameserver configured, the
# client's resolver then has an empty Routes["."] and
# returns an INSTANT authoritative SERVFAIL locally
# (forwarder.go servfailResponse, aa=1, 0 ms, no I/O) —
# i.e. devices using this router as their exit node could
# not resolve PUBLIC names. Including peerapiserver makes
# the node serve the exit-node DoH DNS proxy, so clients
# get public DNS automatically (the normal exit-node
# behavior) with no tailnet DNS config required.
# peerapiserver has NO Deps and pulls in no large
# subsystems — a small addition. (outboundproxy is NOT
# needed for this and stays omitted.)
# #
# Everything else remains omitted, including (rationale): # Everything else remains omitted, including (rationale):
# clientupdate — DELIBERATELY removed. The built-in updater would download # clientupdate — DELIBERATELY removed. The built-in updater would download
@@ -159,9 +182,11 @@ RUN sed -i \
# which is exactly the flash wear we want to avoid. # which is exactly the flash wear we want to avoid.
# logtail — no persistent log writes to flash; also pass # logtail — no persistent log writes to flash; also pass
# --no-logs-no-support at runtime # --no-logs-no-support at runtime
# netstack+gro — userspace networking; router uses kernel TUN
# ssh — not needed; access via MikroTik SSH + tailscale CLI # ssh — not needed; access via MikroTik SSH + tailscale CLI
# all GUI/desktop/cloud/k8s features — irrelevant for a headless router # all GUI/desktop/cloud/k8s features — irrelevant for a headless router
#
# NOTE: netstack/gro are NOT in this omit list — see the opted-in section above
# for why MagicDNS quad-100 serving structurally requires them in v1.98.5.
RUN mkdir -p /out && \ RUN mkdir -p /out && \
ALL_OMIT=$(GOOS= GOARCH= go run ./cmd/featuretags --min --add=osrouter) && \ ALL_OMIT=$(GOOS= GOARCH= go run ./cmd/featuretags --min --add=osrouter) && \
@@ -178,6 +203,9 @@ RUN mkdir -p /out && \
-e 's/ts_omit_iptables,\{0,1\}//g' \ -e 's/ts_omit_iptables,\{0,1\}//g' \
-e 's/ts_omit_unixsocketidentity,\{0,1\}//g' \ -e 's/ts_omit_unixsocketidentity,\{0,1\}//g' \
-e 's/ts_omit_ipnbus,\{0,1\}//g' \ -e 's/ts_omit_ipnbus,\{0,1\}//g' \
-e 's/ts_omit_netstack,\{0,1\}//g' \
-e 's/ts_omit_gro,\{0,1\}//g' \
-e 's/ts_omit_peerapiserver,\{0,1\}//g' \
-e 's/,$//' \ -e 's/,$//' \
) && \ ) && \
echo "Build tags: ${TAGS}" && \ echo "Build tags: ${TAGS}" && \
@@ -236,7 +264,7 @@ RUN printf '%s\n' \
chmod +x /out/usrlocalbin/entrypoint.sh chmod +x /out/usrlocalbin/entrypoint.sh
# ============================================================================= # =============================================================================
# Stage 2: Custom minimal busybox # Stage 2: Custom minimal busybox (cross-compiled, runs natively on build host)
# ============================================================================= # =============================================================================
# The official busybox:musl image ships all ~404 applets at ~1.24 MB. For a # The official busybox:musl image ships all ~404 applets at ~1.24 MB. For a
# debug shell on a flash-constrained router we only need ~100 applets, so we # debug shell on a flash-constrained router we only need ~100 applets, so we
@@ -253,15 +281,56 @@ RUN printf '%s\n' \
# acceptable for an occasional debug shell. RouterOS stores the EXTRACTED # acceptable for an occasional debug shell. RouterOS stores the EXTRACTED
# rootfs on disk (overlayfs), so the ~190 kB UPX saving is real on-disk space. # rootfs on disk (overlayfs), so the ~190 kB UPX saving is real on-disk space.
# #
# This stage runs on the TARGET platform (no --platform override): gcc then # This stage runs NATIVELY on the build host (--platform=$BUILDPLATFORM) and
# produces native target-arch binaries directly. Under buildx this is # cross-compiles busybox for the target architecture using clang+lld via the
# transparently emulated via binfmt/QEMU for non-native targets. # tonistiigi/xx helpers. This eliminates QEMU emulation from the busybox build,
FROM alpine:3.24.0@sha256:a2d49ea686c2adfe3c992e47dc3b5e7fa6e6b5055609400dc2acaeb241c829f4 AS busybox # which was the main source of slowness for arm64/arm/v7 targets.
#
# Cross-compilation setup:
# - xx-apk installs musl-dev and linux-headers for the TARGET arch under
# /<triple> (a secondary sysroot), while clang/lld/upx/make stay native.
# - xx-clang --setup-target-triple creates <triple>-clang / <triple>-cc
# aliases in PATH that busybox's Makefile picks up via CROSS_COMPILE.
# - Busybox make receives:
# CROSS_COMPILE=<triple>- → picks up <triple>-clang (from xx aliases)
# CC=clang → use clang (aliased target via CROSS_COMPILE)
# HOSTCC=gcc → compile host helper tools with native gcc
# - upx (native x86_64 binary) can compress target-arch binaries since UPX
# operates on the ELF file format regardless of the target ISA.
#
# Applet symlink probing: for native-arch builds the probe runs directly;
# for cross-compiled binaries we use QEMU user-mode emulation (from binfmt)
# only for this one lightweight probe step (busybox --help per applet), not
# for the compile itself. The probe can alternatively be skipped by using
# a pre-enumerated applet list, but the current approach is simpler.
FROM --platform=$BUILDPLATFORM alpine:3.24.0@sha256:a2d49ea686c2adfe3c992e47dc3b5e7fa6e6b5055609400dc2acaeb241c829f4 AS busybox
# Copy xx cross-compilation helpers (xx-clang, xx-apk, xx-info, etc.)
COPY --from=xx / /
# renovate: datasource=docker depName=busybox versioning=docker # renovate: datasource=docker depName=busybox versioning=docker
ARG BUSYBOX_VERSION=1.38.0 ARG BUSYBOX_VERSION=1.38.0
RUN apk add --no-cache build-base linux-headers wget bzip2 perl upx # Target platform ARGs (provided automatically by buildx).
ARG TARGETPLATFORM
ARG TARGETARCH
ARG TARGETVARIANT
# Native build tools (clang/lld for cross-compiling; gcc/make/upx run natively).
# xx-apk installs the target-arch sysroot: musl-dev (C library headers + CRT),
# gcc (provides crtbeginS.o/crtendS.o and libgcc needed by clang on Alpine),
# and linux-headers (required by busybox for <linux/*.h> / <net/*.h>).
RUN apk add --no-cache \
clang \
lld \
llvm \
gcc \
make \
wget \
bzip2 \
perl \
upx && \
xx-apk add --no-cache musl-dev gcc linux-headers
RUN wget -q https://busybox.net/downloads/busybox-${BUSYBOX_VERSION}.tar.bz2 \ RUN wget -q https://busybox.net/downloads/busybox-${BUSYBOX_VERSION}.tar.bz2 \
&& tar xf busybox-${BUSYBOX_VERSION}.tar.bz2 && tar xf busybox-${BUSYBOX_VERSION}.tar.bz2
@@ -269,7 +338,34 @@ WORKDIR /busybox-${BUSYBOX_VERSION}
# allnoconfig = every feature OFF; then enable only the curated applet set. # allnoconfig = every feature OFF; then enable only the curated applet set.
COPY busybox-applets.config /tmp/applets.config COPY busybox-applets.config /tmp/applets.config
RUN make allnoconfig && \ # Set up xx cross-compiler aliases (<triple>-clang, <triple>-cc, etc.) and
# build busybox.
#
# Key make variables:
# ARCH — busybox ARCH; must match the cross-target, not the build
# host. busybox's Makefile would otherwise read SUBARCH from
# `uname -m` (the BUILD host's arch) which is wrong when
# cross-compiling. We map TARGETARCH to busybox's arch name.
# busybox uses -include arch/$(ARCH)/Makefile; missing arch
# dirs are silently ignored, so any value is safe.
# CC — busybox defaults to $(CROSS_COMPILE)gcc. We override CC to
# the full <triple>-clang path so it resolves to the xx alias
# (which sets --target and --sysroot for the cross-compiler).
# Setting CC= avoids needing a <triple>-gcc symlink.
# HOSTCC — native compiler for host-side build tools (scripts/kconfig,
# gen_build_files, etc.); must NOT be the cross-compiler.
# SKIP_STRIP — defer stripping to after symlink probing (we strip below
# with llvm-strip, which handles any target ELF arch).
RUN xx-clang --setup-target-triple && \
CROSS=$(xx-info triple) && \
# Map TARGETARCH to the busybox ARCH value.
case "${TARGETARCH}" in \
amd64) BUSYBOX_ARCH=x86_64 ;; \
arm64) BUSYBOX_ARCH=aarch64 ;; \
arm) BUSYBOX_ARCH=arm ;; \
*) BUSYBOX_ARCH=${TARGETARCH} ;; \
esac && \
make allnoconfig ARCH="${BUSYBOX_ARCH}" && \
while read -r sym; do \ while read -r sym; do \
case "$sym" in ''|\#*) continue ;; esac; \ case "$sym" in ''|\#*) continue ;; esac; \
if grep -q "^# CONFIG_${sym} is not set" .config; then \ if grep -q "^# CONFIG_${sym} is not set" .config; then \
@@ -278,9 +374,15 @@ RUN make allnoconfig && \
echo "CONFIG_${sym}=y" >> .config; \ echo "CONFIG_${sym}=y" >> .config; \
fi; \ fi; \
done < /tmp/applets.config && \ done < /tmp/applets.config && \
yes "" | make oldconfig >/dev/null 2>&1 && \ yes "" | make oldconfig ARCH="${BUSYBOX_ARCH}" >/dev/null 2>&1 && \
make -j"$(nproc)" >/dev/null 2>&1 && \ make -j"$(nproc)" \
strip busybox ARCH="${BUSYBOX_ARCH}" \
CROSS_COMPILE="${CROSS}-" \
CC="${CROSS}-clang" \
HOSTCC=gcc \
SKIP_STRIP=y \
>/dev/null 2>&1 && \
llvm-strip busybox
# Lay out a minimal rootfs with busybox + an applet symlink per applet. # Lay out a minimal rootfs with busybox + an applet symlink per applet.
# Symlinks (argv[0] dispatch) are how busybox selects an applet and make the # Symlinks (argv[0] dispatch) are how busybox selects an applet and make the
@@ -290,6 +392,10 @@ RUN make allnoconfig && \
# for non-applet symbols like FEATURE_* / STATIC, which we filter out). # for non-applet symbols like FEATURE_* / STATIC, which we filter out).
# We generate symlinks from the UNCOMPRESSED binary (so the probe is reliable), # We generate symlinks from the UNCOMPRESSED binary (so the probe is reliable),
# then UPX-compress the binary in place afterwards. # then UPX-compress the binary in place afterwards.
#
# Note: probing cross-compiled binaries requires binfmt/QEMU user-mode. This
# is only a lightweight per-applet help-flag check, not a full emulated build.
# If QEMU is unavailable in CI, replace the probe with a static applet list.
RUN mkdir -p /rootfs/bin && \ RUN mkdir -p /rootfs/bin && \
grep '^CONFIG_.*=y' .config \ grep '^CONFIG_.*=y' .config \
| sed -e 's/^CONFIG_//' -e 's/=y$//' \ | sed -e 's/^CONFIG_//' -e 's/=y$//' \
+2 -1
View File
@@ -12,7 +12,8 @@
# #
# Requirements: # Requirements:
# - docker with buildx # - docker with buildx
# - For non-native targets: binfmt/QEMU emulators registered, e.g.: # - For non-native targets: binfmt/QEMU emulators registered for the applet
# symlink probe step (a minor step; the full C/Go compile is native):
# docker run --privileged --rm tonistiigi/binfmt --install arm64,arm # docker run --privileged --rm tonistiigi/binfmt --install arm64,arm
set -eu set -eu
+172 -15
View File
@@ -15,22 +15,26 @@ Measured flattened rootfs for the arm64 image:
| Component | On-disk size | | Component | On-disk size |
|---|---| |---|---|
| `tailscale.combined` (UPX-compressed) | ~2.98 MB | | `tailscale.combined` (UPX-compressed) | ~3.47 MB |
| custom static busybox (UPX, ~100 applets) | ~218 kB | | custom static busybox (UPX, ~100 applets) | ~218 kB |
| CA certificates | ~213 kB | | CA certificates | ~213 kB |
| **Total extracted rootfs** | **~3.4 MB** | | **Total extracted rootfs** | **~3.9 MB** |
(The compressed image / transfer tarball is ~3.34.3 MB depending on arch.) The `tailscale.combined` figure includes `netstack` (gVisor), which adds
~0.5 MB on disk over a netstack-omitted build — a deliberate inclusion, see
[Why netstack is required (even with a kernel TUN)](#why-netstack-is-required-even-with-a-kernel-tun).
(The compressed image / transfer tarball is ~3.84.3 MB depending on arch.)
| Arch | Image (compressed) | | Arch | Image (compressed) |
|---|---| |---|---|
| amd64 | ~4.2 MB | | amd64 | ~4.3 MB |
| arm64 | ~3.5 MB | | arm64 | ~4.0 MB |
| arm/v7 | ~3.5 MB | | arm/v7 | ~4.0 MB |
On a deployed RouterOS device the container consumes **~3.7 MiB of flash** On a deployed RouterOS device the container consumes **~4.2 MiB of flash**
(measured by `free-hdd-space` delta). Note that `du` *inside* the container (measured by `free-hdd-space` delta). Note that `du` *inside* the container
reports roughly double that (~7 MB) — that is RouterOS block-allocation reports roughly double that (~8 MB) — that is RouterOS block-allocation
rounding, **not** real usage or duplication; see rounding, **not** real usage or duplication; see
[Avoiding overlayfs layer duplication](#avoiding-overlayfs-layer-duplication) [Avoiding overlayfs layer duplication](#avoiding-overlayfs-layer-duplication)
for how to measure correctly. for how to measure correctly.
@@ -118,13 +122,13 @@ delta**, not `du`:
/system/resource/print # note free-hdd-space before and after adding the container /system/resource/print # note free-hdd-space before and after adding the container
``` ```
The container should consume **~3.7 MiB** of flash (e.g. 94.6 → 90.9 MiB free). The container should consume **~4.2 MiB** of flash (e.g. 94.6 → 90.4 MiB free).
Do **not** trust `du` inside the container for this. Busybox `du` reports Do **not** trust `du` inside the container for this. Busybox `du` reports
*allocated blocks*, and RouterOS's container store rounds a ~3 MB file up to *allocated blocks*, and RouterOS's container store rounds the ~3.5 MB binary up
~6 MB of blocks — so `du -sx /` reports ~7 MB even though real flash use is to ~7 MB of blocks — so `du -sx /` reports ~8 MB even though real flash use is
~3.7 MB. `ls -la /usr/local/bin` confirms the binary's true content size ~4.2 MB. `ls -la /usr/local/bin` confirms the binary's true content size
(~3.1 MB) and that it is a single file with two symlinks (no duplication). (~3.5 MB) and that it is a single file with two symlinks (no duplication).
The image itself carries the binary in exactly one layer (verified at the blob The image itself carries the binary in exactly one layer (verified at the blob
level); the inflation is purely the filesystem's block accounting. level); the inflation is purely the filesystem's block accounting.
@@ -149,7 +153,9 @@ that's a separate build, not just a `--platform` change.
| `advertise-routes` | Expose LAN subnets to the tailnet | | `advertise-routes` | Expose LAN subnets to the tailnet |
| `use-exit-node` | Route the router's own traffic via a remote exit node | | `use-exit-node` | Route the router's own traffic via a remote exit node |
| `accept-routes` | Receive subnet routes from other tailnet nodes | | `accept-routes` | Receive subnet routes from other tailnet nodes |
| DNS / MagicDNS | Resolve `*.ts.net` names | | DNS / MagicDNS | Resolve `*.ts.net` names (resolver + resolv.conf manager). **Note:** serving `100.100.100.100` also requires `netstack` — see [Why netstack is required (even with a kernel TUN)](#why-netstack-is-required-even-with-a-kernel-tun) |
| `netstack` + `gro` | gVisor userspace stack. Counter-intuitively **required** to serve MagicDNS on `100.100.100.100`, even though the router uses a real kernel TUN — see [Why netstack is required (even with a kernel TUN)](#why-netstack-is-required-even-with-a-kernel-tun) |
| `peerapiserver` | Serves the PeerAPI, including the `/dns-query` DoH endpoint that lets **exit-node clients resolve public DNS automatically**. A declared dependency of `advertise-exit-node` that the allowlist didn't pull in — see [Why peerapiserver is required for exit-node DNS](#why-peerapiserver-is-required-for-exit-node-dns) |
| portmapper (NAT-PMP/PCP/UPnP) | Punch through upstream NAT | | portmapper (NAT-PMP/PCP/UPnP) | Punch through upstream NAT |
| listenrawdisco | Raw socket disco for better NAT traversal | | listenrawdisco | Raw socket disco for better NAT traversal |
| health | Powers `tailscale status` output | | health | Powers `tailscale status` output |
@@ -166,7 +172,6 @@ that's a separate build, not just a `--platform` change.
| `cachenetmap` | **Deliberately removed** — see [Why netmap disk-caching is removed](#why-netmap-disk-caching-is-removed) | | `cachenetmap` | **Deliberately removed** — see [Why netmap disk-caching is removed](#why-netmap-disk-caching-is-removed) |
| `logtail` | Would attempt persistent log writes; wear flash. Removing it also removes stderr verbosity filtering — restored by an injected filter, see [Log verbosity filtering](#log-verbosity-filtering) | | `logtail` | Would attempt persistent log writes; wear flash. Removing it also removes stderr verbosity filtering — restored by an injected filter, see [Log verbosity filtering](#log-verbosity-filtering) |
| `netlog` | Network flow logging; separate concern | | `netlog` | Network flow logging; separate concern |
| `netstack` + `gro` | Userspace/gVisor networking; router uses kernel TUN |
| `ssh` | Access via MikroTik SSH + `tailscale` CLI instead | | `ssh` | Access via MikroTik SSH + `tailscale` CLI instead |
| `linuxdnsfight` | inotify on `/etc/resolv.conf`; no systemd in container | | `linuxdnsfight` | inotify on `/etc/resolv.conf`; no systemd in container |
| `networkmanager` / `resolved` / `dbus` / `sdnotify` | No systemd stack in container | | `networkmanager` / `resolved` / `dbus` / `sdnotify` | No systemd stack in container |
@@ -226,6 +231,158 @@ the in-memory resilience (the common case) while eliminating per-netmap flash
writes. Only `tailscaled.state` (written on auth / key rotation) ever touches writes. Only `tailscaled.state` (written on auth / key rotation) ever touches
flash. flash.
### Why netstack is required (even with a kernel TUN)
This is the least obvious inclusion in the build, so it is documented in full.
`netstack` is Tailscale's embedded **gVisor userspace TCP/IP stack**. The
natural assumption — and what earlier versions of this build acted on — is that
a router which owns a **real kernel TUN device** (it is *not* run with
`--tun=userspace-networking`) has no use for a userspace stack, so `netstack`
(and its dependent `gro`) can be omitted to save space. That assumption is
**wrong for one specific, important path: MagicDNS.**
**MagicDNS on `100.100.100.100` is served only by netstack.** In Tailscale
v1.98.5 the in-process listener for the Tailscale service IP
(`100.100.100.100:53`, UDP) is installed exclusively by netstack's
`handleLocalPackets`, wired into the TUN wrapper as
`PreFilterPacketOutboundToWireGuardNetstackIntercept`
(`wgengine/netstack/netstack.go`). When a packet leaves the host toward
`100.100.100.100`, this hook absorbs it into the gVisor stack, whose UDP-53
acceptor runs the MagicDNS resolver.
**The "engine fallback" does not actually exist.** The TUN wrapper consults a
second hook, `PreFilterPacketOutboundToWireGuardEngineIntercept`, and a comment
in `net/tstun/wrap.go` claims it "primarily handles quad-100 if netstack is not
installed." In v1.98.5 that comment is **false on Linux**: the engine
`handleLocalPackets` (`wgengine/userspace.go`) only reflects loopback on
darwin/ios/plan9 and otherwise returns `Accept` — it never touches
`100.100.100.100`. So with `ts_omit_netstack` there is **no** code that absorbs
quad-100 packets at all.
**`dns` and `netstack` are independent tags.** The `dns` feature (which this
build opts in) links the resolver and the `/etc/resolv.conf` manager, but it has
no dependency on `netstack` and does **not** install any quad-100 transport.
The net result of `dns` on + `netstack` off is a resolver that is correctly
wired up but that **never receives any packets** — the worst kind of silent
breakage. Symptoms observed on the device:
- `/etc/resolv.conf` correctly points at `100.100.100.100` (the manager works),
- but `dig anything @100.100.100.100` from inside the container **times out**
("no servers could be reached"),
- and even tailnet-internal names fail: `ping host.<tailnet>.ts.net`
`bad address` (a name that needs **no** upstream forwarding still can't
resolve, proving the listener itself is dead, not an upstream-resolver issue),
- while `ping 1.1.1.1` (a raw IP needing no DNS) works fine over the kernel data
path — confirming forwarding/exit-node connectivity is unaffected and isolating
the fault to DNS serving.
**It also fixed a crash.** Omitting `netstack` set `buildfeatures.HasNetstack`
to a compile-time `false`, which turned the guard in
`net/tstun.invertGSOChecksum` (`if !HasNetstack { panic("unreachable") }`) into
an always-panic. That function is called on the packet-injection path used when
enabling exit-node mode, producing `panic: unreachable` and a daemon restart
loop. Enabling `netstack` makes `HasNetstack` a const `true`, so the guard
becomes dead code and the crash disappears as a side effect — fixed at the root
cause rather than patched around.
**Cost.** Measured on arm64, a netstack-enabled build versus a netstack-omitted
one:
| Metric | netstack omitted | netstack enabled | Delta |
|---|---|---|---|
| Extracted rootfs (flash) | ~3.42 MB | ~3.91 MB | **+0.49 MB** |
| `tailscale.combined` on disk (UPX) | ~2.99 MB | ~3.47 MB | +0.48 MB |
| Resident RAM after UPX decompress | ~12.25 MB | ~14.56 MB | **+2.31 MB** |
The flash cost (~0.5 MB) is negligible on a 16 MB-class device. The RAM cost
(~2.3 MB resident) is the real consideration on low-memory models, but is
acceptable given that without it MagicDNS is entirely non-functional. The
trade is: **half a megabyte of flash to make MagicDNS work at all.** `gro`
(Generic Receive Offload) depends on `netstack` and is pulled in alongside it;
it is small and improves throughput on the netstack path.
**Caveat for future Tailscale bumps.** This coupling (quad-100 serving living
only in netstack) is an upstream implementation detail, not a stable contract.
If a future release adds a genuine non-netstack quad-100 path — or the daemon
itself is refactored — re-test whether `netstack` can be dropped again. The
canary is simple: from inside the container, `dig google.com @100.100.100.100`
must return answers and `ping <host>.<tailnet>.ts.net` must resolve.
### Why peerapiserver is required for exit-node DNS
This is a second non-obvious DNS inclusion, and it exposes a limitation of the
allowlist build strategy.
**Symptom.** With `netstack` enabled, MagicDNS worked from the router and from
LAN hosts, including public names. But a device using this router **as its exit
node** could not resolve public names: `dig google.com @100.100.100.100` on the
*client* returned an instant authoritative `SERVFAIL` (`flags: qr aa rd ad`,
`Query time: 0 msec`, "recursion not available"). Tailnet names and raw-IP
connectivity (e.g. `ping 1.1.1.1`) through the exit node worked.
**Root cause.** The `SERVFAIL` is generated **on the client**, locally, with no
network I/O — which is why it is instant and authoritative. The path
(traced through v1.98.5 source):
1. The client's query for `google.com` reaches its in-process resolver, which
determines the name is not a tailnet name and marks it for forwarding
(`net/dns/resolver/tsdns.go`).
2. The forwarder looks up which upstream resolver to use for the catch-all
`"."` route (`net/dns/resolver/forwarder.go``resolvers()`).
3. That route set is **empty**, so `forwardWithDestChan` short-circuits and
synthesises an authoritative `SERVFAIL` (`servfailResponse`, `aa=1`) without
opening any socket. The query never reaches this router at all.
Why the route set is empty: when a client selects an exit node,
`dnsConfigForNetmap` (`ipn/ipnlocal/node_backend.go`) deliberately routes **all**
default DNS through the exit node and drops the client's own LAN/system
resolver — the whole premise of an exit node is "send everything, including
DNS, through me." It does this by setting the client's default resolver to the
exit node's **DoH proxy** URL (`http://<peer>/dns-query`). But that only happens
if `exitNodeCanProxyDNS(thisRouter)` returns true — i.e. if **this router
advertises a working PeerAPI DoH endpoint**. If it does not, and there is no
tailnet global nameserver to fall back to, the client ends up with an empty
default route and returns `SERVFAIL`.
**Why this router didn't advertise the DoH proxy.** The `/dns-query` DoH
endpoint is part of the **PeerAPI server**, gated by
`buildfeatures.HasPeerAPIServer` (`ipn/ipnlocal/peerapi.go`). With
`ts_omit_peerapiserver`, `initPeerAPIListenerLocked()` returns early: no PeerAPI
listener is created, the `PeerAPIDNS` service is never advertised, and
`peerCanProxyDNS()` is false for this node on every client.
**The allowlist gap that caused it.** In `feature/featuretags/featuretags.go`,
`advertiseexitnode` **declares a dependency on `peerapiserver`** ("to run the
ExitDNS server"). Upstream's own `--add` resolution would have pulled it in.
But this build's allowlist works differently: it runs `featuretags --min` to get
the full omit set, then strips the specific `ts_omit_<feature>` tags it wants —
it does **not** re-resolve transitive `Deps`. So opting in `advertiseexitnode`
did not pull in `peerapiserver`, and `featuretags --min` had emitted
`ts_omit_peerapiserver`, leaving the node an exit node *without* its declared
ExitDNS dependency — a feature combination upstream's graph says shouldn't
occur. Including `peerapiserver` explicitly closes the gap.
> **Known limitation:** the allowlist (strip-individual-`ts_omit_`-tags) does
> not resolve feature dependencies. When opting a feature in, check its `Deps`
> in `featuretags.go` and add them explicitly. `peerapiserver` is the only such
> gap found and fixed so far; a full dependency audit has not been done.
**Cost.** Negligible. `peerapiserver` has **no** `Deps` and pulls in no large
subsystems; measured at ~+10 kB on the UPX'd binary (arm64), rootfs unchanged
within measurement noise.
**Result.** The router now serves the exit-node DoH DNS proxy, so devices using
it as their exit node resolve public names automatically — the normal exit-node
behavior — with **no** tailnet DNS configuration required. (Setting a tailnet
global nameserver in the admin console is an alternative runtime fix that also
works, by populating the client's default resolver directly; it is not required
once the router serves the proxy.)
**Canary for future bumps:** from a client using this router as exit node,
`dig google.com @100.100.100.100` must return real answers with `flags: ... ra`
(recursion available) and a non-zero query time.
### Log verbosity filtering ### Log verbosity filtering
Upstream `tailscaled` embeds verbosity tags (`[v1]`, `[v2]`, …) inside its log Upstream `tailscaled` embeds verbosity tags (`[v1]`, `[v2]`, …) inside its log