Homelab

Goals

Wanting to set up homelab kubernetes cluster.

Software

1. Running applications
   1. NAS, backups, security recorder
   2. Online presence, website, email, communicators (ts3, matrix?)
   3. Git server, container registry
   4. Environment to deploy my own apps
   5. Some LLM server, apps for my own use
   6. Public services like Tor, mirrors of linux distros etc.
   7. Some frontends
   8. Awesome-Selfhosted, Awesome Sysadmin
2. Managing them hopefully using GitOps
   1. FluxCD, Argo etc.
   2. State of cluster in git, all apps version pinned
   3. Some bot to inform about updates?
3. It's a homelab
   1. Should be open to experimenting
   2. Avoiding vendor lock-in, changing my mind shouldn't block me for too long
   3. Backups of important data in easy to access format
   4. Expecting downtime, no critical workloads
   5. Trying to keep it reasonably up anyways

Infrastructure

1. Using commodity hardware
2. Reasonably scalable
3. Preferably mobile workloads, software should be a bit more flexible than me moving disks and data
4. Replication is overkill for most data
5. Preferably dynamically configured network
   1. BGP with OpenWRT router
   2. Dynamically allocated host subnets
   3. Load-balancing (MetalLB?), ECMP on router
   4. Static IP configurations on nodes
6. IPv6 native, IPv4 accessible
   1. IPv6 has whole block routed to us which gives us control over address routing and usage
   2. Which allows us to expose services directly to the internet without complex router config
   3. Which allows us to use eg. ExternalDNS to autoconfigure domain names for LB
   4. But majority of the world still runs IPv4, which should be supported for public services
   5. Exposing IPv4 service may require additional reconfiguration of router, port forwarding, manual domain setting or controller doing this some day in future
   6. One public IPv4 address means probably extensive use of rule-based ingress controllers
   7. IPv6 internet from pods should not be NATed
   8. IPv4 internet from pods should be NATed by router

Current implementation idea

1. Cluster server nodes running Talos
2. OpenWRT router
   1. VLAN / virtual interface, for cluster
   2. Configuring using Ansible
   3. Peering with cluster using BGP
   4. Load-balancing using ECMP
3. Cluster networking
   1. Cilium CNI
   2. Native routing, no encapsulation or overlay
   3. Using Cilium's network policies for firewall needs
   4. IPv6 address pool
      1. Nodes: 2001:470:61a3:100::/64
      2. Pods: 2001:470:61a3:200::/64
      3. Services: 2001:470:61a3:300::/112
      4. Load balancer: 2001:470:61a3:400::/112
   5. IPv4 address pool
      1. Nodes: 192.168.1.32/27
      2. Pods: 10.42.0.0/16
      3. Services: 10.43.0.0/16
      4. Load balancer: 10.44.0.0/16
4. Storage
   1. OS is installed on dedicated disk
   2. Mayastor managing all data disks
      1. DiskPool for each data disk in cluster, labelled by type SSD or HDD
      2. Creating StorageClass for each topology need (type, whether to replicate, on which node etc.)

Working with repo

Repo is preconfigured to use with nix and vscode

Install nix, vscode should pick up settings and launch terminals in nix develop with all needed utils.

Bootstrapping cluster

1. Configure OpenWRT, create dedicated interface for connecting server
   1. Set up node subnet, routing
   2. Create static host entry kube-api.homelab.lumpiasty.xyz pointing at ipv6 of first node
2. Connect server
3. Grab Talos ISO, dd it to usb stick
4. Boot it and using keyboard set up static ip ipv6 subnet, should become reachable from pc
5. talosctl gen config homelab https://kube-api.homelab.lumpiasty.xyz:6443
6. Generate secrets talosctl gen secrets, backup, keep secrets.yml safe
7. Generate config files make gen-talos-config
8. Apply config to first node talosctl apply-config --insecure -n 2001:470:61a3:100::2 -f controlplane.yml
9. Wait for reboot then talosctl bootstrap --talosconfig=talosconfig -n 2001:470:61a3:100::2
10. Set up router and CNI

Updating Talos config

Update patches and re-generate and apply configs.

make gen-talos-config
make apply-talos-config