Running WireGuard inside Kubernetes

Have ended up with following

The goal here is to run WireGuard directly inside Kubernetes, which will allow us to resolve and talk to its services, pods, etc

LinuxServer/WireGuard

There is linuxserver/wireguard

Here is working sample:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wireguard
  labels:
    app: wireguard
  annotations:
    kubeforce: skip
spec:
  selector:
    matchLabels:
      app: wireguard
  template:
    metadata:
      labels:
        app: wireguard
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      # initContainers:
      #   - args:
      #     - -c
      #     - |
      #       sysctl -w net.ipv4.ip_forward=1
      #       sysctl -w net.ipv4.conf.all.src_valid_mark=1
      #     # - sysctl -w net.ipv4.ip_forward=1
      #     # - sysctl -w net.ipv4.conf.all.src_valid_mark=1 # Required for client mode
      #     command:
      #     - /bin/sh
      #     image: busybox:1.29
      #     name: sysctl
      #     securityContext:
      #       privileged: true

      # securityContext:
      #   runAsUser: 1000
      #   runAsGroup: 1000
      containers:
      - name: wireguard
        image: linuxserver/wireguard
        imagePullPolicy: IfNotPresent
        env:
        - name: PUID
          value: "1000"
        - name: PGID
          value: "1000"
        - name: TZ
          value: "Europe/Kiev"
        - name: SERVERURL
          value: "4.21.11.210"
        - name: SERVERPORT
          value: "51820"
        - name: PEERS
          value: "1"
        - name: PEERDNS
          value: "10.0.0.10"
        # - name: INTERNAL_SUBNET
        #   # TODO: narrow down?
        #   value: "10.0.0.0/16"
        - name: ALLOWEDIPS
          # ip addresses that will be routed via this vpn
          # value: "0.0.0.0/0, ::/0"
          value: "10.0.0.0/8, 34.11.5.8/32, 20.13.19.8/32"
        - name: LOG_CONFS
          value: "true"
        # Set to `all` or a list of comma separated peers (ie. `1,4,laptop`) for the wireguard server to send keepalive packets to listed peers every 25 seconds. Useful if server is accessed via domain name and has dynamic IP. Used only in server mode.
        - name: PERSISTENTKEEPALIVE_PEERS
          value: "all"
        ports:
        - containerPort: 51820
          protocol: UDP
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 64Mi
        securityContext:
          # privileged: true
          capabilities:
            add:
            - NET_ADMIN
            # - SYS_MODULE
---
apiVersion: v1
kind: Service
metadata:
  name: wireguard
spec:
  type: LoadBalancer
  selector:
    app: wireguard
  ports:
    - port: 51820
      protocol: UDP

Notes:

  • I was experimenting with Azure AKS so notes may be not aplicable everywhere
  • There is no need to touch sysctl - packes successfully routed as wanted
  • There is no need to run in privileged mode
  • There is no need to map and deal with modules - wireguard is build into moder kernels
  • Server URL points to public IP address of our load balancer service

With such setup I was able to run it, grab config and successfully connect

Also the end goal was reached, because we are pointing DNS to CoreDNS I was able to resolve and talk to Kubernetes services

The drawback is that container is kind of static, which means it is good if you are single user or do not need to manage peers dynamically

Alternative Images

There is quadrillion of alternative images, even more, many of them are talking about OIDC, OAuth, 2FA, etc

But at the very end it is only partially truth

Actually what all they are doing is to have some kind of UI to manage WireGuard peer configurations and to access that UI you need to be authenticated

But the keys themselve have nothing to do with any auth provider and will work no matter what

Also the client itself is not aware of anything around this

From my understanding the idea here is just to revoke peers by schedule to enforce end user to relogin into UI and generate new keys

WireGuard from scratch

After realizing that WireGuard is build into the Kernel it seems that there is no need to dedicated image and it should work out of the box

I have started with Ubuntu, but:

apt update && apt install -y wireguard

told me that it will install 800Mb of software

Note: even so WireGuard is build into Kernel, we still need its tools to manage peers, that's why I'm installing it

But on the other hand Alpine image has wireguard-tools package which makes everything much simpler

To proove it might work I have deployed following:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  labels:
    app: demo
  annotations:
    kubeforce: skip
spec:
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      nodeSelector:
        kubernetes.io/os: linux
        poolDestination: app
      # initContainers:
      #   - args:
      #     - -c
      #     - |
      #       sysctl -w net.ipv4.ip_forward=1
      #       sysctl -w net.ipv4.conf.all.src_valid_mark=1
      #     # - sysctl -w net.ipv4.ip_forward=1
      #     # - sysctl -w net.ipv4.conf.all.src_valid_mark=1 # Required for client mode
      #     command:
      #     - /bin/sh
      #     image: busybox:1.29
      #     name: sysctl
      #     securityContext:
      #       privileged: true

      # securityContext:
      #   runAsUser: 1000
      #   runAsGroup: 1000
      containers:
      - name: demo
        image: nginx:alpine
        ports:
        - name: http
          containerPort: 80
          protocol: TCP
        - name: wireguard
          containerPort: 51820
          protocol: UDP
        resources:
          limits:
            cpu: 500m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 256Mi
        securityContext:
          # privileged: true
          capabilities:
            add:
            - NET_ADMIN
            # - SYS_MODULE
---
apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  type: LoadBalancer
  selector:
    app: demo
  ports:
    - name: http
      port: 80
      protocol: TCP
    - name: wireguard
      port: 51820
      protocol: UDP

Notes:

  • I have deployed nginx:alpine so it is Alpine and it wont exit immediately
  • Everything else it similar to previous deployment

Here is outlines from experiment

kubectl apply -f alpine.yml

kubectl get svc demo
# 51.138.228.184

kubectl get po -l app=demo
kubectl exec -it demo-85b7d9f77f-8dx9r -- sh


export IP=51.138.228.184

# install wireguard-tools, the wireguard itself is build into the kernel, we need the `wg` tool
apk add wireguard-tools

# create and configure wg0 interface
ip link add wg0 type wireguard
ip -4 address add 10.14.14.1 dev wg0
ip link set mtu 1420 up dev wg0
ip -4 route add 10.14.14.0/24 dev wg0

# configure iptables to masquarade (nat), requires NET_ADMIN capabilities
iptables -A FORWARD -i wg0 -j ACCEPT
iptables -A FORWARD -o wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# check (should show interface with private ip)
ip addr show wg0
# check (should show interface with some random listening port)
wg

# keys
wg genkey | tee /etc/wireguard/server.privatekey | wg pubkey > /etc/wireguard/server.publickey

tee /etc/wireguard/wg0.conf > /dev/null <<EOT
[Interface]
# server private ip address
Address = 10.14.14.1
# server private key
PrivateKey = $(cat /etc/wireguard/server.privatekey)
# port to listen
ListenPort = 51820

EOT

# apply
wg syncconf wg0 <(wg-quick strip wg0)

# check (should show interface with keys and desired 51820 listening port)
wg


# ADDING PEER

wg genkey | tee /etc/wireguard/peer1.privatekey | wg pubkey > /etc/wireguard/peer1.publickey

# append peer to wg0.conf
tee -a /etc/wireguard/wg0.conf > /dev/null <<EOT
[Peer]
# client public key
PublicKey = $(cat /etc/wireguard/peer1.publickey)
# client private ip
AllowedIPs = 10.14.14.2/32

EOT


# apply
wg syncconf wg0 <(wg-quick strip wg0)

# check (should show not only wg0, but added peer as well)
wg


tee /etc/wireguard/peer1.conf > /dev/null <<EOT
[Interface]
# client private ip
Address = 10.14.14.2
# client private key
PrivateKey = $(cat /etc/wireguard/peer1.privatekey)
DNS = 10.0.0.10

[Peer]
# server public key
PublicKey = $(cat /etc/wireguard/server.publickey)
# server public ip and port
Endpoint = $IP:51820
# ip addresses that should be routed via vpn, 34.117.59.81 for ipinfo.io
AllowedIPs = 10.0.0.0/8, 34.117.59.81/32, 20.13.179.68/32
PersistentKeepalive = 25
EOT

cat /etc/wireguard/peer1.conf


# CHECKS

# ping vpn
ping 10.14.14.1
# prometheus svc
nc -vz 10.0.15.183 80
# prometheus pod
nc -vz 10.64.2.5 9090
# coredns svc
nc -vz 10.0.0.10 53
# dns
nc -vz prometheus 80
# sql
nc -vz 10.54.10.4 1433
# iis
nc -vz 10.54.11.4 80
# nat
curl ipinfo.io

# SECOND CLIENT

wg genkey | tee /etc/wireguard/peer2.privatekey | wg pubkey > /etc/wireguard/peer2.publickey

# append peer to wg0.conf
tee -a /etc/wireguard/wg0.conf > /dev/null <<EOT
[Peer]
# client public key
PublicKey = $(cat /etc/wireguard/peer2.publickey)
# client private ip
AllowedIPs = 10.14.14.3/32

EOT


# apply
wg syncconf wg0 <(wg-quick strip wg0)

# check (should show not only wg0, but added peer as well)
wg


tee /etc/wireguard/peer2.conf > /dev/null <<EOT
[Interface]
# client private ip
Address = 10.14.14.3
# client private key
PrivateKey = $(cat /etc/wireguard/peer2.privatekey)
DNS = 10.0.0.10

[Peer]
# server public key
PublicKey = $(cat /etc/wireguard/server.publickey)
# server public ip and port
Endpoint = $IP:51820
# ip addresses that should be routed via vpn, 34.117.59.81 for ipinfo.io
AllowedIPs = 10.0.0.0/8, 34.117.59.81/32, 20.13.179.68/32
PersistentKeepalive = 25
EOT

cat /etc/wireguard/peer2.conf

# CHECKS
# same checks as for client 1

# REMOVING CLIENT 2

# remove corresponding peer from wg0.conf (todo: script)

# apply
wg syncconf wg0 <(wg-quick strip wg0)

# check (second peer should dissapear, even if he was connected it will stop working)
wg

# CLEANUP

ip link delete dev wg0
iptables -D FORWARD -i wg0 -j ACCEPT
iptables -D FORWARD -o wg0 -j ACCEPT
iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

So we actualy may have wireguard up and running out of the box

Beside that - if instead of nginx we will run some kind of our UI we actually will have such an alternative image

alternative experiment without config files

following experiment demonstrates alternative approach where we are not dealing with configuration files at all, everything is done wia wg commands (the idea behind was to dynamically fill peers instead of parsing and writing ini configuration files)

kubectl apply -f alpine.yml

kubectl get svc demo
# 51.138.227.23

kubectl get po -l app=demo
kubectl exec -it demo-85b7d9f77f-6fqs5 -- sh


export IP=51.138.227.23

apk add wireguard-tools


ip link add wg0 type wireguard
ip -4 address add 10.14.14.1 dev wg0
ip link set mtu 1420 up dev wg0
ip -4 route add 10.14.14.0/24 dev wg0
iptables -A FORWARD -i wg0 -j ACCEPT
iptables -A FORWARD -o wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# check (should show interface with private ip)
ip addr show wg0
# check (should show interface with some random listening port)
wg

# keys
wg genkey | tee /etc/wireguard/server.privatekey | wg pubkey > /etc/wireguard/server.publickey

# apply private key, configure listening port
wg set wg0 listen-port 51820 private-key /etc/wireguard/server.privatekey

# check (should show interface with keys and desired 51820 listening port)
wg


# ADDING PEER

wg genkey | tee /etc/wireguard/peer1.privatekey | wg pubkey > /etc/wireguard/peer1.publickey

wg set wg0 peer "$(cat /etc/wireguard/peer1.publickey)" allowed-ips 10.14.14.2/32

# check (should show not only wg0, but added peer as well)
wg


tee /etc/wireguard/peer1.conf > /dev/null <<EOT
[Interface]
# client private ip
Address = 10.14.14.2
# client private key
PrivateKey = $(cat /etc/wireguard/peer1.privatekey)
DNS = 10.0.0.10

[Peer]
# server public key
PublicKey = $(cat /etc/wireguard/server.publickey)
# server public ip and port
Endpoint = $IP:51820
# ip addresses that should be routed via vpn, 34.117.59.81 for ipinfo.io
AllowedIPs = 10.0.0.0/8, 34.117.59.81/32, 20.13.179.68/32
PersistentKeepalive = 25
EOT

# print config for a client
cat /etc/wireguard/peer1.conf
# remove sensitive files, no need to store them
rm /etc/wireguard/peer1.conf
rm /etc/wireguard/peer1.privatekey


# ...

# https://serverfault.com/questions/1101002/wireguard-client-addition-without-restart

# REMOVING CLIENT

wg set wg0 peer "$(cat /etc/wireguard/peer1.publickey)" remove

So in general we may start from alpine, install wireguard tools and put our app which will just call all that commands, e.g. something like:

package wireguard

import (
	"errors"
	"fmt"
	"io"
	"net/netip"
	"os"
	"os/exec"
	"strings"
)

type Peer struct {
	PublicKey           string
	PresharedKey        string
	Endpoint            string
	AllowedIPs          string
	LatestHandshake     string
	TransferRX          string
	TransferTX          string
	PersistentKeepalive string
}

type WireGuard struct {
	cidr             string
	serverPrivateIP  string
	serverPublicIP   string
	serverPrivateKey string
	serverPublicKey  string
	dns              string
	allowedIPs       string
	port             string
}

type PeerConfig struct {
	ClientPrivateIP     string
	ClientPrivateKey    string
	DNS                 string
	ServerPublicKey     string
	ServerPublicIP      string
	ServerPort          string
	AllowedIPs          string
	PersistentKeepalive bool
}

func NewWireGuard(cidr string, serverPrivateKey string, serverPublicKey string, serverPublicIP string, dns string, allowedIPs string) (*WireGuard, error) {
	if !netip.MustParsePrefix(cidr).IsValid() {
		return nil, errors.New("invalid cidr")
	}
	if !netip.MustParsePrefix(cidr).Addr().Next().Is4() {
		return nil, errors.New("non ip4 cidr")
	}
	serverPrivateIP := netip.MustParsePrefix(cidr).Addr().Next().String()

	_, err := exec.Command("ip", "link", "add", "wg0", "type", "wireguard").Output()
	if err != nil {
		return nil, err
	}

	_, err = exec.Command("ip", "-4", "address", "add", serverPrivateIP, "dev", "wg0").Output()
	if err != nil {
		return nil, err
	}

	_, err = exec.Command("ip", "link", "set", "mtu", "1420", "up", "dev", "wg0").Output()
	if err != nil {
		return nil, err
	}

	_, err = exec.Command("ip", "-4", "route", "add", cidr, "dev", "wg0").Output()
	if err != nil {
		return nil, err
	}

	_, err = exec.Command("iptables", "-A", "FORWARD", "-i", "wg0", "-j", "ACCEPT").Output()
	if err != nil {
		return nil, err
	}

	_, err = exec.Command("iptables", "-A", "FORWARD", "-o", "wg0", "-j", "ACCEPT").Output()
	if err != nil {
		return nil, err
	}

	_, err = exec.Command("iptables", "-t", "nat", "-A", "POSTROUTING", "-o", "eth0", "-j", "MASQUERADE").Output()
	if err != nil {
		return nil, err
	}

	// TODO: server keys should be stored and passed here as parameters
	// serverPrivateKey, serverPublicKey, err := generateWireGuardKeys()
	// if err != nil {
	// 	return nil, err
	// }
	os.WriteFile("/etc/wireguard/server.privatekey", []byte(serverPrivateKey), 0600)

	_, err = exec.Command("wg", "set", "wg0", "listen-port", "51820", "private-key", "/etc/wireguard/server.privatekey").Output()
	if err != nil {
		return nil, err
	}

	return &WireGuard{
		cidr:             cidr,
		serverPrivateIP:  serverPrivateIP,
		serverPublicIP:   serverPublicIP,
		serverPrivateKey: serverPrivateKey,
		serverPublicKey:  serverPublicKey,
		dns:              dns,
		allowedIPs:       allowedIPs,
		port:             "51820",
	}, nil
}

func (w *WireGuard) ListPeers() (peers []Peer, err error) {
	b, err := exec.Command("wg", "show", "wg0", "dump").Output()
	if err != nil {
		return peers, err
	}
	for idx, line := range strings.Split(strings.Trim(string(b), "\n"), "\n") {
		if idx == 0 {
			continue
		}
		cells := strings.Split(line, "\t")
		peers = append(peers, Peer{
			PublicKey:           cells[0],
			PresharedKey:        cells[1],
			Endpoint:            cells[2],
			AllowedIPs:          cells[3],
			LatestHandshake:     cells[4],
			TransferRX:          cells[5],
			TransferTX:          cells[6],
			PersistentKeepalive: cells[7],
		})
	}
	return peers, err
}

func (w *WireGuard) AddPeer2() (privateKey string, publicKey string, ip string, err error) {
	ip, err = w.getNextIP()
	if err != nil {
		return "", "", "", err
	}

	privateKey, publicKey, err = GenerateWireGuardKeys()
	if err != nil {
		return "", "", "", err
	}

	_, err = exec.Command("wg", "set", "wg0", "peer", publicKey, "allowed-ips", ip).Output()
	if err != nil {
		return "", "", "", err
	}
	return privateKey, publicKey, ip, nil
}

func (w *WireGuard) AddPeer() (c PeerConfig, err error) {
	ip, err := w.getNextIP()
	if err != nil {
		return c, err
	}

	privateKey, publicKey, err := GenerateWireGuardKeys()
	if err != nil {
		return c, err
	}

	_, err = exec.Command("wg", "set", "wg0", "peer", publicKey, "allowed-ips", ip).Output()
	if err != nil {
		return c, err
	}
	return PeerConfig{
		ClientPrivateIP:     ip,
		ClientPrivateKey:    privateKey,
		DNS:                 w.dns,
		ServerPublicKey:     w.serverPublicKey,
		ServerPublicIP:      w.serverPublicIP,
		ServerPort:          w.port,
		AllowedIPs:          w.allowedIPs,
		PersistentKeepalive: true,
	}, nil
}

func (w *WireGuard) RemovePeer(peerPublicKey string) error {
	_, err := exec.Command("wg", "set", "wg0", "peer", peerPublicKey, "remove").Output()
	if err != nil {
		return err
	}
	return nil
}

func (w *WireGuard) getNextIP() (IP string, err error) {
	takenIPAddresses, err := w.ListPeers()
	if err != nil {
		return "", err
	}

	allIPAddresses, err := getAllIPAddressesFor(w.cidr)
	if err != nil {
		return "", err
	}

	for _, addr := range allIPAddresses {
		IP = fmt.Sprintf("%s/32", addr.String())
		if IP == fmt.Sprintf("%s/32", w.serverPrivateIP) {
			continue
		}
		found := false
		for _, peer := range takenIPAddresses {
			if peer.AllowedIPs == IP {
				found = true
				break
			}
		}
		if !found {
			return IP, err
		}
	}
	return "", errors.New("can not find next IP")
}

func GenerateWireGuardKeys() (privateKey string, publicKey string, err error) {
	b, err := exec.Command("wg", "genkey").Output()
	if err != nil {
		return "", "", err
	}
	privateKey = strings.Trim(string(b), "\n")

	c := exec.Command("wg", "pubkey")
	stdin, err := c.StdinPipe()
	if err != nil {
		return "", "", err
	}
	go func() {
		defer stdin.Close()
		io.WriteString(stdin, privateKey)
	}()
	b, err = c.CombinedOutput()
	if err != nil {
		return "", "", err
	}
	publicKey = strings.TrimSuffix(string(b), "\n")

	return privateKey, publicKey, nil
}

// https://gist.github.com/kotakanbe/d3059af990252ba89a82?permalink_comment_id=4105265#gistcomment-4105265
func getAllIPAddressesFor(cidr string) ([]netip.Addr, error) {
	prefix, err := netip.ParsePrefix(cidr)
	if err != nil {
		panic(err)
	}

	var ips []netip.Addr
	for addr := prefix.Addr(); prefix.Contains(addr); addr = addr.Next() {
		ips = append(ips, addr)
	}

	if len(ips) < 2 {
		return ips, nil
	}

	return ips[1 : len(ips)-1], nil
}

func (c PeerConfig) String() string {
	s := "[Interface]\n"
	s += fmt.Sprintf("Address = %s\n", c.ClientPrivateIP)
	s += fmt.Sprintf("PrivateKey = %s\n", c.ClientPrivateKey)
	if c.DNS != "" {
		s += fmt.Sprintf("DNS = %s\n", c.DNS)
	}
	s += "\n[Peer]\n"
	s += fmt.Sprintf("PublicKey = %s\n", c.ServerPublicKey)
	s += fmt.Sprintf("Endpoint = %s:%s\n", c.ServerPublicIP, c.ServerPort)
	if c.AllowedIPs != "" {
		s += fmt.Sprintf("AllowedIPs = %s\n", c.AllowedIPs)
	} else {
		s += "AllowedIPs = 0.0.0.0/0\n"
	}
	if c.PersistentKeepalive {
		s += "PersistentKeepalive = 25\n"
	}
	return s
}

package main

import (
	"fmt"
	"io"
	"net/http"
	"os/exec"

	"github.com/mac2000/gowg/wireguard"
)

func main() {
	serverPublicIP := "X.X.X.X"
	serverPrivateKey, serverPublicKey, err := wireguard.GenerateWireGuardKeys()
	if err != nil {
		panic(err)
	}

	wg, err := wireguard.NewWireGuard("10.14.14.0/24", serverPrivateKey, serverPublicKey, serverPublicIP, "10.0.0.10", "10.0.0.0/8, 34.117.59.81/32, 20.13.179.68/32")
	if err != nil {
		panic(err)
	}

	c, err := wg.AddPeer()
	if err != nil {
		panic(err)
	}
	fmt.Println(c)

	c, err = wg.AddPeer()
	if err != nil {
		panic(err)
	}
	fmt.Println(c)

	peers, err := wg.ListPeers()
	if err != nil {
		panic(err)
	}
	for _, peer := range peers {
		fmt.Println(peer)
		if peer.AllowedIPs == "(none)" {
			err := wg.RemovePeer(peer.PublicKey)
			if err != nil {
				panic(err)
			}
		}
	}

	b, err := exec.Command("wg").Output()
	if err != nil {
		panic(err)
	}
	fmt.Println(string(b))

	http.HandleFunc("/", getRoot)
	http.ListenAndServe(":80", nil)
}

func getRoot(w http.ResponseWriter, r *http.Request) {
	fmt.Printf("got / request\n")
	io.WriteString(w, "This is my website!\n")
}

note: yes, yes, i know there are libraries for that but they did not work out of the box and i am not sure whether they are working in user space or kernel space, that's why i did simple exec, also it will work with any language

With that in place we may start building our own UI

And for storage we may even use Kubernetes itself, e.g. how about using ConfigMap as a storage for a peer configs, aka:

config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(clientcmd.NewDefaultClientConfigLoadingRules(), &clientcmd.ConfigOverrides{}).ClientConfig()
if err != nil {
  panic(err.Error())
}

clientset, err := kubernetes.NewForConfig(config)
if err != nil {
  panic(err)
}

nsList, err := clientset.CoreV1().Namespaces().List(context.TODO(), metav1.ListOptions{})
if err != nil {
  panic(err)
}
fmt.Println("got", len(nsList.Items), "namespaces")

cm, err := clientset.CoreV1().ConfigMaps("dev").Get(context.Background(), "demo", metav1.GetOptions{})
if err != nil {
  c := &v1.ConfigMap{
    ObjectMeta: metav1.ObjectMeta{
      Name: "demo",
    },
    Data: map[string]string{
      "foo":  "bar",
      "acme": "42",
    },
  }
  _, err := clientset.CoreV1().ConfigMaps("dev").Create(context.Background(), c, metav1.CreateOptions{})
  if err != nil {
    panic(err)
  }
  fmt.Println("CREATED")
  cm, err = clientset.CoreV1().ConfigMaps("dev").Get(context.Background(), "demo", metav1.GetOptions{})
  if err != nil {
    panic(err)
  }
}

for k, v := range cm.Data {
  fmt.Println(k, v)
}

if cm.Data["foo"] != "Hello World" {
  cm.Data["foo"] = "Hello World"
  _, err := clientset.CoreV1().ConfigMaps("dev").Update(context.Background(), cm, metav1.UpdateOptions{})
  if err != nil {
    panic(err)
  }
  fmt.Println("UPDATED")
}

if cm.Data["acme"] != "5" {
  // clientset.CoreV1().ConfigMaps("dev").Patch(context.Background(), "demo", types.MergePatchType)
  payloadBytes, _ := json.Marshal([]patchStringValue{{
    Op:    "replace",
    Path:  "/data/acme",
    Value: "5",
  }})
  _, err := clientset.CoreV1().ConfigMaps("dev").Patch(context.Background(), "demo", types.JSONPatchType, payloadBytes, metav1.PatchOptions{})
  if err != nil {
    panic(err)
  }
  fmt.Println("PATCHED")
}

So we may create config map at start, save server keys to it, and whenever peers are added or removed just update config map

Also we may add any authorization we want (e.g. oidc to Azure Active Directory or GitHub with 2FA)

And to revoke keys on schedule we are going to remove peers:

  • that did not connected last X days
  • that did not login to UI last Y days
  • if possible grab actual users list, lets say from Active Directory and remove peers for deactivated users

With that it place it will be the same as password rotation

And there is even more, instead of building such an app, we always may install fastcgi and just run simple scripts instead and to not deal with auth we may just run oauth2-proxy sidecar which means we may build everything for pieces

So our nginx is protected with oauth2-proxy which is configured whatever you want (e.g. AAD+2Fa)

The main page points to script which will create new peer keys, add it to wireguard and print your config

So the workflow will be like - you as a user go to https://wg.contoso.com/ it asks you to login, after that prints your config

Also we may put as many technicall scripts as we want, e.g. /wg.sh, /wg/show/wg0/dump.sh, etc

We may protect them with help of nginx location directives if needed

To not bother with storage we may restart deployment every week

In other words there are two options:

  • build some web app that will handle everything (auth, wireguard) in your favorite language
  • utilize fastcgi and oauth2-proxy and build solution on top of them

WireGuard custom UI

So here we go with UI build with bare shell scripts

Here is dockerfile with wireguard tools and some wanted utilities

FROM nginx:alpine

RUN apk add jq fcgiwrap spawn-fcgi iptables wireguard-tools nmap netcat-openbsd

HEALTHCHECK --interval=5m --timeout=3s --start-period=5s CMD nc -vz localhost 51820

COPY ./01-wireguard-init.sh /docker-entrypoint.d/01-wireguard-init.sh
COPY ./02-spawn-fcgi.sh /docker-entrypoint.d/02-spawn-fcgi.sh
COPY ./default.conf /etc/nginx/conf.d/default.conf
COPY ./html/ /usr/share/nginx/html/

ENV CIDR=10.13.14.0/24
ENV DNS=10.0.0.10
ENV ENDPOINT=TODO:51820
ENV ALLOWEDIPS="10.0.0.0/8, 34.117.59.81/32, 20.13.179.68/32"
ENV CONF_FILE_NAME=dev

EXPOSE 80 51820

Here are script for a home page:

#!/bin/sh

echo 'content-type: text/html'
echo 'cache-control: no-store'
echo ''
echo '<h3>WireGuard</h3>'
echo '<form method="get" action="/add" target="_blank" onsubmit="setTimeout(() => location.reload(), 500)"><input type="submit" value="add" /></form>'
if [[ $(wg show wg0 peers | wc -l) -ne 0 ]]
then
echo '<table border="1" cellspacing="0" cellpadding="5">'
echo '<thead>'
echo '<tr>'
echo '<th>public key</th>'
echo '<th>preshared key</th>'
echo '<th>endpoint</th>'
echo '<th>allowed ip</th>'
echo '<th>latest handshake</th>'
echo '<th>transfer rx</th>'
echo '<th>transfer tx</th>'
echo '<th>persistent keepalive</th>'
echo '<th></th>'
echo '</tr>'
echo '</thead>'
echo '<tbody>'
wg show wg0 dump | sed 1d | awk '{ print "<tr><td>"$1"</td><td>"$2"</td><td>"$3"</td><td>"$4"</td><td>"$5"</td><td>"$6"</td><td>"$7"</td><td>"$8"</td><td><form style=\"margin:0\" action=/remove/"$1"><input type=submit value=&times; /></form></td></tr>" }'
echo '</tbody>'
echo '</table>'
fi

And here is script to add peer:

#!/bin/sh

# echo 'content-type: text/plain'
# echo 'status: 302'
# echo 'location: /'
echo "content-disposition: attachment; filename=\"$CONF_FILE_NAME.conf\""
echo ''

PEER_PRIVATE_KEY=$(wg genkey)
PEER_PUBLIC_KEY=$(echo $PEER_PRIVATE_KEY | wg pubkey)
SERVER_PUBLIC_KEY=$(wg show wg0 public-key)

ADDR=$(comm -13 <(
  wg show wg0 allowed-ips | awk '{ split($2, a, "/"); print a[1] }'
) <(
  nmap -sL -n $CIDR | awk '/Nmap scan report/{print $NF}' | tail -n +3 | sed \$d
) | head -n 1)

wg set wg0 peer $PEER_PUBLIC_KEY allowed-ips "$ADDR/32"

cat << EOF
[Interface]
Address = $ADDR
PrivateKey = $PEER_PRIVATE_KEY
DNS = $DNS

[Peer]
PublicKey = $SERVER_PUBLIC_KEY
Endpoint = $ENDPOINT
AllowedIPs = $ALLOWEDIPS
PersistentKeepalive = 25
EOF

And remove peer:

#!/bin/sh

wg set wg0 peer $PEER remove

echo 'status: 302'
echo 'location: /'
echo ''

Nginx configuration:

server {
  listen       80;
  root   /usr/share/nginx/html;

  location ~ ^/$ {
    index  index;
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
    fastcgi_param SCRIPT_FILENAME /usr/share/nginx/html/index.sh;
  }

  location /add {
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
    fastcgi_param SCRIPT_FILENAME  /usr/share/nginx/html/add.sh;
  }
  location ~ ^/remove/(.+)$ {
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
    fastcgi_param PEER $1;
    fastcgi_param SCRIPT_FILENAME  /usr/share/nginx/html/remove.sh;
  }

  location /interfaces {
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
    fastcgi_param SCRIPT_FILENAME  /usr/share/nginx/html/interfaces.sh;
  }

  location /dump {
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
    fastcgi_param SCRIPT_FILENAME  /usr/share/nginx/html/dump.sh;
  }
}

And init scripts:

#!/bin/sh

# ip
SERVER_IP=$(nmap -sL -n $CIDR | awk '/Nmap scan report/{print $NF}' | head -n 2 | tail -n 1)


# interface
ip link add wg0 type wireguard
ip -4 address add $SERVER_IP dev wg0
ip link set mtu 1420 up dev wg0
ip -4 route add $CIDR dev wg0

# nat
iptables -A FORWARD -i wg0 -j ACCEPT
iptables -A FORWARD -o wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# cfg
wg genkey | tee /etc/wireguard/server.privatekey | wg pubkey > /etc/wireguard/server.publickey
wg set wg0 listen-port 51820 private-key /etc/wireguard/server.privatekey

#!/bin/sh

/usr/bin/spawn-fcgi -s /var/run/fcgiwrap.socket -M 766 /usr/bin/fcgiwrap

And finaly kubernetes deployment demo

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  labels:
    app: demo
  annotations:
    kubeforce: skip
spec:
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      containers:
      - name: demo
        image: mac2000/wg
        imagePullPolicy: Always
        env:
        - name: CIDR
          value: 10.13.14.0/24
        - name: DNS
          value: 10.0.0.10
        - name: ENDPOINT
          value: 20.54.8.225:51820
        - name: ALLOWEDIPS
          value: "10.0.0.0/8, 34.117.59.81/32, 20.13.179.68/32"
        - name: CONF_FILE_NAME
          value: demo
        ports:
        - name: http
          containerPort: 80
          protocol: TCP
        - name: wireguard
          containerPort: 51820
          protocol: UDP
        resources:
          limits:
            cpu: 500m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 256Mi
        securityContext:
          # privileged: true
          capabilities:
            add:
            - NET_ADMIN
      - name: auth
        image: quay.io/oauth2-proxy/oauth2-proxy
        args:
        - --cookie-secure=false

        - --provider=github
        - --email-domain=*
        - --upstream=http://localhost:80/
        - --http-address=0.0.0.0:4180
        - --client-id=***********
        - --client-secret=*********
        - --redirect-url=http://20.54.8.225/oauth2/callback
        - --scope=user:email

        - --cookie-secret=acSPNyjKoCtVBNYS9IZjBuCu7FNF95BmsVzedq0Drr8=
        ports:
        - name: auth
          containerPort: 4180
          protocol: TCP
        resources:
          requests:
            cpu: 10m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 300Mi
---
apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  type: LoadBalancer
  selector:
    app: demo
  ports:
    - name: http
      port: 80
      targetPort: 4180
      protocol: TCP
    - name: wireguard
      port: 51820
      protocol: UDP

Just for demo purposes aplied github as an auth provider

screen1

screen2

screen3

And here you go, WireGuard with UI in Kubernetes with external Auth provider that requires 2FA

And because it is stateles we may just restart deployment lets say once per week to enforce users to rotate keys