Kubernetes Ingress behind Cloudflare Access

Suppose we want to cover important services (like Kubernetes dashboard) behind authentication, authorization and make everything encrypted

The easiest way will be to cover it with Cloudflare which will give us everything out of the box

In short here is how it will work:

Things to accomplish:

Communication between user and Cloudflare is encrypted by its certificates.
Communication between Cloudflare and Ingress is always encrypted by Origin Certificate signed and trusted by Cloudflare.
Ingress allows requests only from Cloudflare.
Access to application is possible only after authentication and authorization via configured identity provider.

User to Cloudflare encryption

That is an easy one, all we need to do is to turn on "proxy" checkbox for desired (sub)domain.

Allow requests only from Cloudflare

Thats also easy one, Cloudflare has range of its IP addresses which might be added to a ingress annotation whitelist-source-range

TODO: create simple admission mutation webhook which will listen to ingress creation or updation events and inject list of addresses from config map, also it should watch for config map and if it changes - update all ingresses

Cloudflare to Ingress encryption

Here we have four options under SSL/TLS configurations:

Off - even not disqused
Flexible - its when requests will be user -(https:443)-> cloudflare -(http:80)-> origin - technically, technically, if we are allowing requests only from a Cloudflare it might be ok, but we want encryption everywhere
Full - now requests from cloudflare to origin will also be made over https, but Cloudflare will trust any self signed certificate
Strict - the most interesting for us, it will be encrypted and Cloudflare will validate certificates

If we are talking about certificates for our origin there are two options:

Create certificates with LetsEncrypt (requires cert-manager so 90 days certificates will be updated automaticaly)
Create origin certificate from Cloudflare (will give us certificate for years, trusted by Cloudflare)

We are going to try all options here but at the end stick with Cloudflare certificate

Flexible

Our starting poing, it wont have encryption between Cloudflare and origin but will be used for all further deployments

flexible.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flexible
  labels:
    app: flexible
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flexible
  template:
    metadata:
      labels:
        app: flexible
    spec:
      containers:
        - name: flexible
          image: nginx:alpine
          ports:
            - name: flexible
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: flexible
spec:
  type: ClusterIP
  selector:
    app: flexible
  ports:
    - name: flexible
      protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: flexible
  annotations:
    # ADDED
    nginx.ingress.kubernetes.io/whitelist-source-range: 173.245.48.0/20, 103.21.244.0/22, 103.22.200.0/22, 103.31.4.0/22, 141.101.64.0/18, 108.162.192.0/18, 190.93.240.0/20, 188.114.96.0/20, 197.234.240.0/22, 198.41.128.0/17, 162.158.0.0/15, 104.16.0.0/13, 104.24.0.0/14, 172.64.0.0/13, 131.0.72.0/22, 2400:cb00::/32, 2606:4700::/32, 2803:f800::/32, 2405:b500::/32, 2405:8100::/32, 2a06:98c0::/29, 2c0f:f248::/32

spec:
  rules:
    - host: flexible.marchenko.net.ua
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: flexible
                port:
                  number: 80

Full

For full we are ok to use self signed certificates, so need to create one:

openssl genrsa -out selfsigned.key 2048
openssl req -x509 \
  -new -nodes  \
  -days 10 \
  -key selfsigned.key \
  -out selfsigned.crt \
  -subj "/CN=selfsigned.marchenko.net.ua"

Then with help of commands copy and paste certificate and key to secret

cat selfsigned.crt | base64 | pbcopy
cat selfsigned.key | base64 | pbcopy

And apply following:

selfsigned.yml

---
apiVersion: v1
kind: Secret
metadata:
  name: selfsigned-tls
type: kubernetes.io/tls
data:
  # cat selfsigned.crt | base64
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJBQ0NRQ3RMUzZwaEVLQ1pqQU5CZ2txaGtpRzl3MEJBUXNGQURBbU1TUXdJZ1lEVlFRRERCdHoKWld4bWMybG5ibVZrTG0xaGNtTm9aVzVyYnk1dVpYUXVkV0V3SGhjTk1qRXhNVEl6TVRnd056QTNXaGNOTWpFeApNakF6TVRnd056QTNXakFtTVNRd0lnWURWUVFEREJ0elpXeG1jMmxuYm1Wa0xtMWhjbU5vWlc1cmJ5NXVaWFF1CmRXRXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFEbDA2RXlXK2FRZmxWZWF6R2IKU0JaR09DeThqaC9RS2k4Uy84alhHZ3dUVXVpSlprd080MzdtTGxUVzdVRmkwS2ZxR2YrS21Cak5McE83RzQwTQpZR3FIakhiNERwZjlGSHNDWExsTDVKOUNhbGJheG1ZOUYwQ3pwbjRkNXNVbFhqVDZDY0dmYm92NUJxWDBEWXFiCjlFOWdGZGI2NXJ5eHhSSndKK25XeW14YWI4c2tWTytWbGVsNUdOREx3cGdXRUJDUnRNOFNZY1ZicDYrNXZDQi8KQjRiL1dBalQ3bjYwMHhmdWNHMkc0QTd0RkFTSkpzeVdFdzBoT0tvQVp5WmNDTUxmdWRVb1BYVndpdTVJV1Q3LwpzRG5tUUI0OSt0Yy9sNjFlYjdhZlJSaElIZ0lBZUIrU3ZqQTZIazhNV1R1b28wZ29hVWhwNVpRVnZ0UTlNN21ZCng2RzlBZ01CQUFFd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFKcEVkQTlEUE5Nd3BSRUtBSFZKTCtLL0crTDkKbS9CYS9CcDlQUmNNa011M2VXRkhWTUR6dWQrblNLOWRoY0luV056MjZEZHh6RE5VeDgzVkhCQWNiZnRCTHFVSgpDVEZ5bk9WQzF1cVNoZ3FsVzFCMVdVL1ZiTmZBOWZRS09Kd29CTmtrOWRMdXlYOU55aHJ0TmUrNWpGL3FqM1BxCm1ON2ZORUdlZkViZGVFbVgvY3ZFR1JZL1JWRXpOcVgyQUNoTEdldmg2SnZqTG5FRmVSaElxWmxyT0VZVTV5Q0wKQWRkT0VWYTJhQmxXK015c1hDNnR4bUlzVk5UY3pFUW95em1qdnFrQjl6YnZaeWxKaGhuUDNFc1l2MmRqbnpZSQo3cXhWdURHbDVCOStNcmI1WVlXUXhqYTl4NmlkVzB4Z2pYUloxcUdDT2xDS0FobHhzenBrdEFFbWJTcz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  # cat selfsigned.key | base64
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBNWRPaE1sdm1rSDVWWG1zeG0wZ1dSamdzdkk0ZjBDb3ZFdi9JMXhvTUUxTG9pV1pNCkR1Tis1aTVVMXUxQll0Q242aG4vaXBnWXpTNlR1eHVOREdCcWg0eDIrQTZYL1JSN0FseTVTK1NmUW1wVzJzWm0KUFJkQXM2WitIZWJGSlY0MCtnbkJuMjZMK1FhbDlBMkttL1JQWUJYVyt1YThzY1VTY0NmcDFzcHNXbS9MSkZUdgpsWlhwZVJqUXk4S1lGaEFRa2JUUEVtSEZXNmV2dWJ3Z2Z3ZUcvMWdJMCs1K3ROTVg3bkJ0aHVBTzdSUUVpU2JNCmxoTU5JVGlxQUdjbVhBakMzN25WS0QxMWNJcnVTRmsrLzdBNTVrQWVQZnJYUDVldFhtKzJuMFVZU0I0Q0FIZ2YKa3I0d09oNVBERms3cUtOSUtHbElhZVdVRmI3VVBUTzVtTWVodlFJREFRQUJBb0lCQVFEYUF4N3VmbThBc2xhWQo4bVprUHNXaVpyZEpSaWkzZHN5ZzRUTExtVkhMSXVLak0vOENRSnVvRGxzOFptT1lNUWFYODRHTkh5cW51SDg3CkJxK09CZ2dIL1I3OXdCbzlZS2d2WFhZSW9CN1RDYnpLVzFtd3FkWWVsTmtkMUVNL1oxWXB1Z2VLRFh1S0J0eDEKM2tLV0hmTU4ycW5MNkFVblBTSjRjZVJJZEV6RW0ydGY4NzJWNUtQNVU3dWtLQ3ZKQjdOOGxUaGFVb09lcmhtRAo5YXVsbzV0a3NNOWJWOVZsMk5JaVZVQkk1OWJnQ25EanBTT295c1VraDloWHJNZTU4R0ZvOUVMcUJPYVg4V1FkCllDSWQvN3hqbkxXYnVXTHVLR3FUNXNrYlZpZHlKd3drWlI4TktSejhiSnM2UWNxMmd5SlIzeUp0USszNzNpV3QKdVVBUWg5c0JBb0dCQVBtUWxrbDVPQmdrbXl5U0JNZWt6bjU2TXZtQ2VtblpmS2VKdnpBaEpzTS9CWUtkSjVxUwpuUTNGb1d0emE1US9LSDhPbFdLMVk0UlZDL1VaUkRoRUswYWtVeE5Yc2E2Wm1GV1JmSEhTcndWRERwQVJiZytLCjJsZE1EdzlmVjdSU044ejlCMnVKa3Ixb29td29SMXdoUTF3bDVJRW9INFY4OVFUblRJKzVWSHlwQW9HQkFPdkEKdjUxR2Zjd3pCWndXem1LQnRwazVDa3YyREZQMXFMM2NEWHBaN1lyVEZTS0M2Vml5REN0UjdJd1U3OW9FVXZWawozZUNML3VjTTR4ZUpRc0ZsRlVia1Qwa215cUVpQmxoN0JVSHU5eVh1WFc0dlFXRU91ZC9KNWJqdWJld3R0ZnhOCjYwMktiL1lLVmtSTzJhOVUreFI1NWVqWE5iaHJOZzNnM0daN0NqVDFBb0dBSFliMzRSV1FoM3E5U1ZhMWJoR1gKeE9zY1lOMEpNc0RpdU1mWmNRNWJFYnJCSWlVSjdJWHdReExnK0YrZG1OcW9KZkJKeUhZQm9rU1Z0NWdYdERpTQpqajBlL3lqZkF2TjFUSnA4Q05PT2xhYkxjRXcvbzJNZ1ZqZlBpRmdWZm1aRDUrTUFINzhWTjRmTHY5UThMVEJhClVlckIwRzQ0M0loZVRRZzR6TUIydUpFQ2dZRUEyZmZXOGFISUUzRWxBYUR3WnBGSmN6Y1N0eEtoYVVzYkhaY0QKK3hpRUVMT0N1ZXJSdGtqQUNabHdIRUFMVnNaaXNUSDJGbk01VlFSTVBtbVJCOS9lb1RIYndHV1dPUTZOVVdkRQpnQ1poNlhjT2VSNUYwMmZiSVBhNVo1ZENtSGZXNTVSZk5zMEZJVFdEZHhwMEY3QjVpQUFBT3B3bnRmK2g5WWp1CkhlNHFqTFVDZ1lCMmtFM3lFQlo4N3BjRTA0a0lGSW5UV0JqbnpHbDl5bitIdThoNlZIRFo1Yld4NnBHSDh2emIKeGJaZzBZcTlDTHRKOFdudUo5OU5iK1RGZGIwQWRaeG5QUlZvQlo5TStvM00xeVJVMTdKckozaFBFeVpoMDJZeApLQmMzelJYcUQ5WXhzN0VuRE9TTndJQ0tmdGNoZW5YTVZnSGJHanVIdENkejNUelFwSUcvUXc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: selfsigned
  labels:
    app: selfsigned
spec:
  replicas: 1
  selector:
    matchLabels:
      app: selfsigned
  template:
    metadata:
      labels:
        app: selfsigned
    spec:
      containers:
        - name: selfsigned
          image: nginx:alpine
          ports:
            - name: selfsigned
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: selfsigned
spec:
  type: ClusterIP
  selector:
    app: selfsigned
  ports:
    - name: selfsigned
      protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: selfsigned
  annotations:
    # ADDED
    nginx.ingress.kubernetes.io/whitelist-source-range: 173.245.48.0/20, 103.21.244.0/22, 103.22.200.0/22, 103.31.4.0/22, 141.101.64.0/18, 108.162.192.0/18, 190.93.240.0/20, 188.114.96.0/20, 197.234.240.0/22, 198.41.128.0/17, 162.158.0.0/15, 104.16.0.0/13, 104.24.0.0/14, 172.64.0.0/13, 131.0.72.0/22, 2400:cb00::/32, 2606:4700::/32, 2803:f800::/32, 2405:b500::/32, 2405:8100::/32, 2a06:98c0::/29, 2c0f:f248::/32

spec:
  # ADDED
  tls:
    - hosts:
        - selfsigned.marchenko.net.ua
      secretName: selfsigned-tls

  rules:
    - host: selfsigned.marchenko.net.ua
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: selfsigned
                port:
                  number: 80

If for some reason something does not work check ingress logs with something like:

kubectl -n ingress logs -l name=nginx-ingress-microk8s

Also check what cetificate is for:

openssl x509 -in cloudflare.crt -text -noout

In my case I got bunch of errors complaining local SSL certificate cf/selfsigned-key was not found. Using default certificate because I did removed comments and make certificates in a single line, then because of a wrong secret type.

Now if you will try to connect everything should be ok:

curl -s -i -vv https://selfsigned.marchenko.net.ua

Buf if you will try to bypass cloudflare with help of:

curl -k -s -i --resolve selfsigned.marchenko.net.ua:443:178.20.154.97 https://selfsigned.marchenko.net.ua/

You will receive 403 forbidden

So we already have encryption everywhere plus connections are allowed only from Cloudflare

Strict - Letsencrypt

For strict to work we gonna need real certificates, so here is a demo just in case it will be needed in future

letsencrypt.yml

---
apiVersion: v1
kind: Secret
metadata:
  name: letsencrypt-tls
  # placeholder, will be filled in with certificates by cert-manager
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: letsencrypt
  labels:
    app: letsencrypt
spec:
  replicas: 1
  selector:
    matchLabels:
      app: letsencrypt
  template:
    metadata:
      labels:
        app: letsencrypt
    spec:
      containers:
        - name: letsencrypt
          image: nginx:alpine
          ports:
            - name: letsencrypt
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: letsencrypt
spec:
  type: ClusterIP
  selector:
    app: letsencrypt
  ports:
    - name: letsencrypt
      protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: letsencrypt
  annotations:
    # ADDED
    nginx.ingress.kubernetes.io/whitelist-source-range: 173.245.48.0/20, 103.21.244.0/22, 103.22.200.0/22, 103.31.4.0/22, 141.101.64.0/18, 108.162.192.0/18, 190.93.240.0/20, 188.114.96.0/20, 197.234.240.0/22, 198.41.128.0/17, 162.158.0.0/15, 104.16.0.0/13, 104.24.0.0/14, 172.64.0.0/13, 131.0.72.0/22, 2400:cb00::/32, 2606:4700::/32, 2803:f800::/32, 2405:b500::/32, 2405:8100::/32, 2a06:98c0::/29, 2c0f:f248::/32
    # ask cert-manager to create certificates for us
    cert-manager.io/cluster-issuer: 'letsencrypt'

spec:
  # ADDED
  tls:
    - hosts:
        - letsencrypt.marchenko.net.ua
      secretName: letsencrypt-tls

  rules:
    - host: letsencrypt.marchenko.net.ua
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: letsencrypt
                port:
                  number: 80

The check will be the same as in previous example

Strick - Cloudflare

And now is a cool part, in Cloudflare under SSL/TLS there is an option to create certificates for your origin server which will be trusted by Cloudflare and will be signed for years

After creating certificate Cloudflare will create certificate and key for you, so you can follow exactly same approach as in flexible/selfsigned example

Deployment and checks are the same

Authenticate

Navigate to teams dashboard to add application and its authentication providers

There is no hidden knowledge and everything is done in just few clicks including identity providers

What is cool you can configure different rules per application/group/region/whatever

Whenever you will try open any of your apps, you will be redirected to choosen identity provider for authentication and only after redirected back to an app, and because we are not allowing bypassing Cloudflare its the same as disalowing anonymous access at all.

Also there few other cool features in Cloudflare Acess like Gateways and Tunnels which may make even crazier things