Kubernetes OpenId Connect auth between services

TLDR: For service to service communication inside kubernetes instead of sharing api keys we may use kubernetes auth for passwordless auth between services, e.g. imagine that you have "sms" api running in your cluster, its job - send sms, it is availble only for your other services, and instead of giving it api key we gonna use service accounts so other services may talk to sms api without shared secret

In Azure AKS we have enabled OIDC, as a result can use projected tokens

OIDC issuer discovery for Kubernetes service accounts was my starting point in this journey

Before doing anything else I want to see what we have out of the box

kubectl run my-shell --rm -it --image ubuntu -- bash
cat /var/run/secrets/kubernetes.io/serviceaccount/token
{
  "aud": [
    "https://westeurope.oic.prod-aks.azure.com/695e64b5/0152ceb3/",
    "https://myaks.hcp.westeurope.azmk8s.io",
    "\"myaks.hcp.westeurope.azmk8s.io\""
  ],
  "exp": 1723699192,
  "iat": 1692163192,
  "iss": "https://westeurope.oic.prod-aks.azure.com/695e64b5/0152ceb3/",
  "kubernetes.io": {
    "namespace": "mynamespace",
    "pod": {
      "name": "my-shell",
      "uid": "fb79574f"
    },
    "serviceaccount": {
      "name": "default",
      "uid": "43c70ddf"
    },
    "warnafter": 1692166799
  },
  "nbf": 1692163192,
  "sub": "system:serviceaccount:mynamespace:default"
}

Notes:

  • it is an JWT token
  • it has issuer that is publicly available and has /.well-known/openid-configuration
  • subject is set to default service account
  • expiration is 1 year

So techincally speaking we already may use it in case if we do not care about subjects and audiences

We can change subject by running our application under dedicated service account like so

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mactemp
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mactemp
  labels:
    app: mactemp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mactemp
  template:
    metadata:
      labels:
        app: mactemp
    spec:
      serviceAccountName: mactemp
      containers:
        - name: mactemp
          image: nginx

The only difference from previous example is system:serviceaccount:mynamespace:mactemp being set as subject

Projected Service Account Token

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mactemp
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mactemp
  labels:
    app: mactemp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mactemp
  template:
    metadata:
      labels:
        app: mactemp
    spec:
      serviceAccountName: mactemp
      containers:
        - name: mactemp
          image: nginx
          volumeMounts:
            - name: oidc-token
              mountPath: /var/run/secrets/tokens
      volumes:
        - name: oidc-token
          projected:
            sources:
              - serviceAccountToken:
                  path: oidc-token
                  expirationSeconds: 600
                  audience: hello

Now our token lives here /var/run/secrets/tokens/oidc-token and has following payload:

{
  "aud": [
    "hello"
  ],
  "exp": 1692164244,
  "iat": 1692163644,
  "iss": "https://westeurope.oic.prod-aks.azure.com/695e64b5/0152ceb3/",
  "kubernetes.io": {
    "namespace": "mynamespace",
    "pod": {
      "name": "mactemp-5985c9d795-fvglk",
      "uid": "0178de50"
    },
    "serviceaccount": {
      "name": "mactemp",
      "uid": "690dfae9"
    }
  },
  "nbf": 1692163644,
  "sub": "system:serviceaccount:mynamespace:mactemp"
}

Notes:

  • now we can customize audience aud
  • subject sub is set to service account
  • expiration exp is 10 minutes
  • token is replaced automatically, so after 10 minutes file contents is being updated

With that in place our application may use it to authenticate requests against other apps, and other apps will look something like this (dotnet, partial sample)

using System.Security.Claims;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.Identity.Web;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddAuthorization().AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(options => {
        options.Authority = "https://westeurope.oic.prod-aks.azure.com/695e64b5/0152ceb3/";
        options.Audience = "hello";
    });

var app = builder.Build();

app.UseAuthentication();
app.UseAuthorization();

app.MapGet("/", (ClaimsPrincipal user) => user.Claims.Select(c => new KeyValuePair<string, string>(c.Type, c.Value)).ToList());

app.Run();

And suddenly we have our authentication setup and applications can talk to each other without the need of shared api keys - aka passwordless approach

Also, for AKS this OIDC endpoint is publicly available so can be consumed by external apps as well, which means we can for example authenticate against CloudElastic, Confluent Kafka, etc without passwords

Also there is this kubernetes.io custom clain which will contain namespace and pod names, that theoretically can be used for granular restrictions (aka with this in place we may not bother with custom service accounts at all and leave default aud)

The last caveat is local development, when we are using Azure Identitity it is as easy as calling DefaultCredentials, but what should we do here?

So far I can not find any way to call kube api directly to sign such token and it seems that workaround will be to call kubectl exec mypod cat path/to/token, at least it will definitely work

I do like this approach little bit more than Azure AD because there is no need to create quadrillion managed service accounts for each and every application

kind

just to verify if it will work in non AKS cluster

kind create cluster
kubectl wait --for=condition=ready node --all --timeout=90s
kubectl apply -f projected.yml
kubectl wait --for=condition=ready pods -l app=mactemp --timeout=90s
kubectl exec mactemp-db7686c79-9mwth -- cat /var/run/secrets/tokens/oidc-token
{
  "aud": [
    "hello"
  ],
  "exp": 1692166019,
  "iat": 1692165419,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "mactemp-db7686c79-9mwth",
      "uid": "420f5867"
    },
    "serviceaccount": {
      "name": "mactemp",
      "uid": "08e39557"
    }
  },
  "nbf": 1692165419,
  "sub": "system:serviceaccount:default:mactemp"
}

Note: issuer is now internal and will have self signed certificate

Kube API

kubectl proxy
curl http://localhost:8001/api/v1/namespaces/production/serviceaccounts/mactemp

/api/v1/namespaces//serviceaccounts/: The kubelet queries this endpoint to retrieve the service account details, including the secret name containing the token. /api/v1/namespaces//secrets/: Once the secret name is obtained, the kubelet queries this endpoint to fetch the secret data, which inclu

Another way more detailed article is: Authentication between microservices using Kubernetes identities

And second one Manually create a service account API token

# manually create and assign secret for given service account
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: mactemp-secret
  annotations:
    kubernetes.io/service-account.name: mactemp
type: kubernetes.io/service-account-token
EOF

# retrieve token
kubectl get secret mactemp-secret -o jsonpath='{.data.token}' | base64 -d
# note: after service account deletion this secret will be deleted automatically

and in both case (default or projected) payload is:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "mynamespace",
  "kubernetes.io/serviceaccount/secret.name": "mactemp-secret",
  "kubernetes.io/serviceaccount/service-account.name": "mactemp",
  "kubernetes.io/serviceaccount/service-account.uid": "d3372b60",
  "sub": "system:serviceaccount:mynamespace:mactemp"
}

Verifying token

# note: -o yaml used to print response in yaml
kubectl apply -o yaml -f - <<EOF
kind: TokenReview
apiVersion: authentication.k8s.io/v1
metadata:
  name: test
spec:
  token: $(kubectl get secret mactemp-secret -o jsonpath='{.data.token}' | base64 -d)
EOF

it will return big yaml where we are looking for authenticated: true

And the same for projected token

kubectl apply -o yaml -f - <<EOF
kind: TokenReview
apiVersion: authentication.k8s.io/v1
metadata:
  name: test
spec:
  token: $(kubectl exec $(kubectl get po -l app=mactemp -o custom-columns=:metadata.name --no-headers) -- cat /var/run/secrets/tokens/oidc-token)
EOF

Gives us error: invalid bearer token, token audiences "hello" is invalid for the target audiences "https://westeurope.oic.prod-aks.azure.com/695e64b5", "myaks.hcp.westeurope.azmk8s.io" the server has asked for the client to provide credentials

To fix that we need path audiences like so

kubectl apply -o yaml -f - <<EOF
kind: TokenReview
apiVersion: authentication.k8s.io/v1
metadata:
  name: test
spec:
  audiences:
    - hello
  token: $(kubectl exec $(kubectl get po -l app=mactemp -o custom-columns=:metadata.name --no-headers) -- cat /var/run/secrets/tokens/oidc-token)
EOF

With that in place we can verify token without the need to know anything about oidc and jwks

The caveat here is that: we can not modify audience for default service account approach and we can not retrieve token for projected tokens without running pod first

Also note that manually created secret does not have expiration at all so it will live forever

And finally from How to create ServiceAccount Secret in Kubernetes 1.24 found wanted command:

kubectl create token mactemp

which returns

{
  "aud": [
    "https://westeurope.oic.prod-aks.azure.com/695e64b5/0152ceb3/",
    "https://myaks.hcp.westeurope.azmk8s.io",
    "\"myaks.hcp.westeurope.azmk8s.io\""
  ],
  "exp": 1692173481,
  "iat": 1692169881,
  "iss": "https://westeurope.oic.prod-aks.azure.com/695e64b5/0152ceb3/",
  "kubernetes.io": {
    "namespace": "mynamespace",
    "serviceaccount": {
      "name": "mactemp",
      "uid": "5842a7ce"
    }
  },
  "nbf": 1692169881,
  "sub": "system:serviceaccount:mynamespace:mactemp"
}

Notes:

  • token expires in 1 hour

And thankfully it has some options

kubectl create token mactemp --duration 10m --audience hello

To see whats going on under the hood pass -v=9

And it seems that request will be something like this (note: we are using proxy here to not bother with auth so far)

kubectl proxy
curl -s -X POST -H "Content-Type: application/json" "http://localhost:8001/api/v1/namespaces/production/serviceaccounts/mactemp/token" -d '{
  "kind":"TokenRequest",
  "apiVersion":"authentication.k8s.io/v1",
  "spec":{
    "audiences":["hello"],
    "expirationSeconds":600
  }
}'

and response will be

{
  "kind": "TokenRequest",
  "apiVersion": "authentication.k8s.io/v1",
  "metadata": {
    "name": "mactemp",
    "namespace": "production",
    "creationTimestamp": "2023-08-16T07:19:22Z",
    "managedFields": [
      {
        "manager": "curl",
        "operation": "Update",
        "apiVersion": "authentication.k8s.io/v1",
        "time": "2023-08-16T07:19:22Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:spec": {
            "f:audiences": {},
            "f:expirationSeconds": {}
          }
        },
        "subresource": "token"
      }
    ]
  },
  "spec": {
    "audiences": [
      "hello"
    ],
    "expirationSeconds": 600,
    "boundObjectRef": null
  },
  "status": {
    "token": "eyJhbGciOiJSUzI....",
    "expirationTimestamp": "2023-08-16T07:29:22Z"
  }
}

Which means we should be able to do it programmatically if that is not done yet already in client libraries

Starting from one of the articles going to give it a try in golang first

package main

import (
	"context"
	"fmt"

	authv1 "k8s.io/api/authentication/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/tools/clientcmd"
)

/*
Taken from "out of the cluster" example
https://github.com/kubernetes/client-go/blob/master/examples/out-of-cluster-client-configuration/main.go
*/
func main() {
	// in my case each cluster has its own config file
	config, err := clientcmd.BuildConfigFromFlags("", "/Users/mac/Documents/dotfiles/kube/prod.yml")
	if err != nil {
		panic(err)
	}

	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		panic(err.Error())
	}

	// namespaces, err := clientset.CoreV1().Namespaces().List(context.Background(), metav1.ListOptions{})
	// if err != nil {
	// 	panic(err)
	// }
	// for _, ns := range namespaces.Items {
	// 	fmt.Printf("Namespace: %s\n", ns.Name)
	// }

	expirationSeconds := int64(600)
	r, err := clientset.CoreV1().ServiceAccounts("production").CreateToken(context.Background(), "mactemp", &authv1.TokenRequest{
		Spec: authv1.TokenRequestSpec{
			Audiences:         []string{"hello"},
			ExpirationSeconds: &expirationSeconds,
		},
	}, metav1.CreateOptions{})
	if err != nil {
		panic(err)
	}
	fmt.Println(r.Status.Token)

}

Which techinically doing the same as we did with curl and kubectl create token

Now it is dotnet turn

using k8s;

/*
https://github.com/kubernetes-client/csharp

dotnet add package KubernetesClient
*/

var config = KubernetesClientConfiguration.IsInCluster()
  ? KubernetesClientConfiguration.InClusterConfig()
  : KubernetesClientConfiguration.BuildConfigFromConfigFile("/Users/mac/Documents/dotfiles/kube/prod.yml"); // leave empty to use default config
var client = new Kubernetes(config);

// var namespaces = client.CoreV1.ListNamespace();
// foreach (var ns in namespaces.Items) {
//     Console.WriteLine(ns.Metadata.Name);
// }

var response = await client.CoreV1.CreateNamespacedServiceAccountTokenAsync(new k8s.Models.Authenticationv1TokenRequest {
  Spec = new k8s.Models.V1TokenRequestSpec {
    Audiences = new [] { "hello" },
    ExpirationSeconds = 600
  }
}, "mactemp", "production");

Console.WriteLine(response.Status.Token);

So now the goal is to have something similar to Azure Identity where we have single Default Credentials method that makes everything automatically under the hood, iterating over all possible options and choosing first one, in our case there are two options - we are either inside kubernetes or outside

dotnet http client with kubernetes tokens

From this answer on stackoverflow landed to article: Make HTTP requests using IHttpClientFactory in ASP.NET Core

So just to recap how it may look like in dotnet (boilerplate, also I do believe there should be something that already handles all this)

Here is our consumer

using System.Net;
using System.Security.Claims;
using Microsoft.AspNetCore.Authentication.JwtBearer;

var builder = WebApplication.CreateBuilder(args);

// just for demo, listen 0.0.0.0:8000
builder.WebHost.ConfigureKestrel((context, serverOptions) => serverOptions.Listen(IPAddress.Any, 8000, listenOptions => listenOptions.UseConnectionLogging()));

builder.Services.AddAuthorization().AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(options => {
        options.Authority = "https://westeurope.oic.prod-aks.azure.com/695e64b5-2d13-4ea8-bb11-a6fda2d60c41/0152ceb3-63e2-4b13-9240-c0dab87fdd0f/";
        options.Audience = "hello";
    });

var app = builder.Build();

app.UseAuthentication();
app.UseAuthorization();

app.MapGet("/", (ClaimsPrincipal user) => user.Claims.Select(c => new KeyValuePair<string, string>(c.Type, c.Value)).ToList());

app.Run();

we may call it like so:

curl http://localhost:8000/ -H "Authorization: Bearer $(kubectl -n production create token mactemp --duration 10m --audience hello)"

and it will respond:

[
  {
    "key": "aud",
    "value": "hello"
  },
  {
    "key": "exp",
    "value": "1692179705"
  },
  {
    "key": "iat",
    "value": "1692179105"
  },
  {
    "key": "iss",
    "value": "https://westeurope.oic.prod-aks.azure.com/695e64b5-2d13-4ea8-bb11-a6fda2d60c41/0152ceb3-63e2-4b13-9240-c0dab87fdd0f/"
  },
  {
    "key": "kubernetes.io",
    "value": "{\"namespace\":\"production\",\"serviceaccount\":{\"name\":\"mactemp\",\"uid\":\"5842a7ce-efec-413d-8cf9-c7d992287017\"}}"
  },
  {
    "key": "nbf",
    "value": "1692179105"
  },
  {
    "key": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier",
    "value": "system:serviceaccount:production:mactemp"
  }
]

With that in place our client will be something like this (please not it is just an boilerplate demo):

using System.Net.Http.Headers;
using k8s;
using Refit;

var builder = WebApplication.CreateBuilder(args);

// depending on where our app is running we need either cluster or local token provider
builder.Services.AddSingleton<IKubernetesTokenProvider>(_ => KubernetesClientConfiguration.IsInCluster() ? new ClusterTokenProvider("/var/run/secrets/tokens/oidc-token") : new LocalTokenProvider("production", "mactemp", new [] { "hello" }, "/Users/mac/Documents/dotfiles/kube/prod.yml" /* do not pass me to use default kube config */));
builder.Services.AddSingleton<BearerTokenHandler>();

// register our client with handler
builder.Services
  .AddHttpClient("demo", c => c.BaseAddress = new Uri("http://localhost:8000"))
  .AddTypedClient(RestService.For<IDemoClient>)
  .AddHttpMessageHandler<BearerTokenHandler>();

var app = builder.Build();

app.MapGet("/", async (IDemoClient demo) => await demo.GetAsync());

app.Run();

public interface IKubernetesTokenProvider
{
  Task<string> GetTokenAsync(CancellationToken cancellationToken);
}

// will read mounted token file
public class ClusterTokenProvider: IKubernetesTokenProvider
{
  private readonly string _tokenFilePath;

  public ClusterTokenProvider(string tokenFilePath)
  {
    _tokenFilePath = tokenFilePath;
  }

  public async Task<string> GetTokenAsync(CancellationToken cancellationToken)
  {
    return await File.ReadAllTextAsync(_tokenFilePath, cancellationToken);
  }
}

// for local development, will use kubernetes api to get token
public class LocalTokenProvider: IKubernetesTokenProvider
{
  private readonly IKubernetes _kubernetes;
  private readonly string _namespaceName;
  private readonly string _serviceAccountName;
  private readonly IList<string> _audiences;

  public LocalTokenProvider(string namespaceName, string serviceAccountName, IList<string> audiences, string? kubeConfigFilePath = null)
  {
    _namespaceName = namespaceName;
    _serviceAccountName = serviceAccountName;
    _audiences = audiences;
    _kubernetes = new Kubernetes(KubernetesClientConfiguration.BuildConfigFromConfigFile(kubeConfigFilePath));
  }

  public async Task<string> GetTokenAsync(CancellationToken cancellationToken)
  {
    var response = await _kubernetes.CoreV1.CreateNamespacedServiceAccountTokenAsync(new k8s.Models.Authenticationv1TokenRequest {
      Spec = new k8s.Models.V1TokenRequestSpec {
        Audiences = _audiences,
        ExpirationSeconds = 600
      }
    }, _serviceAccountName, _namespaceName, cancellationToken: cancellationToken);
    return response.Status.Token;
  }
}

public class BearerTokenHandler : DelegatingHandler
{
  private readonly IKubernetesTokenProvider _tokenProvider;

  public BearerTokenHandler(IKubernetesTokenProvider tokenProvider)
  {
    _tokenProvider = tokenProvider;
  }

  protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  {
    if (!request.Headers.Contains("Authorization"))
    {
      // TODO: instead doing it each time may and should be cached till expired / or should it be done in token provider instead or should we have cached token provider
      request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", await _tokenProvider.GetTokenAsync(cancellationToken));
    }
    return await base.SendAsync(request, cancellationToken);
  }
}

// our client interface (using refit)
public interface IDemoClient
{
  [Get("/")]
  Task<IList<KeyValuePair<string, string>>> GetAsync();
}

With this in place we are making an call to our client, under the hood it does receive access token and calls our consumer with this token bypassing the response - profit