Sync Azure KeyVault secrets into Kubernetes

If, for some reasone, you want some secrets to be stored in KeyValut and side by side you want them in Kubernetes you have few options

Secrets CSI - powerfull but is like a portal to the hell
External Secrets Operator - deploys quadrillion of CRD, deplyoemtns and so on aka overcomplicated (but still chosen)

Here, I'm noting for my self how similar thing may be achieved without relying on 3rd party dependencies

Here is wanted pieces described via Terraform

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "4.8.0"
    }
  }
}

provider "azurerm" {
  tenant_id       = "00000000-0000-0000-0000-000000000000"
  subscription_id = "00000000-0000-0000-0000-000000000000"
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "example"
  location = "northeurope"
}

resource "azurerm_key_vault" "example" {
  name                      = "example"
  resource_group_name       = azurerm_resource_group.example.name
  location                  = azurerm_resource_group.example.location
  tenant_id                 = "00000000-0000-0000-0000-000000000000"
  sku_name                  = "standard"
  enable_rbac_authorization = true
}

# # let's pretend we will put here secret for other resources
# resource "azurerm_key_vault_secret" "example" {
#   name         = "example"
#   value        = "helloworld"
#   key_vault_id = azurerm_key_vault.example.id
# }

# let's create service account, which we will use inside kubernetes to access keyvault
resource "azurerm_user_assigned_identity" "example" {
  name                = "example"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

data "azurerm_kubernetes_cluster" "kube" {
  name                = "my-awesome-kubernetes-cluster"
  resource_group_name = "example"
}

# federated credentials will allow azure-workload-identity to impersonate service account
resource "azurerm_federated_identity_credential" "example" {
  name                = "example"
  resource_group_name = azurerm_resource_group.example.name
  issuer              = data.azurerm_kubernetes_cluster.kube.oidc_issuer_url
  parent_id           = azurerm_user_assigned_identity.example.id
  audience            = ["api://AzureADTokenExchange"]
  subject             = "system:serviceaccount:default:example"
}

# this one is needed if we want reactive updates
resource "azurerm_eventgrid_system_topic" "example" {
  name                   = "example"
  resource_group_name    = azurerm_resource_group.example.name
  location               = azurerm_resource_group.example.location
  source_arm_resource_id = azurerm_key_vault.example.id
  topic_type             = "Microsoft.KeyVault.vaults"
}

# allow access for our service account to keyvault
resource "azurerm_role_assignment" "example-managed-identity" {
  principal_id         = azurerm_user_assigned_identity.example.principal_id
  role_definition_name = "Key Vault Administrator"
  scope                = azurerm_key_vault.example.id
}

# allow access for ourselves
resource "azurerm_role_assignment" "example-alexandrm" {
  principal_id         = "00000000-0000-0000-0000-000000000000"
  role_definition_name = "Key Vault Administrator"
  scope                = azurerm_key_vault.example.id
}

# # just as an example, instead, here we will add endpoint that will create kubernetes job
# resource "azurerm_eventgrid_system_topic_event_subscription" "example" {
#   name                = "example"
#   system_topic        = azurerm_eventgrid_system_topic.example.name
#   resource_group_name = azurerm_resource_group.example.name

#   webhook_endpoint {
#     url = "https://webhook.site/1b1b1b1b-1b1b-1b1b-1b1b-1b1b1b1b1b1b"
#   }
# }

Nothing special here, we are creating key vault, and service account and adjusting few related settings

From a Kubernetes side we gonna need:

apiVersion: kustomize.config.k8s.io/v1beta1
resources:
  - serviceaccount.yml
  - clusterrolebinding.yml
  - cronjob.yml
configMapGenerator:
  - name: example
    files:
      - sync.ps1
generatorOptions:
  disableNameSuffixHash: true

I am ommiting service account and cluster role binding - nothing interesting there (but do not forget about azure.workload.identity/use label)

Cron job is as simple as

apiVersion: batch/v1
kind: CronJob
metadata:
  name: example
spec:
  suspend: true
  schedule: "0 0 * * *"
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            azure.workload.identity/use: "true"
        spec:
          serviceAccountName: example
          restartPolicy: Never
          containers:
            - name: example
              image: mcr.microsoft.com/powershell
              command:
                - /bin/pwsh
                - /example/sync.ps1
              volumeMounts:
                - name: example
                  mountPath: /example
          volumes:
            - name: example
              configMap:
                name: example

As you may guess, nothing special here as well, once a while we are running sync script

$secretName = 'example'
$namespace = (Get-Content /var/run/secrets/kubernetes.io/serviceaccount/namespace)
$body = @{
  apiVersion = 'v1'
  kind       = 'Secret'
  metadata   = @{
    name      = $secretName
    namespace = $namespace
  }
  data       = @{
    # foo = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes('bar'))
  }
}

$secret = Invoke-RestMethod "https://$($env:KUBERNETES_SERVICE_HOST)/api/v1/namespaces/$namespace/secrets/$secretName" -Headers @{ authorization = "Bearer $(Get-Content /var/run/secrets/kubernetes.io/serviceaccount/token)" } -SkipCertificateCheck -SkipHttpErrorCheck -StatusCodeVariable status
if ($status -ne 200) {
  $secret = Invoke-RestMethod -Method Post "https://$($env:KUBERNETES_SERVICE_HOST)/api/v1/namespaces/$namespace/secrets" -Headers @{ authorization = "Bearer $(Get-Content /var/run/secrets/kubernetes.io/serviceaccount/token)" } -SkipCertificateCheck -ContentType "application/json" -Body (ConvertTo-Json -Depth 100 -InputObject $body)
}
else {
  $secret.data.PSObject.Properties | ForEach-Object {
    $body.data[$_.Name] = $_.Value
  }
}

$changed = $false
$token = $(Invoke-RestMethod -Method Post "https://login.microsoftonline.com/$($env:AZURE_TENANT_ID)/oauth2/v2.0/token" -ContentType "application/x-www-form-urlencoded" -Body "scope=https%3A%2F%2Fmactemp3.vault.azure.net%2F.default&client_id=$($env:AZURE_CLIENT_ID)&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer&client_assertion=$(Get-Content $env:AZURE_FEDERATED_TOKEN_FILE)&grant_type=client_credentials" | Select-Object -ExpandProperty access_token)
$urls = Invoke-RestMethod "https://example.vault.azure.net/secrets?api-version=7.4" -Headers @{ authorization = "Bearer $token" } | Select-Object -ExpandProperty value | Select-Object -ExpandProperty id
foreach ($url in $urls) {
  $name = $url.Split("/")[-1]
  $value = Invoke-RestMethod "$($url)?api-version=7.4" -Headers @{ authorization = "Bearer $token" } | Select-Object -ExpandProperty value
  $encoded = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($value))
  if ($encoded -ne $body.data[$name]) {
    $body.data.$name = $encoded
    $changed = $true
  }
}

if ($changed) {
  Invoke-RestMethod -Method Put "https://$($env:KUBERNETES_SERVICE_HOST)/api/v1/namespaces/$namespace/secrets/$secretName" -Headers @{ authorization = "Bearer $(Get-Content /var/run/secrets/kubernetes.io/serviceaccount/token)" } -SkipCertificateCheck -ContentType "application/json" -Body (ConvertTo-Json -Depth 100 -InputObject $body)
}
else {
  Write-Host "No changes"
}

here we are exchanging kubernetes token provided by azure workload identity to access token

retrieving list of secrets from key vault

and updating kubernetes secret if needed

profit

Why I do like this approach - it is 40 LoC of plain script that we may run any time we want, compare this to number of thigs being deployed by default out of the box by