Octopus Deploy self-hosted worker in Kubernetes

Becasue Octopus Deploy workers have no static IP addresses we are enforced to host our own workers, which has some benefits (worker from inside Kubernetes sees all services and may be used to run some integration stuff)

Creating workers on a Kubernetes cluster describes over all process of creating workers in Kuberntes

Docker image

Original image has no additional tools

So we are building our own

FROM octopusdeploy/tentacle:6.2.218

# Set to Y to indicate that you accept the EULA
# The port on the Octopus Server that the Tentacle will poll for work. Defaults to 10943. Implies a polling Tentacle
ENV ServerPort="10943"
# The Url of the Octopus Server the Tentacle should register with
ENV ServerUrl="https://contoso.octopus.app"
# The name of the space which the Tentacle will be added to. Defaults to the default space
ENV Space="RUA"

# utils
RUN apt-get update \
    && apt-get install -y \
        curl \
        wget \
        ca-certificates \
        jq \
        sudo \
    && echo done

# powershell - https://learn.microsoft.com/en-us/powershell/scripting/install/install-debian?view=powershell-7.2#installation-on-debian-10-via-package-repository
RUN wget https://packages.microsoft.com/config/debian/10/packages-microsoft-prod.deb \
    && sudo dpkg -i packages-microsoft-prod.deb \
    && sudo apt-get update \
    && sudo apt-get install -y powershell \
    && rm packages-microsoft-prod.deb

# kubectl - https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management
RUN curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg \
    && echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list \
    && apt-get update \
    && apt-get install -y kubectl

# helm - https://helm.sh/docs/intro/install/#from-apt-debianubuntu
RUN curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null \
    && apt-get install apt-transport-https --yes \
    && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | tee /etc/apt/sources.list.d/helm-stable-debian.list \
    && apt-get update \
    && apt-get install helm


Because we preventing access to kubernetes api from outside we have "chicken vs egg" problem, aka we deploy everything via Octopus Deploy, but without worker being deployed we can not deploy anything, thats why worker itself should be deployed somehow outside of Octopus

Note about stateful set

Because removed pods are not removed from Octopus Deploy automatically it will become mess in future

That's why we are deploying stateful set instead of deployment so naming will be always the same

Note that there are examples with pre-stop lifecycle hooks which also may be used to solve this, but they not guarantied to run and add unwanted complexity

Here is an example of manifest used:

apiVersion: apps/v1
kind: StatefulSet
  name: octoworker
  namespace: production
    app: octoworker
  serviceName: octoworker
  replicas: 1
  revisionHistoryLimit: 1
    type: RollingUpdate
      app: octoworker
        app: octoworker
        kubernetes.io/os: linux
      - name: octoworker
        image: gcr.io/majestic-cairn-171208/octoworker:latest
        imagePullPolicy: IfNotPresent
        - name: ServerApiKey
          value: "API-XXXXXXXXXXX"
        - name: TargetWorkerPool
          value: "aks"
            cpu: 50m
            memory: 256Mi
            cpu: 500m
            memory: 2048Mi
          privileged: true

Octopus Deploy

From Octopus Deploy side we should switch from "Dynamic worker pool" to created one

In case if there are multiple environments and cluster it may be a good idea to create dedicated common variable, which may hold worker pool name dependant on environment


So now everything is almost done, but there is an problem, you may have 100500 projects in Octopus Deploy which needs to be changed

Here is an sample script that may be used as an starting point to do so:

$headers = @{'X-Octopus-ApiKey' = $env:OCTOPUS_CLI_API_KEY }
$projects = Invoke-RestMethod -Uri "https://contoso.octopus.app/api/projects?skip=0&take=1000" -Headers $headers | Select-Object -ExpandProperty Items
Write-Host "Got $($projects.Count) projects"

foreach ($project in $projects) {
  # $project = $projects |? Name -eq 'my-awesome-app'
  Write-Host '----------------------------------'
  Write-Host $project.Name
  if ($project.IsDisabled) {
    Write-Host "skipping disabled project..." -ForegroundColor Yellow
  $process = Invoke-RestMethod -Uri "https://contoso.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers
  $changed = $false
  foreach ($step in $process.Steps) {
    # $process.Steps | Select-Object name
    # $step = $process.Steps[0]
    if (-not ($step.Actions | Where-Object IsDisabled -EQ $false)) {
      Write-Host "skipping '$($step.name)' actions disabled..." -ForegroundColor Yellow
    if ($step.Properties.'Octopus.Action.TargetRoles' -ne 'kube-azure') {
      Write-Host "skipping '$($step.Name)' has '$($step.Properties.'Octopus.Action.TargetRoles')' taget role instead of expected 'kube-azure'..." -ForegroundColor Yellow
    foreach ($action in $step.Actions) {
      # $step.Actions | Select-Object name
      # $action = $step.Actions[0]
      if ($action.IsDisabled) {
        Write-Host "skipping disabled action..." -ForegroundColor Yellow
      $actionTypes = @(
      if ($action.ActionType -notin $actionTypes) {
        Write-Host "skipping '$($action.ActionType)' non kubernetes action..." -ForegroundColor Yellow
      # raw yaml deployments has no such property but in our case it does not matter
      # if ($action.Properties.'Octopus.Action.KubernetesContainers.DeploymentResourceType' -notin @('Deployment', 'StatefulSet')) {
      #   Write-Host "skipping '$($action.Properties.'Octopus.Action.KubernetesContainers.DeploymentResourceType')' non deployment action..." -ForegroundColor Yellow
      #   continue
      # }
      if ($action.WorkerPoolId) {
        Write-Host "skipping '$($action.WorkerPoolId)' action has non default worker pool..." -ForegroundColor Yellow

      $action.WorkerPoolVariable = 'octoworker' - will switch to our self hosted workers
      $action.WorkerPoolVariable = $null - will switch to default octopus workers

      if ($action.WorkerPoolVariable -eq 'octoworker') {
        Write-Host "skipping already has octoworker variable..." -ForegroundColor Yellow
      $action.WorkerPoolVariable = 'octoworker'

      # if ($action.WorkerPoolVariable -ne 'octoworker') {
      #   Write-Host "skipping already has octoworker variable..." -ForegroundColor Yellow
      #   continue
      # }
      # $action.WorkerPoolVariable = $null

      $changed = $true

  if ($changed) {
    try {
      Invoke-RestMethod -Method Put "https://contoso.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers -Body ($process | ConvertTo-Json -Depth 100) | Out-Null
      Write-Host "$($project.Name) - success" -ForegroundColor Green
    catch {
      Write-Host "$($project.Name) - failed" -ForegroundColor Red

Followup steps

  • tune resource requests and limits
  • check if we can have probes, at least liveness
  • consider dedicated node pool
  • rover subgraph checks may now access all containers in environment
  • consider priviledged service account so can run some scripts