octochecks

In our case we do allow engineers to create and edit projects in Octopus Deploy

But as a result it is hard to maintain some wanted policies (like ensure each project has owner variable or that Kubernetes deployments have resources and so on)

Asked community but it seems that at moment there is no built in way to achieve that

But there is an "workaround" which not fully solve the issue but will allow move forward

What

Here is an idea:

we will have script template with powershell script that will make sure all checks are green
external script that will add this script as very first and required step in every project
subscription pointing to service that will listen for deployments and check if step was disabled, if so - reenable it and post an message to slack

Notes:

yes, engineers can still mark step as not required and deploy, but immediatelly it will be marked as required back and sooner or later everyone will be tired of this and will just fix issues
for external script we may use github action with scheduled workflow or cronjob in Kubernetes

Checks

Lets start simple with checks script, to make things clear I am going to use example that will check that project has owner variable being set, but technically it can do way more checks

Also for script to work we need Octopus API Key variable

Everything else configured with defaults

Here is starting point:

if ($OctopusParameters) {
  $headers = @{"X-Octopus-ApiKey" = $OCTOPUS_APIKEY}
  $projectId = $OctopusParameters["Octopus.Project.Id"]
} else {
  $headers = @{"X-Octopus-ApiKey" = $env:OCTOPUS_APIKEY}
  $projectId = "Projects-1"
}

$project = Invoke-RestMethod "https://robota.octopus.app/api/projects/$projectId" -Headers $headers

Write-Host $project.Name

As you can see at very top we have condition, so we can run and test it locally, and here is more complex example

if ($OctopusParameters) { # we are inside octopus
  $headers = @{"X-Octopus-ApiKey" = $OCTOPUS_APIKEY}
  $projectId = $OctopusParameters["Octopus.Project.Id"]
} else { # local run
  $headers = @{"X-Octopus-ApiKey" = $env:OCTOPUS_APIKEY}
  $projectId = "Projects-1"
}

$project = Invoke-RestMethod "https://example.octopus.app/api/projects/$projectId" -Headers $headers
$failed = 0

# VARIABLES

$variables = Invoke-RestMethod "https://example.octopus.app$($project.Links.Variables)" -Headers $headers | Select-Object -ExpandProperty Variables
$owner = $variables | Where-Object Name -EQ 'Owner'
if (-not $owner) {
  Write-Host "- 'Owner' variable is missing"
  $failed += 1
} else {
  # TODO: check if owner is a valid email
  # TODO: cehck if owner is not deactivated
}

$repository = $variables | Where-Object Name -EQ 'Repository'
if (-not $repository) {
  Write-Host "- 'Repository' variable is missing"
  $failed += 1
}

# CHANNELS

$channels = Invoke-RestMethod "https://example.octopus.app/api/projects/$projectId/channels" -Headers $headers | Select-Object -ExpandProperty Items
$process = Invoke-RestMethod "https://example.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers

foreach($step in $process.Steps) {
  foreach($action in $step.Actions) {
    if ($action.IsDisabled) { continue }
    foreach($package in $action.Packages) {
      $name = $package.Name
      if (-not $name) {
        $name = $package.PackageId
      }
      foreach($channel in $channels) {
        $found = $false
        foreach($rule in $channel.Rules) {
          if ($rule.ActionPackages | Where-Object DeploymentAction -EQ $action.Name | Where-Object { -not $_.PackageReference -or $_.PackageReference -EQ $package.Name }) {
            $found = $true
            break
          }
        }
        if (-not $found) {
          Write-Host "- '$name' has no rules for '$($channel.Name)' channel"
          $failed += 1
        }
      }
    }
  }
}


# KUBERNETES RESOURCES

foreach($step in $process.Steps) {
  foreach($action in $step.Actions) {
    if ($action.IsDisabled) { continue }
    if ($action.ActionType -ne 'Octopus.KubernetesDeployContainers') { continue }
    $containers = $action.Properties.'Octopus.Action.KubernetesContainers.Containers' | ConvertFrom-Json
    foreach($container in $containers) {
      if (-not $container.Resources.limits.cpu) {
        Write-Host "- '$($container.Name)' container has no cpu limits defined"
        $failed += 1
      }
      if (-not $container.Resources.limits.memory) {
        Write-Host "- '$($container.Name)' container has no memory limits defined"
        $failed += 1
      }
      if (-not $container.Resources.requests.cpu) {
        Write-Host "- '$($container.Name)' container has no cpu requests defined"
        $failed += 1
      }
      if (-not $container.Resources.requests.memory) {
        Write-Host "- '$($container.Name)' container has no memory requests defined"
        $failed += 1
      }
    }
  }
}

if ($failed) {
  Write-Host ""
  Write-Host "$failed checks failed"
  exit 1
} else {
  Write-Host "All checks passed"
}

Install

Now we need an script that will add this script to all projects

Here is what I have ended up with:

$headers = @{"X-Octopus-ApiKey" = $env:OCTOPUS_APIKEY}

$templates = Invoke-RestMethod "https://example.octopus.app/api/actiontemplates/all" -Headers $headers
$template  = $templates | Where-Object Name -EQ 'octochecks'

$step = @{
  Name    = $template.Name
  Actions = @(@{
    Name               = $template.Name
    ActionType         = $template.ActionType
    IsRequired         = $true
    WorkerPoolVariable = 'octoworker' # NOTE: if using default workers remove me
    Properties = @{
      "Octopus.Action.Script.ScriptSource" = $template.Properties.'Octopus.Action.Script.ScriptSource'
      "Octopus.Action.Script.Syntax"       = $template.Properties.'Octopus.Action.Script.Syntax'
      "Octopus.Action.Template.Version"    = $template.Version
      "Octopus.Action.Script.ScriptBody"   = $template.Properties.'Octopus.Action.Script.ScriptBody'
      "Octopus.Action.RunOnServer"         = $true
      "Octopus.Action.Template.Id"         = $template.Id
    }
  })
}

$projects = Invoke-RestMethod "https://example.octopus.app/api/projects/all" -Headers $headers

foreach($project in $projects) {
  # $project = $projects | Where-Object Name -EQ 'Prometheus'
  $process = Invoke-RestMethod "https://example.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers
  if ($process.Steps.Count -eq 0) { continue } # project without any step

  $steps = @()
  $found = $process.Steps | Where-Object { $_.Actions[0].Properties.'Octopus.Action.Template.Id' -eq $template.Id }
  if ($found) {
    $steps += $found
  } else {
    $steps += $step
  }
  foreach($item in $process.Steps) {
    if ($item.Actions[0].Properties.'Octopus.Action.Template.Id' -ne $template.Id) {
      $steps += $item
    }
  }

  $changed = $false
  if ($steps[0].Actions[0].IsDisabled) {
    $steps[0].Actions[0].IsDisabled = $false
    $changed = $true
  }
  if (-not $steps[0].Actions[0].IsRequired) {
    $steps[0].Actions[0].IsRequired = $true
    $changed = $true
  }
  if ((ConvertTo-Json -Depth 100 -InputObject $steps) -ne (ConvertTo-Json -Depth 100 -InputObject $process.Steps)) {
    $changed = $true
  }

  if ($changed) {
    try {
      $process.Steps = $steps
      Invoke-RestMethod -Method Put "https://example.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers -Body (ConvertTo-Json -Depth 100 -Compress -InputObject $process) | Out-Null
      Write-Host $project.Name -ForegroundColor Green
    } catch {
      Write-Host $project.Name -ForegroundColor Red
    }
  } else {
    Write-Host $project.Name -ForegroundColor Cyan
  }
}

Notes:

Be sure to manually add step to some project and get its data to make corresponding changes, for example in my case I am using worker pools which may be not needed in your setup
Be extremely careful when performing the change - you may delete settings

Subscription

Ok, so from now on we have our custom checks added to all projects

We can run installation script every day, week or month to make sure all new projects have it as well

But there is an better way, we are going to subscribe to deployments events, so whenever something is deployed our webhook will be called

It's job is to ensure that checks script in place, if not make - fix it and notify via Slack

From Octopus subscriptions there is nothing interesting just select "Deployment queued" from "Select event categories" menu, give it some name and webhook URL

For webhook itself we are going to cheat little bit and also will use powershell like in previous note

So here are files

default.conf

server {
  listen 80;
  root   /usr/share/nginx/html;

  location /octopus {
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
    fastcgi_param REQUEST_BODY $request_body;
    fastcgi_param SCRIPT_FILENAME /usr/share/nginx/html/webhook.ps1;
  }
}

entrypoint.sh

#!/bin/sh

apk add -q fcgiwrap spawn-fcgi powershell

/usr/bin/spawn-fcgi -s /var/run/fcgiwrap.socket -M 766 /usr/bin/fcgiwrap

webhook.ps1

#!/usr/bin/pwsh

Write-Host 'content-type: text/plain'
Write-Host 'cache-control: no-store'
Write-Host ''
$body = $env:REQUEST_BODY | ConvertFrom-Json
<#
$body = @{
    Payload = @{
        Event = @{
            RelatedDocumentIds = @(
                "Deployments-69481"
            )
        }
    }
}
#>
$headers = @{"X-Octopus-ApiKey" = $env:OCTOPUS_APIKEY}
$deployment = $body.Payload.Event.RelatedDocumentIds | Where-Object { $_.StartsWith("Deployments-") }
$test = $body.Payload.Event.RelatedDocumentIds.Count -eq 1
$deployment = Invoke-RestMethod "https://example.octopus.app/api/deployments/$deployment" -Headers $headers
$project = Invoke-RestMethod "https://example.octopus.app/api/projects/$($deployment.ProjectId)" -Headers $headers
$process = Invoke-RestMethod "https://example.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers
$skipped = $process.Steps | Select-Object -ExpandProperty Actions | Where-Object Id -in $deployment.SkipActions | Select-Object -ExpandProperty Name
$templates = Invoke-RestMethod "https://example.octopus.app/api/actiontemplates/all" -Headers $headers
$template  = $templates | Where-Object Name -EQ 'octochecks'

$step = @{
  Name    = $template.Name
  Actions = @(@{
    Name               = $template.Name
    ActionType         = $template.ActionType
    IsRequired         = $true
    WorkerPoolVariable = 'octoworker' # NOTE: if using default workers remove me
    Properties = @{
      "Octopus.Action.Script.ScriptSource" = $template.Properties.'Octopus.Action.Script.ScriptSource'
      "Octopus.Action.Script.Syntax"       = $template.Properties.'Octopus.Action.Script.Syntax'
      "Octopus.Action.Template.Version"    = $template.Version
      "Octopus.Action.Script.ScriptBody"   = $template.Properties.'Octopus.Action.Script.ScriptBody'
      "Octopus.Action.RunOnServer"         = $true
      "Octopus.Action.Template.Id"         = $template.Id
    }
  })
}


$steps = @()
$found = $process.Steps | Where-Object { $_.Actions[0].Properties.'Octopus.Action.Template.Id' -eq $template.Id }
if ($found) {
  $steps += $found
} else {
  $steps += $step
}
foreach($item in $process.Steps) {
  if ($item.Actions[0].Properties.'Octopus.Action.Template.Id' -ne $template.Id) {
    $steps += $item
  }
}

$changed = $false
if ($steps[0].Actions[0].IsDisabled) {
  $steps[0].Actions[0].IsDisabled = $false
  $changed = $true
}
if (-not $steps[0].Actions[0].IsRequired) {
  $steps[0].Actions[0].IsRequired = $true
  $changed = $true
}
if ((ConvertTo-Json -Depth 100 -InputObject $steps) -ne (ConvertTo-Json -Depth 100 -InputObject $process.Steps)) {
  $changed = $true
}

if ($changed) {
  try {
    $process.Steps = $steps
    Invoke-RestMethod -Method Put "https://example.octopus.app$($project.Links.DeploymentProcess)" -Headers $headers -Body (ConvertTo-Json -Depth 100 -Compress -InputObject $process) | Out-Null
    Write-Host "$($project.Name) - updated"
  } catch {
    Write-Host "$($project.Name) - failed"
  }
}

if ($skipped -contains 'octochecks') {
    Invoke-RestMethod -Method Post "https://slack.com/api/chat.postMessage" -ContentType 'application/json' -Headers @{ Authorization = "Bearer $($env:SLACK_TOKEN)" } -Body (@{channel = '@mac'; text = "$($project.Name) deployed without checks by $($deployment.DeployedBy)"} | ConvertTo-Json)
}

With this scripts in place we may run our docker like so:

docker run -it --rm -p 8080:80 -e OCTOPUS_APIKEY=$OCTOPUS_APIKEY -v $PWD/entrypoint.sh:/docker-entrypoint.d/00-init.sh -v $PWD/default.conf:/etc/nginx/conf.d/default.conf -v $PWD/webhook.ps1:/usr/share/nginx/html/webhook.ps1 nginx:alpine

And optionally (if have nowhere to deploy) expose it:

ngrok http 8080 --region=eu

Once again, it does not solves initial goal, but at least better than nothing, plus slack notifications wont allow engineers to skip it often and sooner or later checks will be fixed

Hopefully in future there will be something similar to Kubernetes admission webhooks