Upgrades

An upgrade is one command. This page is about the handful of things that make that command safe to run against an install with real data in it.

# The version you are moving TO — the newer tarball we sent you.
export SWARMD_CHART=~/Downloads/swarmd-<new-version>.tgz

helm upgrade swarmd $SWARMD_CHART -n swarmd \
  -f my-values.yaml \
  --wait --timeout 15m

Getting the new version

We send you a new swarmd-<version>.tgz for each release, with notes covering what changed and anything that needs care on the way to it. There is no registry to poll and no --version flag: the file is the version. That has a couple of practical consequences.

Keep old tarballs. helm rollback restores the previous revision from what Helm already has, so it works without the file. Reinstalling an older release from scratch does not — that needs the tarball again, and asking us for it is a slower path than keeping a copy.
Name your releases after the file. helm -n swarmd list shows the chart version, so a cluster can always tell you what it is running; keeping the tarballs in one directory makes it just as easy in the other direction.

Read the release notes for every version between where you are and where you are going — not just the target. Helm applies one jump; the notes are written per version.

Check the file before you run it against a live install:

helm show chart $SWARMD_CHART

It prints the name, version and appVersion in two lines, which is the quickest way to catch a stale download or the wrong file in ~/Downloads.

What survives an upgrade

	Preserved?	Notes
Generated credentials	✅	The chart reads the existing Secret and reuses it. See the rotation guard.
Postgres data	✅	The PVC is untouched. Schema changes run as migration Jobs.
Keycloak realm	✅	The bootstrap Job checks whether the realm exists and exits early. Customisations you made through the Admin UI are never overwritten.
ClickHouse data	✅	PVC untouched.
Your `values.yaml`	⚠️	Only what you pass. Helm does not merge in the values from the previous release unless you use `--reuse-values`, which has its own surprises — pass the same `-f` file every time instead.

The one that bricks installs

The chart generates swarmd-generated-credentials on first install and, on every upgrade, reads the existing Secret back so nothing rotates. That read is a Helm lookup, and lookup only works when Helm has a live connection to the cluster. When it doesn’t, every credential falls through to a freshly generated random value. Applying that render rotates the Postgres password, the Keycloak admin password and encryption-key in a single step. The first two stop the platform; the third is worse, because registry, relay and teams encrypt rows at rest with that key and a rotated key makes existing rows permanently unreadable. No backup of the database helps — you need the old key. The chart now refuses rather than letting that happen:

Refusing to upgrade: could not read the existing Secret
"swarmd-generated-credentials" in namespace "swarmd".

Two situations produce it.

helm upgrade --dry-run

Plain --dry-run is client-side and disables lookup. Use --dry-run=server when you want to preview an upgrade — it keeps the cluster connection, so the Secret is read and the render matches what a real upgrade would apply.

helm upgrade swarmd $SWARMD_CHART -n swarmd \
  -f my-values.yaml --dry-run=server

GitOps that renders with helm template

Argo CD renders charts with helm template by default and applies the output. lookup returns nothing in that mode, and because the render reports itself as an install the guard cannot tell it apart from a genuine first install — so it does not fire, and the rotation happens silently.If you deploy through a template-and-apply GitOps tool, do not let the chart own its credentials. Turn generation off and manage the Secret yourself:

global:
  generatedSecrets:
    enabled: false
    name: swarmd-generated-credentials

Then pre-create a Secret of that name with the keys listed in Configuration → Secrets, sourced from your secret manager (Vault, OpenBao, External Secrets). This is the right shape for GitOps anyway — the credentials stop being a side effect of a render.

Back up encryption-key before your first upgrade, separately from your database, and keep it somewhere a cluster failure cannot take with it.

kubectl -n swarmd get secret swarmd-generated-credentials \
  -o jsonpath='{.data.encryption-key}' | base64 -d

Before you run it

Take a database backup

Migrations run forwards only. There is no down script, so a rollback of the chart does not roll back the schema — see Rolling back.

Confirm you can read the credentials Secret

kubectl -n swarmd get secret swarmd-generated-credentials

If this is missing, stop. Restore it before upgrading.

Preview the change

helm diff upgrade swarmd $SWARMD_CHART -n swarmd \
  -f my-values.yaml

helm diff is a plugin (helm plugin install https://github.com/databus23/helm-diff) and is the single most useful thing you can install for this. Failing that, --dry-run=server and read the output.

Check you have headroom for the ECR pull

The licence bootstrap Job refreshes the image-pull Secret as a pre-upgrade hook, so an expired registry token is not a problem. A licence that has expired is — the Job fails and the upgrade stops before touching any workload. Check the expiry on your licence first.

What happens during the rollout

Services are replaced in place rather than surged: maxSurge: 0, maxUnavailable: 1. That means a single-replica service is briefly unavailable while its pod is replaced — roughly 40–70 seconds for a JVM service. It is deliberate. The Kubernetes default starts a second pod before retiring the first, which doubles the memory every service wants at the same moment; on a node sized for the steady state the kubelet starts OOMKilling, and it kills the old pods too. A routine upgrade takes down a working install. If you need a genuinely zero-downtime rollout, run more than one replica and put the surge back on that service — and size the node for the surge:

services:
  gateway:
    replicaCount: 2
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

Expect to see, in order:

Migration Jobs run to completion (pre-upgrade hooks, Flyway, forwards only).
Pods replaced one service at a time, each held at Init:0/1 until Keycloak’s realm answers.
The realm bootstrap Job for the new revision runs and exits early, because the realm already exists.

Zero restarts is the expected outcome. A pod that restarts during an upgrade is a signal, not noise — check kubectl -n swarmd describe pod for exit code 137 (OOMKilled) and see JVM sizing.

Rolling back

helm rollback swarmd -n swarmd --wait --timeout 15m

This restores the previous chart revision — image tags, resource limits, settings, all of it.

It does not roll back the database. Flyway migrations are forwards-only, so after a rollback the schema is still the new one while the images are the old ones. That is fine for additive migrations and is not fine for anything that dropped or renamed a column.Treat helm rollback as the fast path for a configuration mistake or a bad image, and your database backup as the actual recovery path for a failed migration. The release notes call out any version where a rollback needs a restore.

Skipping versions

Upgrading across several versions in one jump works — migrations are cumulative and run in order. What it costs you is the ability to tell which version broke something. On an install carrying data you care about, step through one minor version at a time and let each one settle. On a rebuildable environment, jump.

Verifying afterwards

# Every pod Running, zero restarts
kubectl -n swarmd get pods

# Migration Jobs completed, not failed
kubectl -n swarmd get jobs

# The release is at the version you asked for
helm -n swarmd list

Then exercise one real path — log in to the UI, or call an agent through the gateway. Pods being Running only tells you the JVMs started.

Configuration

Secrets, JVM sizing, and the core/worker split.

Databases

Layouts, external Postgres and backups.

Get started

How it works

Shape your deployment

Reference

Upgrades

Upgrades

Getting the new version

What survives an upgrade

The one that bricks installs

Before you run it

What happens during the rollout

Rolling back

Skipping versions

Verifying afterwards

Next

Configuration

Databases

​Upgrades

​Getting the new version

​What survives an upgrade

​The one that bricks installs

​Before you run it

​What happens during the rollout

​Rolling back

​Skipping versions

​Verifying afterwards

​Next

Configuration

Databases

Upgrades

Getting the new version

What survives an upgrade

The one that bricks installs

Before you run it

What happens during the rollout

Rolling back

Skipping versions

Verifying afterwards

Next