> ## Documentation Index
> Fetch the complete documentation index at: https://docs.swarmd.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrades

> Moving between chart versions without taking your install down — what is preserved, what to check first, and how to roll back.

# Upgrades

An upgrade is one command. This page is about the handful of things that make
that command safe to run against an install with real data in it.

```bash theme={null}
# The version you are moving TO — the newer tarball we sent you.
export SWARMD_CHART=~/Downloads/swarmd-<new-version>.tgz

helm upgrade swarmd $SWARMD_CHART -n swarmd \
  -f my-values.yaml \
  --wait --timeout 15m
```

***

## Getting the new version

We send you a new `swarmd-<version>.tgz` for each release, with notes covering
what changed and anything that needs care on the way to it. There is no
registry to poll and no `--version` flag: **the file is the version.**

That has a couple of practical consequences.

* **Keep old tarballs.** `helm rollback` restores the previous *revision* from
  what Helm already has, so it works without the file. Reinstalling an older
  release from scratch does not — that needs the tarball again, and asking us
  for it is a slower path than keeping a copy.
* **Name your releases after the file.** `helm -n swarmd list` shows the chart
  version, so a cluster can always tell you what it is running; keeping the
  tarballs in one directory makes it just as easy in the other direction.

<Note>
  Read the release notes for every version between where you are and where you
  are going — not just the target. Helm applies one jump; the notes are written
  per version.
</Note>

<Tip>
  Check the file before you run it against a live install:

  ```bash theme={null}
  helm show chart $SWARMD_CHART
  ```

  It prints the name, version and appVersion in two lines, which is the quickest
  way to catch a stale download or the wrong file in `~/Downloads`.
</Tip>

***

## What survives an upgrade

|                       | Preserved? | Notes                                                                                                                                                                                      |
| --------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Generated credentials | ✅          | The chart reads the existing Secret and reuses it. See [the rotation guard](#the-one-that-bricks-installs).                                                                                |
| Postgres data         | ✅          | The PVC is untouched. Schema changes run as migration Jobs.                                                                                                                                |
| Keycloak realm        | ✅          | The bootstrap Job checks whether the realm exists and exits early. Customisations you made through the Admin UI are never overwritten.                                                     |
| ClickHouse data       | ✅          | PVC untouched.                                                                                                                                                                             |
| Your `values.yaml`    | ⚠️         | Only what you pass. Helm does not merge in the values from the previous release unless you use `--reuse-values`, which has its own surprises — pass the same `-f` file every time instead. |

***

## The one that bricks installs

The chart generates `swarmd-generated-credentials` on first install and, on
every upgrade, reads the existing Secret back so nothing rotates. That read is
a Helm `lookup`, and **`lookup` only works when Helm has a live connection to
the cluster.**

When it doesn't, every credential falls through to a freshly generated random
value. Applying that render rotates the Postgres password, the Keycloak admin
password and `encryption-key` in a single step. The first two stop the
platform; the third is worse, because registry, relay and teams encrypt rows
at rest with that key and **a rotated key makes existing rows permanently
unreadable**. No backup of the database helps — you need the old key.

The chart now refuses rather than letting that happen:

```
Refusing to upgrade: could not read the existing Secret
"swarmd-generated-credentials" in namespace "swarmd".
```

Two situations produce it.

<AccordionGroup>
  <Accordion title="helm upgrade --dry-run">
    Plain `--dry-run` is client-side and disables `lookup`. Use
    `--dry-run=server` when you want to preview an upgrade — it keeps the
    cluster connection, so the Secret is read and the render matches what a
    real upgrade would apply.

    ```bash theme={null}
    helm upgrade swarmd $SWARMD_CHART -n swarmd \
      -f my-values.yaml --dry-run=server
    ```
  </Accordion>

  <Accordion title="GitOps that renders with `helm template`">
    Argo CD renders charts with `helm template` by default and applies the
    output. `lookup` returns nothing in that mode, and because the render
    reports itself as an *install* the guard cannot tell it apart from a
    genuine first install — so it does not fire, and the rotation happens
    silently.

    **If you deploy through a template-and-apply GitOps tool, do not let the
    chart own its credentials.** Turn generation off and manage the Secret
    yourself:

    ```yaml theme={null}
    global:
      generatedSecrets:
        enabled: false
        name: swarmd-generated-credentials
    ```

    Then pre-create a Secret of that name with the keys listed in
    [Configuration → Secrets](/self-hosting/configuration#secrets), sourced
    from your secret manager (Vault, OpenBao, External Secrets). This is the
    right shape for GitOps anyway — the credentials stop being a side effect
    of a render.
  </Accordion>
</AccordionGroup>

<Warning>
  **Back up `encryption-key` before your first upgrade**, separately from your
  database, and keep it somewhere a cluster failure cannot take with it.

  ```bash theme={null}
  kubectl -n swarmd get secret swarmd-generated-credentials \
    -o jsonpath='{.data.encryption-key}' | base64 -d
  ```
</Warning>

***

## Before you run it

<Steps>
  <Step title="Take a database backup">
    Migrations run forwards only. There is no `down` script, so a rollback of
    the chart does not roll back the schema — see [Rolling back](#rolling-back).
  </Step>

  <Step title="Confirm you can read the credentials Secret">
    ```bash theme={null}
    kubectl -n swarmd get secret swarmd-generated-credentials
    ```

    If this is missing, stop. Restore it before upgrading.
  </Step>

  <Step title="Preview the change">
    ```bash theme={null}
    helm diff upgrade swarmd $SWARMD_CHART -n swarmd \
      -f my-values.yaml
    ```

    `helm diff` is a plugin (`helm plugin install https://github.com/databus23/helm-diff`)
    and is the single most useful thing you can install for this. Failing
    that, `--dry-run=server` and read the output.
  </Step>

  <Step title="Check you have headroom for the ECR pull">
    The licence bootstrap Job refreshes the image-pull Secret as a
    `pre-upgrade` hook, so an expired registry token is not a problem. A
    licence that has *expired* is — the Job fails and the upgrade stops before
    touching any workload. Check the expiry on your licence first.
  </Step>
</Steps>

***

## What happens during the rollout

Services are replaced in place rather than surged: `maxSurge: 0`,
`maxUnavailable: 1`.

That means a single-replica service is **briefly unavailable** while its pod
is replaced — roughly 40–70 seconds for a JVM service. It is deliberate. The
Kubernetes default starts a second pod before retiring the first, which
doubles the memory every service wants at the same moment; on a node sized for
the steady state the kubelet starts OOMKilling, and it kills the *old* pods
too. A routine upgrade takes down a working install.

If you need a genuinely zero-downtime rollout, run more than one replica and
put the surge back on that service — and size the node for the surge:

```yaml theme={null}
services:
  gateway:
    replicaCount: 2
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
```

Expect to see, in order:

1. Migration Jobs run to completion (`pre-upgrade` hooks, Flyway, forwards only).
2. Pods replaced one service at a time, each held at `Init:0/1` until
   Keycloak's realm answers.
3. The realm bootstrap Job for the new revision runs and exits early, because
   the realm already exists.

**Zero restarts is the expected outcome.** A pod that restarts during an
upgrade is a signal, not noise — check `kubectl -n swarmd describe pod` for
exit code 137 (OOMKilled) and see [JVM sizing](/self-hosting/configuration#jvm-sizing).

***

## Rolling back

```bash theme={null}
helm rollback swarmd -n swarmd --wait --timeout 15m
```

This restores the previous chart revision — image tags, resource limits,
settings, all of it.

<Warning>
  **It does not roll back the database.** Flyway migrations are forwards-only,
  so after a rollback the schema is still the new one while the images are the
  old ones. That is fine for additive migrations and is not fine for anything
  that dropped or renamed a column.

  Treat `helm rollback` as the fast path for a configuration mistake or a bad
  image, and your database backup as the actual recovery path for a failed
  migration. The release notes call out any version where a rollback needs a
  restore.
</Warning>

***

## Skipping versions

Upgrading across several versions in one jump works — migrations are
cumulative and run in order. What it costs you is the ability to tell which
version broke something.

On an install carrying data you care about, step through one minor version at
a time and let each one settle. On a rebuildable environment, jump.

***

## Verifying afterwards

```bash theme={null}
# Every pod Running, zero restarts
kubectl -n swarmd get pods

# Migration Jobs completed, not failed
kubectl -n swarmd get jobs

# The release is at the version you asked for
helm -n swarmd list
```

Then exercise one real path — log in to the UI, or call an agent through the
gateway. Pods being `Running` only tells you the JVMs started.

***

## Next

<CardGroup cols={2}>
  <Card title="Configuration" icon="sliders" href="/self-hosting/configuration">
    Secrets, JVM sizing, and the core/worker split.
  </Card>

  <Card title="Databases" icon="database" href="/self-hosting/databases">
    Layouts, external Postgres and backups.
  </Card>
</CardGroup>
