Skip to content

Scenario 03 — Disaster recovery on a fresh cluster

The cluster is gone. A failed upgrade, a deleted namespace, a dead control plane — but the repository in object storage survives. That's the entire point of off-cluster backups. You stand up a new cluster, apply your GitOps repo, and the app's data comes back as part of that apply, with no "fresh install or recovery?" branching.

This is the headline deploy-or-restore pattern, hardened for DR with two changes from a normal install.

What makes this a DR bundle (vs. example 05)

  1. The Repository connects to the existing repo (create.enabled: false) — it must already exist; we are not initializing a new empty one. A typo in the bucket then surfaces as a connect error instead of silently creating a second, empty repository at the wrong address.
  2. A passive Restore (source.fromPolicy, no target, onMissingSnapshot: Continue) is wired into the PVC's dataSourceRef as a volume populator, so the PVC restores the latest snapshot before the app starts.

Identity must match the old cluster

kopia finds the surviving snapshots by username@hostname:path. The defaults are username = SnapshotPolicy name and hostname = namespace, so rebuilding with the same name in the same namespace resolves the same snapshots automatically. This bundle pins identity explicitly anyway, so recovery still works even if you rebuild into a differently-named namespace. The KOPIA_PASSWORD must also be the original one — kopia cannot decrypt the repo with a new password.

The values you must get right

Field Must equal Why
KOPIA_PASSWORD the original repo password kopia can't decrypt otherwise.
backend.s3.bucket / prefix the surviving bucket/prefix that's where the snapshots are.
create.enabled false connect, don't re-initialize.
identity.username / hostname what the old cluster recorded so fromPolicy resolves the old snapshots. 
# Scenario 03 — Disaster recovery: rebuild an app on a fresh cluster
#
# The cluster (or the whole namespace) is gone, but the repository in object
# storage survives — that's the whole point of off-cluster backups. You stand up
# a new cluster and apply your GitOps repo; this bundle brings the app's DATA
# back as part of that apply, with NO "is this a fresh install or a recovery?"
# branching.
#
# Two differences from a normal install make this a DR bundle:
#   1. The Repository CONNECTS to the existing repo (create.enabled: false) — it
#      must already exist; we are not initializing a new empty one.
#   2. A PASSIVE populator Restore (source.fromPolicy, target.populator: {},
#      onMissingSnapshot: Continue) is wired into the PVC's dataSourceRef, so
#      the PVC restores the latest snapshot BEFORE the app starts. On a truly
#      empty repo the PVC just comes up blank and is backed up going forward.
#
# The load-bearing subtlety — IDENTITY must match the old cluster. kopia finds
# the old snapshots by `username@hostname:path`. By default username = the
# SnapshotPolicy name and hostname = the namespace, so re-creating the config with
# the SAME name in the SAME namespace resolves the same snapshots automatically.
# We pin `identity` explicitly here so the recovery is robust even if you rebuild
# into a differently-named namespace.
#
# Field shapes verified against crates/api. See also example 05 (deploy-or-
# restore) and the Restores guide.
---
apiVersion: v1
kind: Secret
metadata:
  name: postgres-repo-creds
  namespace: billing
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "REPLACE_ME"
  AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
  # MUST be the same password the old cluster used — kopia can't decrypt otherwise.
  KOPIA_PASSWORD: "REPLACE_ME_with_the_ORIGINAL_repo_password"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Repository
metadata:
  name: postgres-primary
  namespace: billing
spec:
  backend:
    s3:
      # Point at the EXISTING bucket/prefix the old cluster wrote to.
      bucket: my-backups
      prefix: billing/postgres/
      endpoint: s3.us-east-1.amazonaws.com
      region: us-east-1
      auth:
        secretRef:
          name: postgres-repo-creds
  encryption:
    passwordSecretRef:
      name: postgres-repo-creds
      key: KOPIA_PASSWORD
  create:
    enabled: false # CONNECT to the surviving repo; do NOT initialize a new one
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: SnapshotPolicy
metadata:
  name: postgres-data
  namespace: billing
spec:
  repository:
    name: postgres-primary
  sources:
    - pvc:
        name: postgres-data
  # Pin the identity to what the OLD cluster recorded, so fromPolicy below
  # resolves the surviving snapshots no matter where you rebuild.
  identity:
    username: postgres-data
    hostname: billing
  retention:
    keepDaily: 14
    keepWeekly: 6
---
# PASSIVE populator Restore (target.populator: {}). It does nothing on its own;
# the PVC below consumes it via spec.dataSourceRef.
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Restore
metadata:
  name: postgres-data-restore
  namespace: billing
spec:
  source:
    fromPolicy:
      name: postgres-data
      offset: 0 # 0 = latest surviving snapshot
  # `target` is REQUIRED; populator intent is the explicit empty form.
  target:
    populator: {}
  policy:
    # Default for fromPolicy; explicit here. Empty repo ⇒ PVC comes up blank
    # instead of failing the deploy.
    onMissingSnapshot: Continue
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: billing
spec:
  storageClassName: fast-ssd
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  # The populator handshake — needs k8s >= 1.24 (AnyVolumeDataSource). The PVC is
  # provisioned by restoring the snapshot the passive Restore above resolves.
  dataSourceRef:
    apiGroup: kopiur.home-operations.com
    kind: Restore
    name: postgres-data-restore
---
# Resume nightly backups on the new cluster, so DR also re-establishes protection.
apiVersion: kopiur.home-operations.com/v1alpha1
kind: SnapshotSchedule
metadata:
  name: postgres-data-nightly
  namespace: billing
spec:
  policyRef:
    name: postgres-data
  schedule:
    cron: "H 2 * * *"
    jitter: 30m
    runOnCreate: false

What happens on apply

flowchart LR
  R[Repository<br/>connects to existing repo] --> BC[SnapshotPolicy<br/>identity pinned]
  BC --> RS[passive Restore<br/>fromPolicy: latest]
  RS -->|dataSourceRef populator| PVC[PVC postgres-data<br/>restored before app starts]
  BC --> SCH[SnapshotSchedule<br/>protection resumes]

On a cluster pointed at the existing repo, the PVC is provisioned by restoring the latest snapshot. On a genuinely empty repo, onMissingSnapshot: Continue lets the PVC come up blank and be backed up going forward — the same manifests either way.

Verify the recovery

$ kubectl get repository postgres-primary -n billing
NAME               PHASE   AGE
postgres-primary   Ready   20s

$ kubectl get pvc postgres-data -n billing
NAME            STATUS   VOLUME    CAPACITY   AGE
postgres-data   Bound    pvc-...   100Gi      35s

$ kubectl get restore postgres-data-restore -n billing
NAME                    PHASE       AGE
postgres-data-restore   Completed   40s

A Bound PVC and a Completed populator Restore mean the data is back; start the app against it.

Kubernetes ≥ 1.24

The volume-populator handshake needs the AnyVolumeDataSource feature (GA from 1.24). The optional volume-data-source-validator surfaces a malformed dataSourceRef as an event instead of a silently-stuck PVC.

See also