Skip to content

Scenario 03 — Disaster recovery on a fresh cluster

The cluster is gone. A failed upgrade, a deleted namespace, a dead control plane. But the repository in object storage survived, which is the entire point of off-cluster backups. You stand up a new cluster, apply your GitOps repo, and the app's data comes back as part of that apply. There is no "fresh install or recovery?" branch to pick.

This is the headline deploy-or-restore pattern, hardened for disaster recovery with two changes from a normal install.

What makes this a DR bundle (vs. example 05)

  1. The Repository connects to the existing repository, with create.enabled: false. The repository must already exist; we are not initializing a new empty one. A typo in the bucket then shows up as a connect error, instead of quietly creating a second, empty repository at the wrong address.
  2. A passive Restore is attached to the PVC's dataSourceRef as a volume populator. It uses source.fromPolicy, has no target, and sets onMissingSnapshot: Continue. The PVC therefore restores the latest snapshot before the app starts.

If you were mirroring the repository off-site

This scenario connects to the surviving repository, so the rebuilt cluster keeps writing into it.

If what survived is an off-site mirror and you want the new cluster to have its own repository back, with the mirror left intact, seed a new one instead. See Scenario 10: DR from a replicated repository.

Identity must match the old cluster

kopia finds the surviving snapshots by username@hostname:path. The defaults are username = SnapshotPolicy name and hostname = namespace, so rebuilding with the same name in the same namespace resolves the same snapshots automatically.

This bundle pins identity explicitly anyway, so recovery still works even if you rebuild into a differently-named namespace.

The KOPIA_PASSWORD must also be the original one. kopia cannot decrypt the repository with a new password.

Check retention before re-applying policies over surviving history

The SnapshotPolicy below adopts the repository's surviving snapshots. An adopted snapshot is then governed by GFS retention like any produced backup.

Under the default deletionPolicy: Delete, everything outside spec.retention is pruned from the repository immediately. Retention prunes deliberately bypass the mass-deletion breaker.

So a five-year history re-adopted under keepDaily: 14 loses the rest. Widen retention to what you actually intend to keep, or set the policy's defaultDeletionPolicy: Retain so pruning a row deletes only the Snapshot CR.

The values you must get right

Field Must equal Why
KOPIA_PASSWORD the original repo password kopia can't decrypt otherwise.
backend.s3.bucket / prefix the surviving bucket/prefix that's where the snapshots are.
create.enabled false connect, don't re-initialize.
identity.username / hostname what the old cluster recorded so fromPolicy resolves the old snapshots. 
# Scenario 03 — Disaster recovery: rebuild an app on a fresh cluster
#
# The cluster (or the whole namespace) is gone, but the repository in object
# storage survives — that's the whole point of off-cluster backups. You stand up
# a new cluster and apply your GitOps repo; this bundle brings the app's DATA
# back as part of that apply, with NO "is this a fresh install or a recovery?"
# branching.
#
# Two differences from a normal install make this a DR bundle:
#   1. The Repository CONNECTS to the existing repo (create.enabled: false) — it
#      must already exist; we are not initializing a new empty one.
#   2. A PASSIVE populator Restore (source.fromPolicy, target.populator: {},
#      onMissingSnapshot: Continue) is wired into the PVC's dataSourceRef, so
#      the PVC restores the latest snapshot BEFORE the app starts. On a truly
#      empty repo the PVC just comes up blank and is backed up going forward.
#
# The load-bearing subtlety — IDENTITY must match the old cluster. kopia finds
# the old snapshots by `username@hostname:path`. By default username = the
# SnapshotPolicy name and hostname = the namespace, so re-creating the config with
# the SAME name in the SAME namespace resolves the same snapshots automatically.
# We pin `identity` explicitly here so the recovery is robust even if you rebuild
# into a differently-named namespace.
#
# Field shapes verified against crates/api. See also example 05 (deploy-or-
# restore) and the Restores guide.
---
apiVersion: v1
kind: Secret
metadata:
  name: postgres-repo-creds
  namespace: billing
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "REPLACE_ME"
  AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
  # MUST be the same password the old cluster used — kopia can't decrypt otherwise.
  KOPIA_PASSWORD: "REPLACE_ME_with_the_ORIGINAL_repo_password"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Repository
metadata:
  name: postgres-primary
  namespace: billing
spec:
  backend:
    s3:
      # Point at the EXISTING bucket/prefix the old cluster wrote to.
      bucket: my-backups
      prefix: billing/postgres/
      endpoint: s3.us-east-1.amazonaws.com
      region: us-east-1
      auth:
        secretRef:
          name: postgres-repo-creds
  encryption:
    passwordSecretRef:
      name: postgres-repo-creds
      key: KOPIA_PASSWORD
  create:
    enabled: false # CONNECT to the surviving repo; do NOT initialize a new one
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: SnapshotPolicy
metadata:
  name: postgres-data
  namespace: billing
spec:
  repository:
    name: postgres-primary
  sources:
    - pvc:
        name: postgres-data
  # Pin the identity to what the OLD cluster recorded, so fromPolicy below
  # resolves the surviving snapshots no matter where you rebuild.
  identity:
    username: postgres-data
    hostname: billing
  retention:
    keepDaily: 14
    keepWeekly: 6
---
# PASSIVE populator Restore (target.populator: {}). It does nothing on its own;
# the PVC below consumes it via spec.dataSourceRef.
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Restore
metadata:
  name: postgres-data-restore
  namespace: billing
spec:
  source:
    fromPolicy:
      name: postgres-data
      offset: 0 # 0 = latest surviving snapshot
  # `target` is REQUIRED; populator intent is the explicit empty form.
  target:
    populator: {}
  policy:
    # Default for fromPolicy; explicit here. Empty repo ⇒ PVC comes up blank
    # instead of failing the deploy.
    onMissingSnapshot: Continue
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: billing
spec:
  storageClassName: fast-ssd
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  # The populator handshake — needs k8s >= 1.24 (AnyVolumeDataSource). The PVC is
  # provisioned by restoring the snapshot the passive Restore above resolves.
  dataSourceRef:
    apiGroup: kopiur.home-operations.com
    kind: Restore
    name: postgres-data-restore
---
# Resume nightly backups on the new cluster, so DR also re-establishes protection.
apiVersion: kopiur.home-operations.com/v1alpha1
kind: SnapshotSchedule
metadata:
  name: postgres-data-nightly
  namespace: billing
spec:
  policyRef:
    name: postgres-data
  schedule:
    cron: "H 2 * * *"
    jitter: 30m
    runOnCreate: false

What happens on apply

flowchart LR
  R[Repository<br/>connects to existing repo] --> BC[SnapshotPolicy<br/>identity pinned]
  BC --> RS[passive Restore<br/>fromPolicy: latest]
  RS -->|dataSourceRef populator| PVC[PVC postgres-data<br/>restored before app starts]
  BC --> SCH[SnapshotSchedule<br/>protection resumes]

On a cluster pointed at the existing repository, the PVC is provisioned by restoring the latest snapshot. On a genuinely empty repository, onMissingSnapshot: Continue lets the PVC come up blank and be backed up going forward. The same manifests work either way.

Verify the recovery

$ kubectl get repository postgres-primary -n billing
NAME               PHASE   AGE
postgres-primary   Ready   20s

$ kubectl get pvc postgres-data -n billing
NAME            STATUS   VOLUME    CAPACITY   AGE
postgres-data   Bound    pvc-...   100Gi      35s

$ kubectl get restore postgres-data-restore -n billing
NAME                    PHASE       AGE
postgres-data-restore   Completed   40s

A Bound PVC and a Completed populator Restore mean the data is back. Start the app against it.

Kubernetes ≥ 1.24

The volume-populator handshake needs the AnyVolumeDataSource feature, which is GA from 1.24. The optional volume-data-source-validator surfaces a malformed dataSourceRef as an event instead of a PVC that silently never binds.

See also