Skip to content

Scenario 02 — Recover from accidental data loss

Something deleted the data. A bad migration dropped a table, a script ran rm -rf in the wrong directory, an app bug truncated a file. You have nightly backups from scenario 01 and you want yesterday's data back.

Don't overwrite the live volume first. Restore the chosen snapshot into a new, side-by-side PVC, verify it, then cut over. Overwriting a mounted, running database in place can corrupt it, and if you picked the wrong snapshot the original is gone too.

Step 1 — Pick the snapshot from before the incident

Restoring means picking a row. There is no timestamp math. List the candidate Snapshot CRs for the recipe, newest last, and choose the last one from before things broke:

$ kubectl get snapshots -n billing \
    -l kopiur.home-operations.com/config=postgres-data \
    --sort-by=.status.timing.startTime
NAME                          PHASE       ORIGIN      SNAPSHOT    AGE
postgres-data-20260522-021300 Succeeded   scheduled   k1a9...     2d
postgres-data-20260523-021300 Succeeded   scheduled   k1f1...     1d   # <- last good
postgres-data-20260524-021300 Succeeded   scheduled   k2c4...     2h   # incident was here

Step 2 — Restore beside the original

The bundle below restores the chosen Snapshot into a fresh PVC named postgres-data-recovered. The live postgres-data PVC is untouched. The in-place variant, for once you have decided the live data is unrecoverable, is included as a commented block at the bottom of the file.

enableFileDeletion turns the target into a mirror

A restore is additive by default. It writes the snapshot's files and leaves anything else in the target alone.

enableFileDeletion: true deletes files in the target that aren't in the snapshot, making the target an exact mirror. That is what you need for a faithful in-place restore, and it is dangerous if you point it at the wrong PVC. Use it deliberately.

# Scenario 02 — Recover from accidental data loss (someone deleted the data)
#
# A bad migration dropped a table; a `rm -rf` took out a directory. You have
# nightly backups (scenario 01) and want yesterday's data back. The safe play is
# NOT to overwrite the live volume — it's to restore the chosen snapshot into a
# NEW, side-by-side PVC, verify it, then cut the app over. This file is that
# safe restore; the in-place variant is shown (commented) at the bottom.
#
# First, pick the snapshot from BEFORE the incident (restore is "pick a row"):
#
#   kubectl get snapshots -n billing \
#     -l kopiur.home-operations.com/snapshot-policy=postgres-data \
#     --sort-by=.status.timing.startTime
#
# Field shapes verified against crates/api: source/target are externally tagged
# (source.snapshotRef, target.pvc creates a PVC / target.pvcRef writes an existing
# one). Explicit sources default to onMissingSnapshot: Fail (fail-closed).
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Restore
metadata:
  name: postgres-recover-pre-incident
  namespace: billing
spec:
  source:
    snapshotRef:
      # The specific Snapshot CR you picked from the list above — the last good one.
      name: postgres-data-20260523-021300
      namespace: billing
  target:
    # Restore BESIDE the original into a fresh PVC; the live volume is untouched.
    pvc:
      name: postgres-data-recovered
      storageClassName: fast-ssd
      capacity: 100Gi
      accessModes:
        - ReadWriteOnce
  options:
    # Additive (the default): write the snapshot's files, leave anything already
    # in the target alone. Into a fresh PVC this is moot, but it's the safe habit.
    enableFileDeletion: false
  policy:
    onMissingSnapshot: Fail # an explicit recovery that finds nothing is an error
    waitTimeout: 5m
#
# Watch it:   kubectl get restore postgres-recover-pre-incident -n billing -w
# Then point the app at postgres-data-recovered (or copy the rows you need out
# of it), confirm, and delete the original / the recovered PVC as appropriate.
#
# ── In-place variant (overwrites the LIVE volume — scale the app DOWN first) ──
# Only do this once you've decided the live data is unrecoverable. Restoring into
# a mounted, running database is a corruption hazard.
#
# apiVersion: kopiur.home-operations.com/v1alpha1
# kind: Restore
# metadata:
#   name: postgres-recover-in-place
#   namespace: billing
# spec:
#   source:
#     snapshotRef:
#       name: postgres-data-20260523-021300
#   target:
#     pvcRef:
#       name: postgres-data        # the EXISTING live PVC
#   options:
#     enableFileDeletion: true     # make the volume an EXACT mirror of the snapshot
#   policy:
#     onMissingSnapshot: Fail

Step 3 — Watch it, then cut over

$ kubectl get restore postgres-recover-pre-incident -n billing -w
NAME                            PHASE        AGE
postgres-recover-pre-incident   Resolving    2s
postgres-recover-pre-incident   Restoring    9s
postgres-recover-pre-incident   Completed    41s

Now point the app at postgres-data-recovered, or copy the rows you need out of it. Confirm the data is what you expected. Only then retire the original.

Explicit restores fail closed with onMissingSnapshot: Fail. If the snapshot you named isn't there, the restore errors instead of quietly producing an empty volume.

See also