Permissions, UID & GID

The most common reason a backup runs but reads nothing, or a restore writes files the app then cannot open, is a UID or GID mismatch. This page shows how to find the right numbers, how to set them, and how to check it worked, with no guesswork.

The mental model: the mover is a separate pod

A backup does not run inside your app’s pod. Kopiur launches a short-lived mover Job that mounts your PVC and runs kopia. Linux file permissions do not care that it is “your” data. They only see the UID and GID the mover process runs as. So the rule is simply:

• Backup: the mover’s UID and GID must be able to read every file in the source PVC.
• Restore: the mover’s UID and GID must be able to write into the target PVC.

Get the numbers to line up and permissions stop being a problem.

Looking for the field reference?

This page is the task guide: how to find the owning UID and make a stuck backup or restore work.

For the full securityContext reference, meaning every field, the hardened default, inheriting from a workload, root and privileged movers, and the awkward cases such as RWX volumes and preserving ownership on restore, see The mover security context.

What the mover runs as by default

Out of the box the mover runs unprivileged, as the mover image’s user, which is UID 65532, the distroless nonroot user. It runs with a hardened security context: runAsNonRoot: true, allowPrivilegeEscalation: false, all Linux capabilities dropped, and seccomp RuntimeDefault.

That default reads data that is world-readable or owned by 65532. If your app writes files 0600 or 0640 owned by some other UID, which is very common because most images run as 1000, 1001 or 999, an unprivileged mover at 65532 gets permission denied on those files.

You then have three options, best first:

1. Run the mover as the same UID and GID that owns the data.
2. Run the mover with a GID that matches a group the files are readable by.
3. Run the mover as root, which reads anything but is elevated and needs an admin opt-in. This is the last resort.

The rest of this page is how to do options 1 and 2 reliably, and when to reach for option 3.

Step 1 — Find the UID/GID that owns your data

You want the numeric owner of the files in the PVC. Numeric, not names: the mover image has no /etc/passwd entry for your app’s user, so ls -l showing a name is misleading. Use -n for numeric.

If the workload is running, read it straight from the app pod:

$ kubectl exec -n app deploy/myapp -- id
uid=1000(app) gid=1000(app) groups=1000(app)

$ kubectl exec -n app deploy/myapp -- ls -ln /data
drwxr-xr-x 2 1000 1000 4096 Jun  6 12:00 .
-rw------- 1 1000 1000  512 Jun  6 12:00 secret.key   # 0600, owner-only

Here the data is owned by 1000:1000 and some files are owner-only at 0600, so the mover must run as UID 1000. Matching the group is not enough for 0600 files.

If nothing is mounting the PVC, for example a fresh restore target or a scaled-down app, spin up a throwaway pod that mounts it read-only and inspect it:

$ kubectl run pvc-inspect -n app --rm -it --restart=Never \
    --image=busybox --overrides='
{
  "spec": {
    "containers": [{
      "name": "x", "image": "busybox", "command": ["sh"], "stdin": true, "tty": true,
      "volumeMounts": [{"name": "d", "mountPath": "/data", "readOnly": true}]
    }],
    "volumes": [{"name": "d", "persistentVolumeClaim": {"claimName": "app-data"}}]
  }
}' -- sh

/ # stat -c '%u %g %a %n' /data /data/*    # numeric uid, gid, mode, name
1000 1000 755 /data
1000 1000 600 /data/secret.key

Look for the strictest file. If any file you need is 0600 owned by 1000, the mover has to be UID 1000. If everything is at least group-readable, so 0640 or 0750, and shares a GID, matching the GID is enough.

Step 2 — Set the mover’s UID/GID in the SnapshotPolicy

Set it per recipe under spec.mover.securityContext, which is a standard Kubernetes container SecurityContext. Match what you found in Step 1:

spec:
  mover:
    securityContext:
      runAsUser: 1000 # the UID that owns the data
      runAsGroup: 1000 # the GID that owns the data
      runAsNonRoot: true # keep the unprivileged guarantee
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
      seccompProfile:
        type: RuntimeDefault

A complete, apply-ready example, with a Repository and a SnapshotPolicy carrying this block plus the root-mover variant commented out, is Example 09:

fsGroup lives on mover.podSecurityContext

runAsUser and runAsGroup above are container-level, under spec.mover.securityContext. fsGroup is pod-level, so it has its own sibling field, spec.mover.podSecurityContext.fsGroup.

fsGroup is the right tool when an unprivileged mover must write a freshly provisioned restore volume, because the kubelet makes the mount group-writable by that GID. For reading source data, prefer matching the owning runAsUser and runAsGroup. See The mover security context → fsGroup.

fsGroup does nothing on NFS

fsGroup relies on the kubelet chowning the volume, and the kubelet does not do that for in-tree NFS mounts. So fsGroup cannot grant write access to an NFS source or an NFS-backed filesystem repository.

Use one of these instead: podSecurityContext.supplementalGroups against a group-writable export, since supplemental GIDs are honored over NFS; securityContext.runAsUser matching the export owner; or a server-side remap such as TrueNAS Mapall or all_squash. See Security context → NFS filesystem repositories.

Step 3 — Verify it worked

Re-run the backup and confirm it actually read files, rather than quietly snapshotting an empty or partial tree:

# the mover Job's exact name lives on the Snapshot (it's named after the Snapshot CR):
$ kubectl get snapshot <snapshot-name> -n app -o jsonpath='{.status.job.name}'
app-data-manual-abc12

# its pod (by the standard Job-managed pod label), then the container's effective UID:
$ kubectl get pods -n app --selector=job-name=app-data-manual-abc12
$ kubectl get pod <mover-pod> -n app \
    -o jsonpath='{.spec.containers[0].securityContext.runAsUser}{"\n"}'
1000

# permission errors, if any, surface in the mover log and on the Snapshot status:
$ kubectl logs <mover-pod> -n app | grep -i "permission denied"
$ kubectl get snapshot <snapshot-name> -n app -o jsonpath='{.status.conditions}'

A healthy backup ends Succeeded with non-zero files and bytes in status. A backup that “succeeded” but shows zero files is the classic sign the mover could not read the data. Recheck the UID.

When you can’t match the UID: the root mover

If the data is owned by assorted UIDs you cannot match, such as a lost+found, a multi-user volume, or an app that writes as root, a privileged root mover reads everything. Set:

spec:
  mover:
    securityContext:
      runAsUser: 0
      runAsNonRoot: false
    privilegedMode: true # root only: adds DAC_OVERRIDE (backup) or CHOWN/DAC_OVERRIDE/FOWNER/FSETID (restore)

You need privilegedMode: true, not just runAsUser: 0. The mover drops every Linux capability by default, and root without DAC_OVERRIDE gets permission denied on a 0660 file owned by another UID and GID, like anyone else. privilegedMode adds back only the capabilities the operation needs, and only to a root mover. It never changes the UID: without runAsUser: 0 it grants nothing and Kopiur warns. The full table, and the capabilities.add workaround for older versions, are in Security context → What privilegedMode grants.

A root, or otherwise elevated, mover is a privileged mover, and granting it is a per-namespace admin decision. If the namespace has not opted in, the Snapshot is refused with a clear MoverPermitted=False condition.

Opt the namespace in by applying a Namespace carrying the opt-in annotation:

deploy/examples/privileged-mover-namespace.yaml

# Privileged-mover namespace opt-in (docs/src/content/docs/permissions.mdx, docs/src/content/docs/security-context.mdx)
#
# A privileged (or root) mover — `runAsUser: 0`, `privileged: true`,
# `allowPrivilegeEscalation: true`, added Linux capabilities, `runAsNonRoot:
# false`, or `privilegedMode: true` — is refused with a `MoverPermitted=False`
# condition until the workload namespace is opted in. This Namespace carries the
# opt-in annotation, so backups, restores, and maintenance in `media` may run an
# elevated mover.
#
# This is the declarative form of the per-namespace admin decision; applying it
# is equivalent to:
#
#   kubectl annotate namespace media kopiur.home-operations.com/privileged-movers=true
#
# Granting a namespace privileged movers is a CLUSTER-ADMIN action: the operator
# mints a mover ServiceAccount in this namespace, and a tenant here could reuse
# it at the mover's privilege. Apply it only for namespaces you trust to run
# elevated workloads. Removing the annotation revokes the opt-in.
---
apiVersion: v1
kind: Namespace
metadata:
  name: media
  annotations:
    # The opt-in. Any other value (or absence) leaves the namespace un-opted-in.
    kopiur.home-operations.com/privileged-movers: "true"

$ kubectl apply -f privileged-mover-namespace.yaml

Or do it imperatively with kubectl annotate namespace <ns> kopiur.home-operations.com/privileged-movers=true.

Anything that trips the privileged detector needs that opt-in: runAsUser: 0, privileged: true, allowPrivilegeEscalation: true, added Linux capabilities, runAsNonRoot: false, or privilegedMode: true. The full detail and the revoke path are in Movers → Privileged movers. Removing the annotation is a real revoke here, because a privileged mover’s privilege lives in the pod spec. The separate stream-exec opt-in is not like that: its privilege lives in a minted ServiceAccount that outlives the annotation and has to be deleted explicitly.

The condition names the layer that actually carries the elevation — Repository.spec.moverDefaults, the recipe’s spec.mover, an inherited workload context, or a per-run Snapshot.spec.mover — so you edit the spec that raised it rather than reaching for the namespace-wide annotation to unblock one ad-hoc run.

Prefer matching the UID over going root

A root mover widens the blast radius of the mover ServiceAccount Kopiur mints. Reach for it only when you genuinely cannot match the owning UID or GID. Most single-app PVCs back up fine as their app’s UID.

Filesystem repositories: the other permission

The UID and GID story above is about reading source data. A filesystem repository, backed by a PVC or NFS, adds a second surface: the repository path itself must be writable by the operator and mover UID.

When create or connect cannot write the repository path, Kopiur does not hang. It emits a Warning Event, and a Bootstrapped=False condition, naming the actual UID it runs as and the fix:

$ kubectl describe repository nas-primary -n backups
...
Warning  PermissionDenied  the repository path is not writable by the operator's UID (65532) —
  fix its ownership/mode (e.g. `chown -R 65532 /repo`) and reconcile again.

The UID in that message is the operator’s real effective UID, which varies with the chart’s podSecurityContext.runAsUser, so the chown it prints is always correct for your install. Run it on the NAS or host backing the PVC, then reconcile.

For an NFS-backed repository whose export is owned by a dedicated UID and GID while your apps run as other UIDs, do not reach for fsGroup, which does nothing on NFS. Make the export group-writable instead, and give every backend writer the shared group: movers through moverDefaults.podSecurityContext.supplementalGroups, and the kopia UI server through server.podSecurityContext.supplementalGroups. That keeps per-policy source reads running as the app’s UID while the group grants repository writes. The full recipe and an apply-ready example are in Security context → NFS filesystem repositories.

Restore-side permissions

A restore writes files into the target PVC, so the same rules apply in reverse. A Restore has the same spec.mover surface a SnapshotPolicy does:

• Restore.spec.mover.securityContext sets the UID and GID the restore mover writes as, so the restored files land owned correctly and the mover can write the target. Before this field existed the restore mover always ran as UID 65532. For a freshly created target PVC, meaning target.pvc, the default is usually fine. For an existing PVC, meaning target.pvcRef, match the UID that owns it.
• Restore.spec.mover.inheritSecurityContextFrom copies the securityContext from a live workload pod by label selector, instead of hard-coding it. It combines with securityContext, which overrides it field by field, and it needs the workload to pin runAsUser. It is handy for “restore as whatever the app runs as”, and is covered fully in Security context → Inherit it from the workload.
• Preserving original ownership. Kopia restores files with the UID and GID they had when snapshotted. Reproducing that ownership requires a root mover with privilegedMode: true, which grants CHOWN, DAC_OVERRIDE, FOWNER and FSETID. An unprivileged mover writes files owned by its own UID instead. A root mover without privilegedMode holds no capabilities, so with the default ignorePermissionErrors: true it completes but leaves every file owned by root. An elevated restore mover, whether root, privilegedMode, or inherited from a root pod, is gated by the same privileged-movers namespace opt-in a backup uses.
• spec.options.ignorePermissionErrors, default true, lets a restore complete and report permission problems through a condition rather than failing hard. Set it false to fail closed when exact permissions matter.

See Restores → Mover, cache & failure policy for the full restore mover surface, and example 12.

Try it end-to-end

See the UID-match fix work, and watch the classic anti-pattern fail, from a clean slate.

The setup is a PVC seeded with files owned 1000:1000, mode 0600, so readable only by UID 1000, backed up by a mover that runs as runAsUser: 1000. The backup reads the files, so status.stats.filesNew is non-zero. The default mover, at UID 65532, would “succeed” while reading zero.

It is one apply-ready bundle, deploy/examples/tryit/permissions-uid.yaml: the app Namespace, a PVC, a seed Job, an S3 Repository, a UID-matched SnapshotPolicy, and a manual Snapshot.

The seed writes owner-only files that only UID 1000 can read:

# Seed the PVC with files owned 1000:1000, mode 0600 (owner-only). Only a mover
# running as UID 1000 can read them — UID 65532 (the default) gets nothing.
apiVersion: batch/v1
kind: Job
metadata:
  name: seed-app-data
  namespace: app
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000 # make the fresh volume writable by GID 1000 so the seed can chown
      containers:
        - name: seed
          image: busybox:1.36
          command: ["sh", "-c"]
          args:
            - |
              set -eu
              mkdir -p /data/config
              echo "owner-only config readable only by uid 1000" > /data/config/app.yaml
              head -c 2097152 /dev/urandom > /data/blob.bin   # ~2 MiB so stats are non-trivial
              chmod 0600 /data/config/app.yaml
              ls -ln /data /data/config
          volumeMounts:
            - name: d
              mountPath: /data
      volumes:
        - name: d
          persistentVolumeClaim:
            claimName: app-data

The policy matches the mover to that owner:

apiVersion: kopiur.home-operations.com/v1alpha1
kind: SnapshotPolicy
metadata:
  name: app-data
  namespace: app
spec:
  repository:
    name: app-primary
  sources:
    - pvc:
        name: app-data
  retention:
    keepDaily: 7
    keepWeekly: 4
  # ── Match the mover UID/GID to the data owner (1000:1000) ─────────────────────
  # This is the fix: without it the default mover (UID 65532) cannot read the
  # 0600 files and the backup "succeeds" with ZERO files. With it, the mover
  # reads everything and status.stats.filesNew is non-zero.
  mover:
    securityContext:
      runAsUser: 1000 # the UID that owns the data
      runAsGroup: 1000 # the GID that owns the data
      runAsNonRoot: true # keep the unprivileged guarantee
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
      seccompProfile:
        type: RuntimeDefault

1. Fill in the credentials, meaning AWS_* and KOPIA_PASSWORD, in the secret section, then apply the bundle:

$ kubectl apply -f deploy/examples/tryit/permissions-uid.yaml
$ kubectl -n app wait --for=condition=Ready repository/app-primary --timeout=2m
$ kubectl -n app wait --for=condition=complete job/seed-app-data --timeout=2m

2. Take the backup. The Snapshot uses generateName, so create it:

$ kubectl create -f deploy/examples/tryit/permissions-uid.yaml
snapshot.kopiur.home-operations.com/app-data-manual-abc12 created

$ kubectl -n app wait --for=jsonpath='{.status.phase}'=Succeeded \
    snapshot/app-data-manual-abc12 --timeout=5m

3. Prove it read real data (deep). The matched UID means non-zero filesNew, and the mover pod ran as 1000:

# illustrative stats — non-zero filesNew/bytesNew is the proof:
$ kubectl -n app get snapshot app-data-manual-abc12 -o jsonpath='{.status.stats}{"\n"}'
{"sizeBytes":2097640,"bytesNew":2097640,"filesNew":2, ...}

# the mover Job is named after the Snapshot CR (no -snap suffix):
$ kubectl -n app get snapshot app-data-manual-abc12 -o jsonpath='{.status.job.name}{"\n"}'
app-data-manual-abc12

# its pod's effective UID — 1000, matching the data owner:
$ kubectl -n app get pods --selector=job-name=app-data-manual-abc12
$ kubectl -n app get pod <mover-pod> \
    -o jsonpath='{.spec.containers[0].securityContext.runAsUser}{"\n"}'
1000

The anti-pattern: the default UID reads nothing

Drop the mover block, or set runAsUser: 65532, and re-run. The backup still ends Succeeded, but status.stats.filesNew reads 0: the unprivileged mover got permission denied on every 0600 file and snapshotted an empty tree without saying so.

A Succeeded snapshot with zero files is the canonical sign of a UID mismatch. Match the owning UID as described above, or, if the data is owned by UIDs you cannot match, reach for a root mover.

Illustrative output

The status.stats numbers and the app-data-manual-abc12 and <mover-pod> names stand in for what your run produces. The shape, and the non-zero filesNew, are the point.

Troubleshooting

Symptom	Where it shows	Cause	Fix
Backup Succeeded but 0 files / 0 bytes	Snapshot .status	Mover UID can’t read the source files.	Match spec.mover.securityContext.runAsUser/Group to the data owner (Steps 1–2). To try a UID without touching the shared recipe, set it on a one-shot Snapshot.spec.mover.
Mover log: permission denied reading source	Mover pod logs	Same as above: partial read.	Same as above; or a root mover with privilegedMode: true if UIDs can’t be matched. Root alone holds no DAC_OVERRIDE.
Backup stuck Pending, MoverPermitted=False, reason PrivilegedMoverNotPermitted	Snapshot condition / Event	The merged mover requests privilege; namespace not opted in. The message names the layer that carries it — moverDefaults, the policy’s spec.mover, an inherited workload context, or a per-run Snapshot.spec.mover.	Drop the elevated context from the object the message names. Only reach for kubectl annotate namespace <ns> kopiur.home-operations.com/privileged-movers=true if you mean to allow it for every mover in that namespace.
Repository Failed, PermissionDenied	Repository Event / condition	Filesystem repo path not writable by the operator UID.	chown -R <uid> <path> (the Event names the UID), then reconcile.
Restored files unreadable by the app	After restore	Files restored as the mover’s UID, not the original owner.	Set Restore.spec.mover.securityContext.runAsUser/Group to the app’s UID (or inheritSecurityContextFrom); use a root mover with privilegedMode: true to preserve the original ownership exactly.
Restore stuck Pending, MoverPermitted=False, reason PrivilegedMoverNotPermitted	Restore condition / Event	Restore mover requests privilege; namespace not opted in.	kubectl annotate namespace <ns> kopiur.home-operations.com/privileged-movers=true, or drop the elevated context.
Backup or restore stuck Pending, MoverPermitted=False, reason StreamExecNotPermitted	Snapshot / Restore condition / Event	A different capability on the same condition: a stream source or streamExec target needs pods/exec in the workload namespace, and that has its own opt-in. The privileged-movers annotation does not grant it.	kubectl annotate namespace <ns> kopiur.home-operations.com/stream-exec-movers=true. Under a namespaced install also set rbacNamespaceReadForStreamSources: true — the check needs cluster-scoped get on namespaces and fails closed without it. See Streamed command sources.

Quick reference

Thing	Value
Default mover UID	65532 (distroless nonroot), runAsNonRoot: true; pod fsGroup: 65532 so the kopia cache is writable on PVC-backed storage
Set the mover UID/GID	SnapshotPolicy.spec.mover.securityContext.runAsUser / runAsGroup (same on Restore.spec.mover / Maintenance.spec.mover)
Override the UID/GID for ONE backup	Snapshot.spec.mover, merged field-wise over the policy’s: moverDefaults < SnapshotPolicy.spec.mover < Snapshot.spec.mover. For an ad-hoc run that must not edit the shared recipe. See Backups → mover for one run
Inherit UID/GID from a workload	spec.mover.inheritSecurityContextFrom: pvcConsumer: {} (backup: auto-derive from the PVC’s consumer) or workloadSelector (by label; required on a Restore). Needs the workload to pin runAsUser; combines with securityContext (explicit wins). See Security context
Mover cache size / warm cache	spec.mover.cache (capacity, storageClassName, mode: Ephemeral/Persistent, content/metadataCacheSizeMb); inherits Repository.spec.moverDefaults.cache. A per-run Snapshot.spec.mover.cache is run-scoped: it moves that run’s budgets and, under mode: Ephemeral, that Job’s cache-volume capacity/storageClassName. It cannot change mode, nor a Persistent cache PVC’s spec — both are policy-owned, since that PVC is shared by every sibling Snapshot
fsGroup	spec.mover.podSecurityContext.fsGroup, which defaults to 65532 (keeps the cache writable); override to make a fresh restore volume writable by a mover running as a different UID
Root / preserve-ownership	runAsUser: 0 + runAsNonRoot: false + privilegedMode: true (needs the namespace opt-in). privilegedMode is what grants the capabilities: backup DAC_OVERRIDE, restore CHOWN/DAC_OVERRIDE/FOWNER/FSETID
Privileged-mover opt-in	kubectl annotate namespace <ns> kopiur.home-operations.com/privileged-movers=true
Stream-exec opt-in (stream / streamExec)	kubectl annotate namespace <ns> kopiur.home-operations.com/stream-exec-movers=true. Separate from, and not implied by, the privileged-mover annotation: it authorizes pods/exec in that namespace, which is arbitrary code execution in every pod there. Fails closed — including when the Namespace cannot be read. See Streamed command sources
Revoking the stream-exec opt-in	Removing the annotation stops new runs but does not revoke the grant. Also kubectl delete rolebinding <release>-stream-mover -n <ns> and kubectl delete serviceaccount <release>-stream-mover -n <ns>. See Revoking the opt-in
Filesystem repo not writable	Event prints chown -R <uid> <path> with the real operator UID
Restore ignore/permission errors	Restore.spec.options.ignorePermissionErrors (default true)

See also

• Movers, RBAC & credentials covers privileged movers, the minted ServiceAccount, and credential placement.
• Backend configuration covers the filesystem and SFTP backends, where ownership matters most.
• Restores covers restore targets and options.
• Streamed command sources covers the stream-exec-movers opt-in, which gates pods/exec for stream sources and streamExec restore targets.