S3 & S3-compatible¶

The S3 backend works for Amazon S3 and for any S3-compatible object store: MinIO, RustFS, Ceph RGW, Wasabi, Backblaze B2's S3 API, Cloudflare R2, SeaweedFS, and others. It is the most common choice, and it's the backend used by the first example.

Reach for S3 when your storage is an object store with an S3 API. Azure Blob, Google Cloud Storage, and native Backblaze B2 have dedicated backends that are easier to configure: Azure, GCS, and B2. For a NAS, use filesystem or SFTP.

Provider prerequisites¶

A bucket. Kopiur does not create it. Make it in the provider console or CLI first.
An access key, meaning an AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY pair, with read, write, list, and delete on that bucket. Prefer a key scoped to the one bucket over a root or admin key.
For AWS, the bucket's region. For a compatible store, its endpoint host: a bare host[:port], with no https:// scheme.

Minimal AWS IAM policy for the key

kopia needs to list, read, write, and delete objects under the repository prefix. Here is a bucket-scoped policy that grants exactly that:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
            "Resource": "arn:aws:s3:::my-backups"
        },
        {
            "Effect": "Allow",
            "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
            "Resource": "arn:aws:s3:::my-backups/*"
        }
    ]
}

s3:DeleteObject is not optional, because retention and maintenance delete expired blobs. A key that can read and write but not delete makes snapshots succeed and maintenance fail later, which is confusing to debug. MinIO and most compatible stores accept the same policy document.

The Secret shape¶

The mover loads this Secret with envFrom. Every key below must be a valid environment-variable name, and each one reaches kopia as an environment variable.

Secret key	Required	What it is
`AWS_ACCESS_KEY_ID`	yes	Access-key ID for the bucket.
`AWS_SECRET_ACCESS_KEY`	yes	Secret access key paired with the ID.
`AWS_SESSION_TOKEN`	no	Only for temporary STS credentials. Omit for a static key.
`KOPIA_PASSWORD`	yes	The repository encryption password (every backend needs it).

stringData:
    AWS_ACCESS_KEY_ID: "REPLACE_ME"
    AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
    KOPIA_PASSWORD: "choose-something-long-and-random"

Lose the password, lose the backups

KOPIA_PASSWORD encrypts the repository. kopia cannot decrypt without it, and there is no recovery. Use a long random value, store it outside the cluster, and back up the Secret itself. See Encryption and repository creation.

The Repository¶

---
apiVersion: v1
kind: Secret
metadata:
  name: s3-repo-creds
  namespace: backups
type: Opaque
stringData:
  # Read by well-known names. AWS_SESSION_TOKEN is optional (STS only).
  AWS_ACCESS_KEY_ID: "REPLACE_ME"
  AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
  # The kopia repository encryption password. Lose it and the backups are
  # unrecoverable — store it outside the cluster too.
  KOPIA_PASSWORD: "choose-something-long-and-random"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Repository
metadata:
  name: s3-primary
  namespace: backups
spec:
  backend:
    s3:
      bucket: my-backups # the bucket holding the kopia repository
      prefix: clusters/prod/ # optional; lets several repos share one bucket
      region: us-east-1 # required by AWS; some compatible stores want it too
      # endpoint: OMIT for AWS. For MinIO/RustFS/Ceph set the host, e.g.
      #   endpoint: minio.storage.svc.cluster.local:9000
      #   tls:
      #     disableTls: true        # in-cluster plain HTTP (kopia assumes HTTPS otherwise)
      #     # For HTTPS signed by your own CA, point caBundleRef at a ConfigMap
      #     # in THIS Repository's namespace (key defaults to ca.crt) — the
      #     # complete example is backends/s3-private-ca.yaml:
      #     # caBundleRef: { configMapName: s3-private-ca }
      #     # insecureSkipVerify: true   # last resort; ignores caBundleRef if both are set
      auth:
        secretRef:
          name: s3-repo-creds
  encryption:
    passwordSecretRef:
      name: s3-repo-creds
      key: KOPIA_PASSWORD
  create:
    enabled: true # initialize the repo if it does not exist yet (default: false)

Fields reference (`backend.s3`)¶

Field	Required	Default	Example	What it controls
`bucket`	yes	—	`my-backups`	The bucket holding the kopia repository. Just the name: no `s3://`, no path.
`prefix`	no	bucket root	`clusters/prod/`	Key prefix so several repositories can share one bucket. End it with `/` or the prefix concatenates into object names.
`endpoint`	no	AWS	`minio.storage.svc:9000`	S3 endpoint host for a compatible store. Omit for AWS. Bare `host[:port]`, no scheme, no trailing slash.
`region`	no	—	`us-east-1`	Region. Required by AWS and some compatible stores. Any string works for MinIO.
`auth.secretRef`	no¹	—	`{ name: s3-repo-creds }`	Names the credential Secret above. Same namespace as the `Repository`; a `ClusterRepository` adds `namespace:`. Mutually exclusive with `workloadIdentity`.
`auth.workloadIdentity.serviceAccountName`	no¹	—	`backup-mover`	Run the mover Jobs as this ServiceAccount instead of using static keys. You create it and federate it with IAM. See Workload identity.
`tls.disableTls`	no	`false`	`true`	Talk plain HTTP instead of HTTPS. This is kopia's `--disable-tls`, for in-cluster MinIO on a plaintext port. It contradicts `caBundleRef`, so that pair is rejected at admission.
`tls.insecureSkipVerify`	no	`false`	`true`	Skip TLS certificate verification. This is kopia's `--disable-tls-verification`, and it is a last resort; prefer `caBundleRef`. If you set both, skip-verify wins, the bundle is ignored, and admission warns.
`tls.caBundleRef`	no	—	`{ configMapName: s3-private-ca }`	A `ConfigMap` key holding a PEM CA bundle, so kopia trusts a private-CA or self-signed endpoint. `key` defaults to `ca.crt`. It is resolved in the `Repository`'s namespace, or the operator's namespace for a `ClusterRepository`, and copied into every mover. See Private-CA HTTPS.

¹ Set exactly one of auth.secretRef or auth.workloadIdentity. The webhook enforces this. You may omit auth entirely when the AWS_* keys live in the encryption-password Secret.

Endpoint & region by provider¶

The same four identifiers cover every S3-compatible store. What changes is which ones you set. ENDPOINT is always a bare host, so strip the https:// the provider's console shows you.

Provider	`endpoint`	`region`	Notes
Amazon S3	omit	the bucket's region	kopia derives the endpoint from the region.
MinIO / RustFS (in-cluster)	`minio.storage.svc.cluster.local:9000`	any string (`main`)	Plain HTTP needs `tls.disableTls: true`. See below.
Cloudflare R2	`<accountid>.r2.cloudflarestorage.com`	`auto`	The account ID is in the R2 dashboard. Use an R2 API token's S3 credentials.
Wasabi	`s3.<region>.wasabisys.com`	matches the endpoint (`us-east-1`)	Region must agree with the endpoint host.
Backblaze B2 (S3 API)	`s3.<region>.backblazeb2.com`	matches the endpoint (`us-west-004`)	Or use the native B2 backend: same storage, application-key auth.
Ceph RGW	`rgw.ceph.svc:7480` (your RGW service)	your zonegroup (often `default`)	For self-signed HTTPS, use `tls.caBundleRef`.

A filled-in stanza for Cloudflare R2, say, differs from the AWS example only in these identifiers:

backend:
    s3:
        bucket: my-backups
        endpoint: 0123456789abcdef.r2.cloudflarestorage.com
        region: auto
        auth:
            secretRef:
                name: s3-repo-creds # AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY from the R2 API token

Customization — the values you actually change¶

bucket and prefix set where snapshots land. Several repositories can share one bucket by using distinct prefixes.
endpoint and tls are what you set for any non-AWS store. See the plain-HTTP MinIO and private-CA HTTPS variants below.
region is required for AWS. Any value works for MinIO and RustFS.
create.enabled defaults to true: kopiur initializes the repository if none exists at the prefix. Set it to false when the repository must already exist, for example a read-only or externally managed one. create.encryption, create.splitter, and create.hash are fixed forever at creation. See repository creation.
moverDefaults.cache sizes the mover's kopia cache PVC. See movers.

In-cluster MinIO / RustFS over plain HTTP¶

kopia's S3 path assumes HTTPS. For a plain-HTTP in-cluster endpoint, set tls.disableTls: true and give a bare host with no scheme.

For an HTTPS endpoint with a certificate your cluster doesn't already trust, use tls.caBundleRef instead. Do not use disableTls, and do not use insecureSkipVerify.

# Backend: S3 — in-cluster MinIO / RustFS over plain HTTP
#
# A variant of the S3 backend for an S3-compatible store reached over plain HTTP
# (e.g. a MinIO/RustFS Service inside the cluster). kopia's S3 path assumes HTTPS,
# so you must opt into plain HTTP with `tls.disableTls: true`. The endpoint is a
# bare host[:port] — NO scheme, NO trailing slash.
#
# Field shapes verified against crates/api: externally-tagged backend
# (`backend.s3`); `tls.disableTls` maps to kopia's `--disable-tls`.
---
apiVersion: v1
kind: Secret
metadata:
  name: minio-repo-creds
  namespace: backups
type: Opaque
stringData:
  # MinIO/RustFS access key pair (the "root" user, or a scoped service account).
  AWS_ACCESS_KEY_ID: "REPLACE_ME"
  AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
  KOPIA_PASSWORD: "choose-something-long-and-random"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Repository
metadata:
  name: minio-primary
  namespace: backups
spec:
  backend:
    s3:
      bucket: kopia # the bucket holding the repository (create it in MinIO first)
      prefix: clusters/prod/ # optional; lets several repos share one bucket
      # Bare host[:port] of the in-cluster Service — no http:// scheme.
      endpoint: minio.storage.svc.cluster.local:9000
      region: us-east-1 # MinIO ignores it but kopia wants a value; any string works
      tls:
        disableTls: true # talk plain HTTP (kopia otherwise assumes HTTPS)
      auth:
        secretRef:
          name: minio-repo-creds
  encryption:
    passwordSecretRef:
      name: minio-repo-creds
      key: KOPIA_PASSWORD
  create:
    enabled: true # initialize the repo if it does not exist yet

Private-CA HTTPS (trusting your own CA)¶

Your endpoint's HTTPS certificate may be signed by your own certificate authority (CA), such as an internal PKI or a self-signed MinIO or Ceph RGW certificate. Point tls.caBundleRef at a ConfigMap key holding the CA bundle in PEM format.

Don't reach for tls.insecureSkipVerify: true. The bundle keeps real certificate verification; skip-verify throws it away.

# Backend: S3 — private-CA HTTPS (internal CA / self-signed endpoint)
#
# A variant of the S3 backend for an S3-compatible store served over HTTPS with
# a certificate your cluster does not already trust — an internal PKI, or a
# self-signed MinIO / Ceph RGW cert. `tls.caBundleRef` names a ConfigMap key
# holding the CA bundle (PEM). The operator resolves the CONTENT at reconcile
# time and inlines it into every mover work spec, so backup, restore,
# maintenance, verification, replication, and the repository server all verify
# the endpoint against your CA.
#
# Namespace rule: the ConfigMap is read from the Repository's OWN namespace
# (for a ClusterRepository: the operator's namespace, KOPIUR_NAMESPACE). It
# does NOT need to exist in workload namespaces — the content travels in the
# work spec, never as a volume reference.
#
# Field shapes verified against crates/api: externally-tagged backend
# (`backend.s3`); `tls.caBundleRef: { configMapName, key }` with `key`
# defaulting to `ca.crt`. kopia builds a FRESH certificate pool from this
# bundle (it does not extend the system roots): if the endpoint's chain also
# needs a public root, concatenate both PEMs into the same key.
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: s3-private-ca
  namespace: backups # a namespaced Repository resolves the ConfigMap HERE
data:
  # `key` defaults to ca.crt, so this name needs no `key:` in the Repository.
  # Paste your CA (or the whole chain, certificates concatenated) PEM-encoded.
  # The block below is a TRUNCATED placeholder, not a real certificate.
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    REPLACE_ME...TRUNCATED-PLACEHOLDER...paste-your-CA-chain-here
    -----END CERTIFICATE-----
---
apiVersion: v1
kind: Secret
metadata:
  name: s3-private-ca-creds
  namespace: backups
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "REPLACE_ME"
  AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
  KOPIA_PASSWORD: "choose-something-long-and-random"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Repository
metadata:
  name: s3-private-ca
  namespace: backups
spec:
  backend:
    s3:
      bucket: kopia # the bucket holding the repository (create it first)
      prefix: clusters/prod/ # optional; lets several repos share one bucket
      # Bare host[:port] of the HTTPS endpoint — no https:// scheme.
      endpoint: s3.internal.example.com
      region: us-east-1 # any string for MinIO/RGW; must be real for AWS-style stores
      tls:
        caBundleRef:
          configMapName: s3-private-ca # the ConfigMap above
          # key: ca.crt               # the default — set it only for another key
        # Do NOT combine caBundleRef with disableTls (rejected at admission:
        # a CA bundle is meaningless over plain HTTP) or insecureSkipVerify
        # (admission warns: skip-verify wins and the bundle is ignored).
      auth:
        secretRef:
          name: s3-private-ca-creds
  encryption:
    passwordSecretRef:
      name: s3-private-ca-creds
      key: KOPIA_PASSWORD
  create:
    enabled: true # initialize the repo if it does not exist yet

How it works: the operator reads the ConfigMap's content at reconcile time and puts the PEM directly into the mover work spec. kopia receives it through its own root-CA mechanism and saves it in the connection config.

Every operation therefore verifies the endpoint against your CA: backup, restore, maintenance, verification, replication including a private-CA destination, the repository server, and kubectl kopiur browse.

Where the ConfigMap lives

The ConfigMap is resolved in the Repository's own namespace, or for a ClusterRepository, in the operator's namespace, which is KOPIUR_NAMESPACE.

It does not need to exist in the workload namespaces movers run in, because the content travels inside the work spec rather than as a volume reference. key defaults to ca.crt.

The bundle replaces the trust store, it doesn't extend it

kopia builds a fresh certificate pool from your bundle for the S3 connection. System roots are not consulted for that endpoint.

Replicating from a private-CA source to public AWS is fine, because each side connects separately with its own trust. But if a single endpoint's chain mixes your CA with a public root, concatenate both PEMs into the one bundle.

Two combinations are rejected at admission, and one is warned about:

caBundleRef without configMapName is rejected. The reference must name a ConfigMap.
caBundleRef with disableTls: true is rejected. The two contradict each other, because a CA bundle means nothing over plain HTTP.
caBundleRef with insecureSkipVerify: true is admitted with a warning. Skip-verify wins and the bundle is ignored while both are set.

Failure modes are typed and visible:

Missing ConfigMap or key. The repository surfaces a MissingCaBundle condition and a Warning event naming the namespace, ConfigMap, and key. kubectl kopiur doctor reports it too. GitOps apply ordering heals itself, because the ConfigMap appearing re-triggers the reconcile.
Not PEM, meaning no BEGIN CERTIFICATE block, or over 64 KiB. You get a validation error naming the ConfigMap. Trim the bundle to the CA chain this endpoint actually needs.

Object lock (ransomware protection)¶

If the operator, the chart, or the cluster is compromised, whoever holds the repository credentials can delete every backup.

Object lock makes the object store refuse the delete. A blob is locked for a fixed period when it is written, and until that expires the storage layer rejects deletion no matter who asks.

Declare it under spec.parameters.blobRetention:

# Example 38 — object-lock blob retention / ransomware protection
# (spec.parameters.blobRetention)
#
# THE SYMPTOM: nothing, until the day it matters. If the Kopiur container, the Helm
# release, or the cluster itself is compromised, an attacker holding the repository
# credentials can delete every backup you have. Backups that a single stolen
# credential can erase are not a recovery plan.
#
# THE FIX: have the object store refuse the delete. S3 (and Azure/GCS) can lock a
# blob for a fixed period at the moment it is written; until that period expires the
# storage layer rejects deletion, no matter who asks. kopia drives this through
# `repository set-parameters --retention-mode/--retention-period`, and this field
# declares it so kopiur re-applies it whenever it drifts.
#
# PREREQUISITE THAT KOPIUR CANNOT DO FOR YOU: the bucket must have Object Lock
# ENABLED AT CREATION TIME. It cannot be turned on afterwards, and these flags do not
# create it. On AWS that is `aws s3api create-bucket --object-lock-enabled-for-bucket`;
# on MinIO/RustFS, `mc mb --with-lock`. Without it, kopia fails with
# `blob-retention: unsupported put-blob option`.
#
# SUPPORTED BACKENDS: s3, azure, gcs. Anything else (filesystem, sftp, webdav,
# rclone, b2, gdrive) is rejected at admission — those protocols have no object lock,
# and kopia would hard-fail on every single reconcile rather than quietly ignoring it.
#
# ABSENT MEANS "DON'T TOUCH", exactly like spec.parameters.epoch. Deleting this block
# from your manifest does NOT turn retention off — it leaves the repository at
# whatever you last applied, so a stray `kubectl apply` can never silently strip
# ransomware protection. To actually turn it off, say so:
#
#   parameters:
#     blobRetention:
#       disabled: true
#
# READ THE RESULT at status.parameters.blobRetention, which mirrors what the
# repository actually reports — so a failed apply shows up as drift from spec rather
# than as silence.
#
# ┌──────────────────────────────────────────────────────────────────────────────┐
# │ LIMITATION — protection is a ROLLING WINDOW, not cumulative immutability.     │
# │                                                                              │
# │ The lock is applied when a blob is WRITTEN. Kopiur does not enable kopia's    │
# │ `maintenance set --extend-object-locks`, so locks on blobs that are still     │
# │ needed are never extended: a blob written today under `period: 720h` becomes  │
# │ deletable again in 30 days. Size the period to comfortably exceed your        │
# │ longest recovery window, and run `kopia maintenance set                       │
# │ --extend-object-locks=true` by hand if you need durable protection.           │
# │                                                                              │
# │ Blobs written BEFORE you enabled retention — including the repository format  │
# │ blob — are never retroactively locked.                                        │
# └──────────────────────────────────────────────────────────────────────────────┘
#
# GOVERNANCE vs COMPLIANCE — choose deliberately:
#
#   governance: locked against ordinary deletes, but a sufficiently privileged
#               identity can still shorten or remove the lock. Start here.
#
#   compliance: NOBODY can shorten or remove the lock before it expires — not you,
#               not the account root, not support. A mistyped period is an unfixable
#               storage-cost commitment for its full duration. Only choose this if
#               you specifically need protection against an attacker who has your
#               cloud admin credentials, and only after testing the period on a
#               throwaway bucket.
#
# PERIOD GRAMMAR: kopiur accepts h/m/s (or a bare number of seconds) — write 30 days
# as `720h`. Note that kopia's own CLI *does* accept `30d`; kopiur deliberately keeps
# one duration grammar across every CRD field, so a period copied from kopia's
# documentation is rejected at admission with a message naming the fix. kopia's
# minimum is 1 day; there is no maximum.
#
# EXPECT THE REPOSITORY TO GROW. Maintenance still issues deletes, but the storage
# layer refuses them until the lock expires, so reclaim lags your retention period.
# That is the cost of the guarantee, not a bug.
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Repository
metadata:
  name: s3-locked
  namespace: kopiur-system
spec:
  backend:
    s3:
      bucket: my-kopiur-backups # MUST have been created with object lock enabled
      region: us-east-1
      auth:
        secretRef:
          name: s3-credentials
  encryption:
    passwordSecretRef:
      name: kopiur-repo-password
      key: KOPIA_PASSWORD
  parameters:
    blobRetention:
      # Exactly one of governance / compliance / disabled — the CRD's own schema
      # enforces it, so "a mode without a period" is not expressible.
      governance:
        period: 720h # 30 days. NOT `30d` — see PERIOD GRAMMAR above.

Locks are not extended, so protection is a rolling window

Kopiur applies the lock at write time only. kopia's maintenance set --extend-object-locks re-locks blobs that are still needed as full maintenance runs, but it is not configurable through Kopiur today. So a blob written under period: 720h becomes deletable again 30 days later, even if your retention policy still needs it.

Treat the period as a rolling floor, not as cumulative immutability. Size it to comfortably exceed your longest recovery window, and run kopia maintenance set --extend-object-locks=true by hand if you need durable protection.

Blobs written before you enabled retention are never locked retroactively. That includes the repository format blob.

compliance cannot be undone

Under governance, a sufficiently privileged identity can still shorten or remove a lock. Under compliance, nobody can. Not you, not the account root.

A mistyped period is then a storage-cost commitment you cannot undo for its full duration. Start with governance, and test any compliance period on a throwaway bucket first.

Three things that bite:

The bucket must have object lock enabled at creation. It cannot be turned on later, and these fields do not create it. Use aws s3api create-bucket --object-lock-enabled-for-bucket, or mc mb --with-lock. Without it kopia fails with blob-retention: unsupported put-blob option, and Kopiur surfaces that as a Warning event on every reconcile.
Write the period as 720h, not 30d. Kopiur accepts h, m, and s only, one grammar across every CRD field. kopia's own CLI does take 30d, so a value copied from kopia's documentation is rejected at admission. The error names the fix. The minimum is 1 day.
Expect the repository to grow. Maintenance still issues deletes, but the storage layer refuses them until locks expire, so space is reclaimed later than your retention period suggests. That is the cost of the guarantee. It is also why s3:DeleteObject stays in the bucket policy: kopia keeps issuing the deletes, and it is the lock, not a missing permission, that declines them.

Leaving the block out means "don't touch". Removing it leaves the repository at whatever you last applied, so a stray apply cannot silently strip protection. Turn it off explicitly with blobRetention: { disabled: true }. Read back what actually landed at status.parameters.blobRetention.

Workload identity (IRSA / EKS Pod Identity)¶

On EKS, or any cluster with AWS Identity and Access Management (IAM) federation, you can drop the static AWS_* keys entirely.

Set auth.workloadIdentity.serviceAccountName and every mover Job for this repository runs as that ServiceAccount. kopia then resolves credentials through the ambient AWS chain: the IRSA web-identity token, an EKS Pod Identity association, or instance metadata. The only secret left in the cluster is KOPIA_PASSWORD.

What you provide:

An IAM role with the bucket policy above.
A ServiceAccount federated to that role, either through IRSA's eks.amazonaws.com/role-arn annotation or through an EKS Pod Identity association. It must exist in every namespace mover Jobs run in: the repository's own namespace, plus each consumer namespace for a ClusterRepository.
auth.workloadIdentity.serviceAccountName on the backend, instead of auth.secretRef.

Kopiur does three things. It checks the ServiceAccount exists before every run, so a missing one surfaces as CredentialsAvailable=False naming it, instead of a Job that never schedules. It binds the kopiur-mover role to that ServiceAccount, because the mover patches its own CR status at runtime. And because kopia's CLI insists on its access-key flags, it invokes kopia with explicitly empty --access-key= and --secret-access-key=, so the AWS credential chain takes over.

Kopiur never touches your ServiceAccount

The ServiceAccount's annotations are how you federate it with AWS. Kopiur only reads it with get and adds a RoleBinding named kopiur-mover-wi-<sa>. Create and manage the ServiceAccount yourself, or through your GitOps tooling.

# Backend: S3 with workload identity — IRSA / EKS Pod Identity
#
# No static AWS keys anywhere: the mover Jobs run as a ServiceAccount federated
# to an IAM role, and kopia resolves credentials through the ambient AWS chain
# (IRSA web-identity token or EKS Pod Identity). The only secret left in the
# cluster is the kopia repository encryption password.
#
# Prerequisites (AWS side):
#   - an IAM role with the bucket policy (see the S3 docs page), trust-federated
#     to this ServiceAccount via your cluster's OIDC provider (IRSA), OR an EKS
#     Pod Identity association pointing at it.
#
# Field shapes verified against crates/api: externally-tagged backend
# (`backend.s3`); `auth.workloadIdentity` is mutually exclusive with
# `auth.secretRef` (webhook-enforced).
---
# You own this ServiceAccount — Kopiur preflights it and binds the mover role to
# it, but never creates or modifies it (the annotation is your contract with the
# cloud's identity webhook). It must exist in every namespace mover Jobs run in.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: backup-mover
  namespace: backups
  annotations:
    # IRSA: the IAM role this SA may assume. For EKS Pod Identity, omit the
    # annotation and create a pod-identity association instead.
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/kopia-backups
---
apiVersion: v1
kind: Secret
metadata:
  name: s3-wi-repo-creds
  namespace: backups
type: Opaque
stringData:
  # Workload identity replaces the AWS_* keys — but the repository encryption
  # password is kopia's, not the cloud's, and is still required.
  KOPIA_PASSWORD: "choose-something-long-and-random"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Repository
metadata:
  name: s3-wi
  namespace: backups
spec:
  backend:
    s3:
      bucket: my-backups
      region: us-east-1 # the bucket's region (kopia derives the AWS endpoint)
      auth:
        # Instead of secretRef: the mover Jobs run AS this ServiceAccount and
        # kopia authenticates via the ambient AWS credential chain.
        workloadIdentity:
          serviceAccountName: backup-mover
  encryption:
    passwordSecretRef:
      name: s3-wi-repo-creds
      key: KOPIA_PASSWORD
  create:
    enabled: true

Replication: don't mix static and workload-identity S3 pairs

A RepositoryReplication between two S3 backends must use the same auth style on both sides. If both use workloadIdentity, they must name the same ServiceAccount.

A mixed pair is rejected at admission. The replication pod's environment carries the static side's AWS_* keys, and the workload-identity side's credential chain would silently pick them up, authenticating as the wrong identity.

Cross-cloud pairs, such as S3 to GCS, are unaffected.

As a `ClusterRepository`¶

The same backend.s3 stanza works on a cluster-scoped ClusterRepository, with two differences. Every Secret reference must carry an explicit namespace:, which the webhook enforces. And the credential Secret must exist in the namespaces the movers run in. See Movers → where the credential Secret lives.

# Backend: S3 as a shared ClusterRepository
#
# The SAME S3 backend stanza, but on a cluster-scoped ClusterRepository owned by a
# platform team and referenced by allow-listed tenant namespaces. Two things differ
# from a namespaced Repository:
#   1. EVERY Secret reference must carry an explicit `namespace:` (webhook-enforced
#      — a cluster-scoped CR has no namespace of its own to default to).
#   2. `allowedNamespaces` gates which namespaces may reference it, and
#      `identityDefaults` derives each consumer's kopia identity from CEL
#      expressions evaluated at admission.
#
# REQUIRES the operator installed with installScope=cluster.
#
# Field shapes verified against crates/api: externally-tagged backend
# (`backend.s3`) and `allowedNamespaces` (list | selector | all).
---
apiVersion: v1
kind: Secret
metadata:
  name: kopia-platform-creds
  namespace: kopiur-system # lives in the platform namespace
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "REPLACE_ME"
  AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
  KOPIA_PASSWORD: "choose-something-long-and-random"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: ClusterRepository
metadata:
  name: shared-s3 # cluster-scoped: no namespace on the CR itself
spec:
  backend:
    s3:
      bucket: org-kopia-repo
      prefix: "" # bucket root; per-tenant separation comes from identity, not prefix
      region: us-east-1
      auth:
        secretRef:
          name: kopia-platform-creds
          namespace: kopiur-system # explicit ns REQUIRED for cluster scope
  encryption:
    passwordSecretRef:
      name: kopia-platform-creds
      namespace: kopiur-system # explicit ns REQUIRED for cluster scope
      key: KOPIA_PASSWORD
  # Tenancy gate (externally-tagged: list | selector | all).
  allowedNamespaces:
    list:
      - billing
      - payments
  # CEL expressions evaluated at admission, pinned to status. The
  # CEL environment is `namespace`, `policyName`, `labels`, `annotations`.
  identityDefaults:
    hostnameExpr: "namespace"
    usernameExpr: "namespace + '-' + policyName"
  # Maintenance is default-managed. Maintenance is namespaced but a
  # ClusterRepository is not, so `namespace` picks where the managed Maintenance
  # CR lands (defaults to the operator's namespace when omitted).
  maintenance:
    namespace: kopiur-system

Try it end-to-end¶

Prove this backend really takes a backup. The same example file carries a tiny smoke-test: a throwaway PVC, a SnapshotPolicy, and a Snapshot, all pointed at the s3-primary repository above. It takes you from "applied" to "a snapshot in my bucket" in one go.

Fill in the credentials first

The smoke backup only goes green once the REPLACE_ME values in the Secret are real S3 keys. With placeholders the Repository stalls at Failed, because kopia can't reach the bucket, and the Snapshot stays Pending.

1. Apply the bundle. That is the backups namespace, the Secret, the Repository, and the smoke-test objects:

$ kubectl apply -f deploy/examples/backends/s3.yaml

2. Wait for the repository to be Ready. Everything else waits on this:

$ kubectl -n backups wait --for=condition=Ready repository/s3-primary --timeout=2m
repository.kopiur.home-operations.com/s3-primary condition met

3. Take the smoke backup. The Snapshot uses generateName, so create it rather than apply it. The namespace, Secret, Repository, PVC, and policy already exist and report unchanged. The Snapshot is the one new object:

$ kubectl create -f deploy/examples/backends/s3.yaml
snapshot.kopiur.home-operations.com/smoke-now-abc12 created

4. Watch it succeed:

$ kubectl -n backups get snapshots -w
NAME              PHASE       ORIGIN   SNAPSHOT     AGE
smoke-now-abc12   Pending     manual                2s
smoke-now-abc12   Running     manual                7s
smoke-now-abc12   Succeeded   manual   k1f1ec0a8    38s

The output above is illustrative. The Snapshot has no fixed Succeeded condition, so to wait on it in a script, key on the phase:

$ kubectl -n backups wait --for=jsonpath='{.status.phase}'=Succeeded \
    snapshot/smoke-now-abc12 --timeout=5m

5. Prove the data really moved. status.stats shows non-zero bytesNew and filesNew, and status.snapshot.kopiaSnapshotID is the kopia snapshot ID in your bucket:

$ kubectl -n backups get snapshot smoke-now-abc12 -o jsonpath='{.status.stats}'
{"sizeBytes":4096,"bytesNew":1280,"filesNew":2,"filesUnchanged":0}

$ kubectl -n backups get snapshot smoke-now-abc12 -o jsonpath='{.status.snapshot.kopiaSnapshotID}'
k1f1ec0a8

Both outputs are illustrative; sizes and the ID vary. Non-zero bytesNew proves the backup uploaded real content to S3.

6. Clean up the smoke-test when you're done:

$ kubectl -n backups delete snapshot --all       # finalizer also deletes the kopia snapshot
$ kubectl -n backups delete snapshotpolicy smoke
$ kubectl -n backups delete pvc smoke-data

Deleting a Snapshot deletes its snapshot

A produced Snapshot defaults to deletionPolicy: Delete, so removing the CR runs kopia snapshot delete through a finalizer. Use Retain or Orphan to keep the data. See Backups → deletionPolicy.

From here the rest of the lifecycle is the same on every backend. Only the Repository differs. Put it on a cron with a SnapshotSchedule, described in Backups & schedules and Example 01. Restore by picking a Snapshot, described in Restores and Example 03.

Troubleshooting¶

Endpoint scheme / HTTPS assumption

The endpoint is a bare host[:port]. No http:// or https://, and no trailing slash.

kopia assumes HTTPS. For an HTTP-only store set tls.disableTls: true, or you'll see TLS handshake errors against a plaintext port.

Self-signed / private-CA certificates

x509: certificate signed by unknown authority means the endpoint's CA isn't trusted.

Use tls.caBundleRef rather than tls.insecureSkipVerify: true. Skipping verification disables a real protection; the bundle keeps it. If both are set, skip-verify wins, the bundle is ignored, and admission warns.

tls.caBundleRef names a ConfigMap holding the CA PEM. key defaults to ca.crt, and the ConfigMap is resolved in the Repository's namespace, or the operator's namespace for a ClusterRepository.

kopia builds a fresh certificate pool from the bundle, so concatenate a public root into it if a single endpoint's chain needs both. For the full walkthrough see Private-CA HTTPS.

403 or access denied. The key lacks list, write, or delete on the bucket, or the bucket, region, and endpoint don't match. Confirm the key is scoped to this bucket.
repository not initialized. Set create.enabled: true, or the prefix points at an empty location. Watch the Repository phase and events.
Workload identity: the bootstrap or mover Job hits its deadline. The cloud webhook never injected the credential environment variables. Either the ServiceAccount is missing its federation annotation, the IRSA or Pod Identity association doesn't exist, or the identity webhook isn't running. kopia's credential chain then falls back to the EC2 metadata service, which hangs on non-EC2 nodes until the Job deadline. The mover pod log carries an explicit warning naming the missing environment hints. Fix the ServiceAccount's federation and the next run picks it up.