Skip to content

S3 & S3-compatible

The S3 backend works for Amazon S3 and for any S3-compatible object store — MinIO, RustFS, Ceph RGW, Wasabi, Backblaze B2's S3 API, Cloudflare R2, SeaweedFS, and friends. It is the most common choice and the backend used by the canonical first example.

Reach for S3 when your storage is an object store with an S3 API. For Azure Blob, Google Cloud Storage, or native Backblaze B2 there are dedicated backends with nicer ergonomics (Azure, GCS, B2); for a NAS use filesystem or SFTP.

Provider prerequisites

  • A bucket — Kopiur does not create it. Make it in the provider console / CLI first.
  • An access key (an AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY pair) with read / write / list / delete on that bucket. Prefer a key scoped to the one bucket over a root/admin key.
  • For AWS: the bucket's region. For a compatible store: its endpoint host (a bare host[:port], no https:// scheme).

Minimal AWS IAM policy for the key

kopia needs to list, read, write, and delete objects under the repository prefix. A bucket-scoped policy that grants exactly that:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
            "Resource": "arn:aws:s3:::my-backups"
        },
        {
            "Effect": "Allow",
            "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
            "Resource": "arn:aws:s3:::my-backups/*"
        }
    ]
}

s3:DeleteObject is not optional — retention and maintenance delete expired blobs. A read-write-but-no-delete key makes snapshots succeed and maintenance fail later, which is confusing to debug. MinIO and most compatible stores accept the same policy document.

The Secret shape

The mover loads this Secret with envFrom, so every key below must be a valid environment-variable name and reaches kopia as an env var.

Secret key Required What it is
AWS_ACCESS_KEY_ID yes Access-key ID for the bucket.
AWS_SECRET_ACCESS_KEY yes Secret access key paired with the ID.
AWS_SESSION_TOKEN no Only for temporary STS credentials. Omit for a static key.
KOPIA_PASSWORD yes The repository encryption password (every backend needs it). 
stringData:
    AWS_ACCESS_KEY_ID: "REPLACE_ME"
    AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
    KOPIA_PASSWORD: "choose-something-long-and-random"

Lose the password, lose the backups

KOPIA_PASSWORD encrypts the repository; kopia cannot decrypt without it and there is no recovery. Use a long random value, store it outside the cluster, and back up the Secret itself. See Encryption and repository creation.

The Repository

---
apiVersion: v1
kind: Secret
metadata:
  name: s3-repo-creds
  namespace: backups
type: Opaque
stringData:
  # Read by well-known names. AWS_SESSION_TOKEN is optional (STS only).
  AWS_ACCESS_KEY_ID: "REPLACE_ME"
  AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
  # The kopia repository encryption password. Lose it and the backups are
  # unrecoverable — store it outside the cluster too.
  KOPIA_PASSWORD: "choose-something-long-and-random"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Repository
metadata:
  name: s3-primary
  namespace: backups
spec:
  backend:
    s3:
      bucket: my-backups # the bucket holding the kopia repository
      prefix: clusters/prod/ # optional; lets several repos share one bucket
      region: us-east-1 # required by AWS; some compatible stores want it too
      # endpoint: OMIT for AWS. For MinIO/RustFS/Ceph set the host, e.g.
      #   endpoint: minio.storage.svc.cluster.local:9000
      #   tls:
      #     disableTls: true        # in-cluster plain HTTP (kopia assumes HTTPS otherwise)
      #     # or, for self-signed HTTPS, prefer a CA bundle over skipping verify:
      #     # caBundleRef: { configMapName: minio-ca, key: ca.crt }
      #     # insecureSkipVerify: true   # last resort
      auth:
        secretRef:
          name: s3-repo-creds
  encryption:
    passwordSecretRef:
      name: s3-repo-creds
      key: KOPIA_PASSWORD
  create:
    enabled: true # initialize the repo if it does not exist yet (default: false)

Fields reference (backend.s3)

Field Required Default Example What it controls
bucket yes — my-backups The bucket holding the kopia repository. Just the name — no s3://, no path.
prefix no bucket root clusters/prod/ Key prefix so several repositories can share one bucket. End it with / or the prefix concatenates into object names.
endpoint no AWS minio.storage.svc:9000 S3 endpoint host for a compatible store. Omit for AWS. Bare host[:port], no scheme, no trailing slash.
region no — us-east-1 Region. Required by AWS and some compatible stores; any string for MinIO.
auth.secretRef no¹ — { name: s3-repo-creds } Names the credential Secret above. Same namespace as the Repository; a ClusterRepository adds namespace:. Mutually exclusive with workloadIdentity.
auth.workloadIdentity.serviceAccountName no¹ — backup-mover Run the mover Jobs as this (user-created, IAM-federated) ServiceAccount instead of using static keys — see Workload identity.
tls.disableTls no false true Talk plain HTTP instead of HTTPS (kopia's --disable-tls). For in-cluster MinIO on a plaintext port.
tls.insecureSkipVerify no false true Skip TLS cert verification (kopia's --disable-tls-verification). Last resort — prefer caBundleRef.
tls.caBundleRef no — { configMapName: ca, key: ca.crt } A ConfigMap key holding a CA bundle (PEM) to trust a self-signed endpoint.

¹ Set exactly one of auth.secretRef or auth.workloadIdentity (webhook-enforced). auth itself may be omitted when the AWS_* keys ride the encryption-password Secret.

Endpoint & region by provider

The same four identifiers cover every S3-compatible store; what changes is which ones you set. ENDPOINT is always a bare host — strip the https:// the provider's console shows you.

Provider endpoint region Notes
Amazon S3 omit the bucket's region kopia derives the endpoint from the region.
MinIO / RustFS (in-cluster) minio.storage.svc.cluster.local:9000 any string (main) Plain HTTP needs tls.disableTls: true — see below.
Cloudflare R2 <accountid>.r2.cloudflarestorage.com auto The account ID is in the R2 dashboard; use an R2 API token's S3 credentials.
Wasabi s3.<region>.wasabisys.com matches the endpoint (us-east-1) Region must agree with the endpoint host.
Backblaze B2 (S3 API) s3.<region>.backblazeb2.com matches the endpoint (us-west-004) Or use the native B2 backend — same storage, application-key auth.
Ceph RGW rgw.ceph.svc:7480 (your RGW service) your zonegroup (often default) Self-signed HTTPS → tls.caBundleRef.

A filled-in stanza for, say, Cloudflare R2 differs from the AWS example only in these identifiers:

backend:
    s3:
        bucket: my-backups
        endpoint: 0123456789abcdef.r2.cloudflarestorage.com
        region: auto
        auth:
            secretRef:
                name: s3-repo-creds # AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY from the R2 API token

Customization — the values you actually change

  • bucket / prefix — where snapshots land. Multiple repos can share a bucket via distinct prefixes.
  • endpoint + tls — set these for any non-AWS store. See the MinIO variant below.
  • region — set for AWS; any value for MinIO/RustFS.
  • create.enabled — true initializes the repository if it doesn't exist; false (default) requires it to already exist. create.encryption / splitter / hash are fixed forever at creation — see repository creation.
  • moverDefaults.cache — sizing for the mover's kopia cache PVC (movers).

In-cluster MinIO / RustFS over plain HTTP

kopia's S3 path assumes HTTPS. For a plain-HTTP in-cluster endpoint set tls.disableTls: true and give a bare host (no scheme). For a self-signed HTTPS endpoint, prefer pointing tls.caBundleRef at a ConfigMap holding the CA over tls.insecureSkipVerify: true.

# Backend: S3 — in-cluster MinIO / RustFS over plain HTTP
#
# A variant of the S3 backend for an S3-compatible store reached over plain HTTP
# (e.g. a MinIO/RustFS Service inside the cluster). kopia's S3 path assumes HTTPS,
# so you must opt into plain HTTP with `tls.disableTls: true`. The endpoint is a
# bare host[:port] — NO scheme, NO trailing slash.
#
# Field shapes verified against crates/api: externally-tagged backend
# (`backend.s3`); `tls.disableTls` maps to kopia's `--disable-tls`.
---
apiVersion: v1
kind: Secret
metadata:
  name: minio-repo-creds
  namespace: backups
type: Opaque
stringData:
  # MinIO/RustFS access key pair (the "root" user, or a scoped service account).
  AWS_ACCESS_KEY_ID: "REPLACE_ME"
  AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
  KOPIA_PASSWORD: "choose-something-long-and-random"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Repository
metadata:
  name: minio-primary
  namespace: backups
spec:
  backend:
    s3:
      bucket: kopia # the bucket holding the repository (create it in MinIO first)
      prefix: clusters/prod/ # optional; lets several repos share one bucket
      # Bare host[:port] of the in-cluster Service — no http:// scheme.
      endpoint: minio.storage.svc.cluster.local:9000
      region: us-east-1 # MinIO ignores it but kopia wants a value; any string works
      tls:
        disableTls: true # talk plain HTTP (kopia otherwise assumes HTTPS)
      auth:
        secretRef:
          name: minio-repo-creds
  encryption:
    passwordSecretRef:
      name: minio-repo-creds
      key: KOPIA_PASSWORD
  create:
    enabled: true # initialize the repo if it does not exist yet

Workload identity (IRSA / EKS Pod Identity)

On EKS (or any cluster with AWS IAM federation), you can drop the static AWS_* keys entirely: set auth.workloadIdentity.serviceAccountName and every mover Job for this repository runs as that ServiceAccount, with kopia resolving credentials through the ambient AWS chain — the IRSA web-identity token, an EKS Pod Identity association, or instance metadata. The only secret left in the cluster is KOPIA_PASSWORD.

What you provide:

  1. An IAM role with the bucket policy above.
  2. A ServiceAccount federated to it — IRSA's eks.amazonaws.com/role-arn annotation, or an EKS Pod Identity association. It must exist in every namespace mover Jobs run in (the repository's own namespace, plus each consumer namespace for a ClusterRepository).
  3. auth.workloadIdentity.serviceAccountName on the backend, instead of auth.secretRef.

What Kopiur does: it checks the ServiceAccount exists before every run (a missing SA surfaces as CredentialsAvailable=False naming it, instead of a Job that never schedules), binds the kopiur-mover role to it (the mover patches its own CR status at runtime), and — since kopia's CLI insists on its access-key flags — invokes kopia with explicitly empty --access-key= / --secret-access-key= so the AWS credential chain takes over.

Kopiur never touches your ServiceAccount

The SA's annotations are your federation contract with AWS; Kopiur only gets it and adds a RoleBinding (kopiur-mover-wi-<sa>). Create and manage the SA yourself (or via your GitOps tooling).

# Backend: S3 with workload identity — IRSA / EKS Pod Identity
#
# No static AWS keys anywhere: the mover Jobs run as a ServiceAccount federated
# to an IAM role, and kopia resolves credentials through the ambient AWS chain
# (IRSA web-identity token or EKS Pod Identity). The only secret left in the
# cluster is the kopia repository encryption password.
#
# Prerequisites (AWS side):
#   - an IAM role with the bucket policy (see the S3 docs page), trust-federated
#     to this ServiceAccount via your cluster's OIDC provider (IRSA), OR an EKS
#     Pod Identity association pointing at it.
#
# Field shapes verified against crates/api: externally-tagged backend
# (`backend.s3`); `auth.workloadIdentity` is mutually exclusive with
# `auth.secretRef` (webhook-enforced).
---
# You own this ServiceAccount — Kopiur preflights it and binds the mover role to
# it, but never creates or modifies it (the annotation is your contract with the
# cloud's identity webhook). It must exist in every namespace mover Jobs run in.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: backup-mover
  namespace: backups
  annotations:
    # IRSA: the IAM role this SA may assume. For EKS Pod Identity, omit the
    # annotation and create a pod-identity association instead.
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/kopia-backups
---
apiVersion: v1
kind: Secret
metadata:
  name: s3-wi-repo-creds
  namespace: backups
type: Opaque
stringData:
  # Workload identity replaces the AWS_* keys — but the repository encryption
  # password is kopia's, not the cloud's, and is still required.
  KOPIA_PASSWORD: "choose-something-long-and-random"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: Repository
metadata:
  name: s3-wi
  namespace: backups
spec:
  backend:
    s3:
      bucket: my-backups
      region: us-east-1 # the bucket's region (kopia derives the AWS endpoint)
      auth:
        # Instead of secretRef: the mover Jobs run AS this ServiceAccount and
        # kopia authenticates via the ambient AWS credential chain.
        workloadIdentity:
          serviceAccountName: backup-mover
  encryption:
    passwordSecretRef:
      name: s3-wi-repo-creds
      key: KOPIA_PASSWORD
  create:
    enabled: true

Replication: don't mix static and workload-identity S3 pairs

A RepositoryReplication between two S3 backends must use the same auth style on both sides (or both workloadIdentity with the same ServiceAccount). A mixed pair is rejected at admission: the replication pod's environment carries the static side's AWS_* keys, and the workload-identity side's credential chain would silently pick them up — authenticating as the wrong identity. Cross-cloud pairs (e.g. S3 → GCS) are unaffected.

As a ClusterRepository

The same backend.s3 stanza works on a cluster-scoped ClusterRepository, with two differences: every Secret reference must carry an explicit namespace: (webhook-enforced), and the credential Secret must exist in the namespaces the movers run in — see Movers → where the credential Secret lives.

# Backend: S3 as a shared ClusterRepository
#
# The SAME S3 backend stanza, but on a cluster-scoped ClusterRepository owned by a
# platform team and referenced by allow-listed tenant namespaces. Two things differ
# from a namespaced Repository:
#   1. EVERY Secret reference must carry an explicit `namespace:` (webhook-enforced
#      — a cluster-scoped CR has no namespace of its own to default to).
#   2. `allowedNamespaces` gates which namespaces may reference it, and
#      `identityDefaults` derives each consumer's kopia identity from CEL
#      expressions evaluated at admission.
#
# REQUIRES the operator installed with installScope=cluster.
#
# Field shapes verified against crates/api: externally-tagged backend
# (`backend.s3`) and `allowedNamespaces` (list | selector | all).
---
apiVersion: v1
kind: Secret
metadata:
  name: kopia-platform-creds
  namespace: kopiur-system # lives in the platform namespace
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "REPLACE_ME"
  AWS_SECRET_ACCESS_KEY: "REPLACE_ME"
  KOPIA_PASSWORD: "choose-something-long-and-random"
---
apiVersion: kopiur.home-operations.com/v1alpha1
kind: ClusterRepository
metadata:
  name: shared-s3 # cluster-scoped: no namespace on the CR itself
spec:
  backend:
    s3:
      bucket: org-kopia-repo
      prefix: "" # bucket root; per-tenant separation comes from identity, not prefix
      region: us-east-1
      auth:
        secretRef:
          name: kopia-platform-creds
          namespace: kopiur-system # explicit ns REQUIRED for cluster scope
  encryption:
    passwordSecretRef:
      name: kopia-platform-creds
      namespace: kopiur-system # explicit ns REQUIRED for cluster scope
      key: KOPIA_PASSWORD
  # Tenancy gate (externally-tagged: list | selector | all).
  allowedNamespaces:
    list:
      - billing
      - payments
  # CEL expressions evaluated at admission, pinned to status. The
  # CEL environment is `namespace`, `policyName`, `labels`, `annotations`.
  identityDefaults:
    hostnameExpr: "namespace"
    usernameExpr: "namespace + '-' + policyName"
  # Maintenance is default-managed. Maintenance is namespaced but a
  # ClusterRepository is not, so `namespace` picks where the managed Maintenance
  # CR lands (defaults to the operator's namespace when omitted).
  maintenance:
    namespace: kopiur-system

Try it end-to-end

Prove this backend really takes a backup. The same example file carries a tiny smoke-test (a throwaway PVC + a SnapshotPolicy + a Snapshot) that targets the s3-primary repository above, so you can go from "applied" to "a snapshot in my bucket" in one arc.

Fill in the credentials first

The smoke backup only goes green once the REPLACE_ME values in the Secret are real S3 keys. With placeholders the Repository stalls at Failed (kopia can't reach the bucket) and the Snapshot stays Pending.

1. Apply the bundle (namespace backups, Secret, Repository, and the smoke-test objects):

$ kubectl apply -f deploy/examples/backends/s3.yaml

2. Wait for the repository to be Ready — the gate everything else waits on:

$ kubectl -n backups wait --for=condition=Ready repository/s3-primary --timeout=2m
repository.kopiur.home-operations.com/s3-primary condition met

3. Take the smoke backup. The Snapshot uses generateName, so create it (the namespace, Secret, Repository, PVC, and policy already exist and report unchanged — the Snapshot is the one new object):

$ kubectl create -f deploy/examples/backends/s3.yaml
snapshot.kopiur.home-operations.com/smoke-now-abc12 created

4. Watch it succeed:

$ kubectl -n backups get snapshots -w
NAME              PHASE       ORIGIN   SNAPSHOT     AGE
smoke-now-abc12   Pending     manual                2s
smoke-now-abc12   Running     manual                7s
smoke-now-abc12   Succeeded   manual   k1f1ec0a8    38s

(Output illustrative.) The Snapshot has no fixed Succeeded condition; to wait on it in a script, key on the phase:

$ kubectl -n backups wait --for=jsonpath='{.status.phase}'=Succeeded \
    snapshot/smoke-now-abc12 --timeout=5m

5. Deep proof — the data really moved. status.stats shows non-zero bytesNew/filesNew, and status.snapshot.kopiaSnapshotID is the kopia snapshot ID in your bucket:

$ kubectl -n backups get snapshot smoke-now-abc12 -o jsonpath='{.status.stats}'
{"sizeBytes":4096,"bytesNew":1280,"filesNew":2,"filesUnchanged":0}

$ kubectl -n backups get snapshot smoke-now-abc12 -o jsonpath='{.status.snapshot.kopiaSnapshotID}'
k1f1ec0a8

(Both outputs illustrative — sizes and the ID vary.) Non-zero bytesNew is the proof the backup uploaded real content to S3.

6. Clean up the smoke-test when you're done:

$ kubectl -n backups delete snapshot --all       # finalizer also deletes the kopia snapshot
$ kubectl -n backups delete snapshotpolicy smoke
$ kubectl -n backups delete pvc smoke-data

Deleting a Snapshot deletes its snapshot

A produced Snapshot defaults to deletionPolicy: Delete, so removing the CR runs kopia snapshot delete via a finalizer. Use Retain (or Orphan) to keep the data — see Backups → deletionPolicy.

From here the full lifecycle is backend-independent — only the Repository differs. Put it on a cron with a SnapshotSchedule (Backups & schedules, Example 01) and restore by picking a Snapshot (Restores, Example 03).

Troubleshooting

Endpoint scheme / HTTPS assumption

The endpoint is a bare host[:port] — no http:///https:// and no trailing slash. kopia assumes HTTPS; for an HTTP-only store set tls.disableTls: true, or you'll see TLS handshake errors against a plaintext port.

Self-signed certificates

Prefer tls.caBundleRef (a ConfigMap with the CA PEM) over tls.insecureSkipVerify: true. Skipping verification disables a real protection; trusting the CA keeps it.

  • 403 / access denied — the key lacks list/write/delete on the bucket, or the bucket/region/endpoint don't match. Confirm the key is scoped to this bucket.
  • repository not initialized — set create.enabled: true, or the prefix points at an empty location. Watch the Repository Phase and Events.
  • Workload identity: the bootstrap/mover Job hits its deadline — the cloud webhook never injected the credential env (the SA is missing its federation annotation, the IRSA/Pod Identity association doesn't exist, or the identity webhook isn't running), so kopia's credential chain falls back to the EC2 metadata service — which hangs on non-EC2 nodes until the Job deadline. The mover pod log carries an explicit warning naming the missing env hints; fix the ServiceAccount's federation and the next run picks it up.

See also