Installing Kopiur

Kopiur is a Kopia-native Kubernetes backup operator, written in Rust on kube-rs. This guide covers installing the operator with the Helm chart and verifying it.

The chart is published as an OCI artifact at oci://ghcr.io/home-operations/charts/kopiur, and that is the preferred way to install. Every published version is cosign-signed and ships all three component images, controller, webhook and mover, pinned by digest to the exact release build.

The in-repo chart at deploy/helm/kopiur is the development copy. Its image digests are empty and its tags float, so use it only when you are working from a checkout.

Status: alpha. API group kopiur.home-operations.com, version v1alpha1. The CRD surface may still change between releases.

Prerequisites

• Kubernetes 1.24 or later. The deploy-or-restore volume-populator path, meaning a Restore plus a PVC spec.dataSourceRef, relies on the AnyVolumeDataSource feature, available from 1.24.
• Helm 3 or 4.
• A kopia repository backend you can reach: S3 or MinIO, Azure Blob, GCS, B2, filesystem on a PVC, SFTP, WebDAV, or rclone.
• A CSI snapshot stack, meaning a snapshot-controller plus a VolumeSnapshotClass for your driver, for the default copyMethod: Snapshot. Many distributions bundle it, including EKS, GKE, AKS, Talos and k3s add-ons. Where yours does not, install the home-operations snapshot-controller chart from oci://ghcr.io/home-operations/charts/snapshot-controller. You do not need it if every SnapshotPolicy sets copyMethod: Direct. See Copy methods → What it requires.
• (Optional) cert-manager, only if you prefer it to manage the admission webhook’s certificate. It is not required: by default the operator manages the webhook cert itself. See Webhook TLS.
• (Optional) volume-data-source-validator, recommended alongside CSI populators so a malformed dataSourceRef surfaces as an event rather than a PVC that is quietly stuck.
• (Optional) Prometheus Operator, if you want the chart’s ServiceMonitor.

Quickstart

# 1. Install the published chart, creating the operator namespace. No extra
#    flags needed — the webhook cert is self-managed by default (no
#    cert-manager required). Add --version x.y.z to pin a release.
helm install kopiur oci://ghcr.io/home-operations/charts/kopiur \
  --namespace kopiur-system --create-namespace

# 2. Wait for rollout. (The webhook pod stays in ContainerCreating until the
#    controller mints its serving Secret — a few seconds after the controller
#    becomes ready.)
kubectl -n kopiur-system rollout status deploy/kopiur-controller
kubectl -n kopiur-system rollout status deploy/kopiur-webhook

# 3. Confirm the 9 CRDs are registered.
kubectl get crd | grep kopiur.home-operations.com

GitOps install (Flux / Argo)

Point your GitOps tool at the same OCI chart. Never point it at the in-repo deploy/helm/kopiur path, whose image digests are empty and whose tags float: a git-sourced install renders image tags from the chart’s appVersion, so a checkout can pull a tag that was never published. The published chart pins all three images by digest.

Flux

apiVersion: v1
kind: Namespace
metadata:
  name: kopiur-system
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: kopiur
  namespace: kopiur-system
spec:
  interval: 1h
  url: oci://ghcr.io/home-operations/charts/kopiur
  ref:
    tag: 0.8.0 # pin the chart version
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kopiur
  namespace: kopiur-system
spec:
  interval: 1h
  chartRef:
    kind: OCIRepository
    name: kopiur
  # values: {}   # override chart values here

The OCIRepository and HelmRelease are namespaced, so kopiur-system must exist before they apply. That is why the Namespace is included above: Flux’s install.createNamespace only creates the release’s target namespace, not the one the HelmRelease object itself lives in. In a real Flux repository the namespace usually comes from the parent Kustomization; it is kept here so the snippet applies on its own.

Argo CD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kopiur
  namespace: argocd
spec:
  project: default
  source:
    repoURL: ghcr.io/home-operations/charts # no oci:// prefix here
    chart: kopiur
    targetRevision: 0.8.0
    # helm:
    #   valuesObject: {}   # override chart values here
  destination:
    server: https://kubernetes.default.svc
    namespace: kopiur-system
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

helm upgrade and GitOps reconciles never update the CRDs

The 9 CRDs ship in the chart’s special crds/ directory, which Helm installs once and never touches on upgrade. See CRD lifecycle below.

A GitOps tool with a CreateReplace or server-side-apply CRD policy handles schema changes automatically. Otherwise, apply deploy/crds/ yourself.

Webhook TLS

The admission webhook always serves TLS, and the API server must trust it. webhook.tls.mode chooses how the serving certificate is provisioned:

webhook.tls.mode	What happens	Needs cert-manager?
self (default)	The operator mints its own CA + serving cert, writes the Secret, injects the caBundle, and auto-rotates the cert (zero-downtime hot-reload).	No
cert-manager	cert-manager issues the cert; its ca-injector populates the caBundle.	Yes
manual	You pre-create the Secret and supply webhook.caBundle.	No

self (default)

There is nothing to configure; this is the helm install above. The operator grants itself the minimal extra RBAC it needs: writing its one serving Secret, and patching the caBundle of its two webhook configurations, scoped to those resource names.

cert-manager

helm install kopiur oci://ghcr.io/home-operations/charts/kopiur \
  --namespace kopiur-system \
  --set webhook.tls.mode=cert-manager
# Optionally point at your own Issuer/ClusterIssuer instead of the chart's
# self-signed one: --set webhook.certManager.issuerRef.name=my-issuer

manual

# create a kubernetes.io/tls Secret named per webhook.tls.secretName, then:
helm install kopiur oci://ghcr.io/home-operations/charts/kopiur \
  --namespace kopiur-system \
  --set webhook.tls.mode=manual \
  --set webhook.tls.secretName=kopiur-webhook-tls \
  --set webhook.caBundle="$(base64 -w0 ca.crt)"

Or turn the webhook off entirely. Validation then relies on the controller’s defensive checks alone, which is not recommended:

helm install kopiur oci://ghcr.io/home-operations/charts/kopiur -n kopiur-system --set webhook.enabled=false

Install scope

Mode	--set installScope=	RBAC	Manages	ClusterRepository
Cluster (default)	cluster	ClusterRole	cluster-wide	reconciled
Namespaced	namespaced	Role	release namespace only	not reconciled

Cluster is the default so that a shared platform repository, a ClusterRepository referenced by many tenant namespaces, works out of the box. A namespace-scoped Role silently disables the cluster-scoped ClusterRepository kind. See deploy/examples/02-cluster-repository.yaml.

Choose namespaced as the explicit least-privilege opt-down for a single-team install.

In namespaced scope the controller’s watches are narrowed to the release namespace, to match the Role-only RBAC; the chart passes --namespace={{ .Release.Namespace }}. Cluster-scoped kinds are skipped entirely: ClusterRepository is not reconciled, and the privileged-movers namespace opt-in check fails open, because the operator is already confined to admin-chosen namespaces there.

The stream-source opt-in fails closed there instead — pods/exec is a far larger grant than an elevated container — so stream sources are refused under a namespaced install until you set rbacNamespaceReadForStreamSources: true, which adds a supplementary ClusterRole granting get on namespaces alone. pvc and nfs sources are unaffected.

Features that need cluster RBAC are refused in namespaced scope

Two features read or write cluster-scoped objects that a Role can never grant: PersistentVolumes, StorageClasses and VolumeSnapshotClasses. A namespaced install refuses them up front with an actionable message rather than retrying forever.

• target.populator restores. Use target.pvc or target.pvcRef, or install with installScope: cluster.
• copyMethod: Snapshot and Clone backups, which need CSI staging. Set copyMethod: Direct on the SnapshotPolicy. Note that copyMethod defaults to Snapshot, so an untouched policy hits this refusal, and the message spells out the fix.

RBAC for the optional web UI

If you use the web UI, meaning spec.server on a Repository or ClusterRepository, the controller manages Deployments, Services, ConfigMaps and Secrets for the kopia server pod. That RBAC ships with the chart in both scopes, so there is nothing extra to grant. The feature stays off until you add a spec.server block.

Private registries

To pull the Kopiur images from a private registry, set imagePullSecrets (or global.imagePullSecrets) to the registry Secret names. They apply to the controller, webhook and console pods, and to every mover pod the operator starts.

Mover pods need the Secret in their own namespace

Movers run in workload and repository namespaces, not only in the operator namespace, and a pod can only use a pull Secret from its own namespace. Kopiur passes the names but does not copy the Secrets. See Movers → private registry for the list of namespaces.

Runtime tuning

The chart’s defaults are sized for a typical homelab to mid-size cluster. A handful of top-level values tune the controller’s runtime footprint and its load on the API server. The full rationale is in the developer docs.

Value	Env var	Default	What it does
workerThreads	KOPIUR_WORKER_THREADS	2	Tokio worker threads. The controller is I/O-bound; raise only for a reconcile-heavy deployment.
streamingLists	KOPIUR_STREAMING_LISTS	true	Stream cluster-wide re-lists via the WatchList API (lower apiserver + controller memory). Auto-downgrades to paged lists when the apiserver answers with a pre-1.32 version; a startup probe that fails to answer keeps the configured value, since a transport failure is not evidence about the server’s version.
reconcileConcurrency	KOPIUR_RECONCILE_CONCURRENCY	8	Per-controller cap on concurrent reconciles. Bounds API-server load and file descriptors during re-list storms and API-server outages. 0 = unbounded (not recommended).
maxConcurrentDeleteJobs	KOPIUR_MAX_CONCURRENT_DELETE_JOBS	0 (no cluster-wide cap)	Opt-in backstop on concurrent snapshot-delete batch Jobs across all repositories; each repository is already limited by spec.concurrency.maxConcurrentDeleteJobs (default 1).
maxConcurrentJobs	KOPIUR_MAX_CONCURRENT_JOBS	0 (uncapped)	Opt-in cluster-wide backstop on concurrent pooled mover Jobs (backups, restores, and the source side of either replication) across all repositories. The per-repository spec.concurrency.maxConcurrentJobs is the primary knob; a run must satisfy both.
leaderElection.flowSchema.enabled	—	true	Give the controller’s leader-election Lease its own API Priority and Fairness lane. See below.

Which of the two job caps to reach for

maxConcurrentJobs here is the cluster operator’s backstop. Set it when the node pool cannot host N movers no matter which repositories they belong to.

Set the per-repository spec.concurrency.maxConcurrentJobs when one backend is the thing that must not be saturated. That is the knob most people want, and it is the one whose numbers appear in a queued run’s status message.

Leaving this one at 0 costs nothing: with no cap set anywhere, the operator makes no extra API calls at all.

Restores are always admitted and never queued, though a running restore still occupies a slot. Maintenance, verification, pin and snapshot-delete Jobs are outside the pool entirely. See Backups → limiting concurrent jobs per repository.

Why the chart ships a FlowSchema

Kubernetes’ built-in system-leader-election FlowSchema routes lease renewals into a guaranteed priority level whose concurrency is never lent away. It matches only kube-controller-manager, kube-scheduler, and ServiceAccounts in kube-system.

An operator in its own namespace therefore falls through to service-accounts and then to workload-low, where its lease renewals queue behind every other ServiceAccount’s bulk traffic in the cluster. On one production cluster the p99 queue wait differed by about 360 times between the two lanes, and the controller was restarting roughly fifteen times a day as a result.

The chart therefore ships a FlowSchema that routes only the operator’s leases get, create and update calls into the built-in leader-election level. Set leaderElection.flowSchema.enabled: false if APF is disabled on your cluster, or if you cannot create cluster-scoped flowcontrol.apiserver.k8s.io objects. Kopiur runs correctly without it; the renew loop tolerates a congested lane, it just has less headroom.

Why reconcileConcurrency is bounded by default

Unbounded reconcile concurrency let an API-server outage exhaust the controller’s file descriptors within seconds, because every re-listed object reconciled, and failed, at once.

The default of 8 per controller clears a re-list of a few hundred objects in seconds while keeping the controller a good citizen toward a struggling control plane.

CRD lifecycle

The 9 CRDs ship in the chart’s special crds/ directory. Helm treats that directory specially: helm install installs the CRDs, but helm upgrade never touches them. There is no toggle for this.

Caution: because helm upgrade skips the crds/ directory, a helm-CLI upgrade that carries a schema change does not apply it. You must apply the new CRDs yourself:

# Server-side apply is required: the SnapshotPolicy CRD embeds a full JobSpec
# (runJob hook) and is too large for the client-side last-applied annotation.
kubectl apply --server-side -f deploy/crds/

A GitOps flow (Flux or Argo with a CreateReplace sync policy) applies CRD changes automatically. If you manage CRDs entirely out of band, just apply deploy/crds/ yourself, since helm install skips the ones that already exist.

Skipping this is usually silent, not loud. When a release adds a new spec field, an apiserver still running the old schema prunes that field out of every object you apply. The object admits cleanly, kubectl get -o yaml shows no trace of the field, and the feature you configured simply never happens. If a newly documented field seems to be ignored, check the live CRD before anything else with kubectl get crd <plural>.kopiur.home-operations.com -o yaml | grep <field>. kubectl kopiur doctor flags known-stale schemas too.

The CRDs and RBAC shipped by the chart are generated by cargo xtask gen-crds and cargo xtask gen-rbac, and checked in under deploy/crds/ and deploy/rbac/. Those xtasks are the source of truth.

Upgrading from 0.5.x to 0.6.0 is a special case. The CRDs used to be release-owned templates and moved into this crds/ directory, so the crossing removes and re-installs them, which cascade-deletes your custom resources, unless you pin them first. Read Upgrading → 0.5.x → 0.6.0 before you upgrade.

First backup

After install, create a repository and start backing up a PVC. The smallest end-to-end example is deploy/examples/01-single-pvc-scheduled.yaml:

kubectl apply -f deploy/examples/01-single-pvc-scheduled.yaml
kubectl get repositories,snapshotpolicies,snapshotschedules -n billing

The first eight examples in deploy/examples/ cover the common patterns. The full numbered ladder is on the Examples page.

File	Pattern
01-single-pvc-scheduled.yaml	Single PVC, scheduled daily
02-cluster-repository.yaml	Shared platform ClusterRepository (cluster scope)
03-restore-by-backup.yaml	Restore by picking a Snapshot
04-multi-pvc-selector.yaml	Multi-PVC label selector + group snapshot
05-deploy-or-restore-gitops.yaml	Deploy-or-restore (PVC dataSourceRef)
06-manual-backup.yaml	Manual one-shot Snapshot
07-restore-discovered.yaml	Restore a discovered / foreign snapshot
08-maintenance.yaml	kopia maintenance schedule + ownership lease

Observability

• The controller serves /metrics, /healthz and /readyz on its single port, set by metrics.port and defaulting to :8081. The webhook serves /metrics on its TLS port, set by webhook.port and defaulting to :8443. All metrics are under the kopiur_ namespace.
• The controller and webhook default to the dual-stack wildcard bind, [::]:8081 and [::]:8443, which serves both IPv4 and IPv6 kubelets and API servers, so probes work out of the box on either. The chart renders [::]:<port> into KOPIUR_HTTP_ADDR for the controller and KOPIUR_WEBHOOK_ADDR for the webhook. On a host where IPv6 is disabled in the pod network namespace a [::] bind fails outright; force an IPv4-only bind by adding KOPIUR_HTTP_ADDR=0.0.0.0:8081 through the controller’s extraEnv. See Helm chart values → Controller Deployment for the full detail.
• The controller’s metrics Service is always created. The :8081 listener hosts /metrics alongside the probes, so there is nothing to disable.
• monitoring.serviceMonitor.enabled=true creates a Prometheus-Operator ServiceMonitor, which requires the Prometheus-Operator CRDs. monitoring.prometheusRule.enabled=true ships the kopiur alert rules.
• monitoring.dashboards.enabled=true ships the Grafana dashboard as a sidecar-discoverable ConfigMap, sourced from deploy/dashboards/kopiur.json. Set monitoring.dashboards.grafanaOperator.enabled=true to render a grafana-operator GrafanaDashboard object from the same JSON instead, and use monitoring.dashboards.grafanaOperator.matchLabels to select the Grafana instance.
• observability.otlp.enabled=true, with observability.otlp.endpoint set, also exports OTLP traces, logs and a metrics push from the controller, the webhook and mover Jobs. It is off by default.

Turn it all on with the ready-made overlay:

helm upgrade kopiur oci://ghcr.io/home-operations/charts/kopiur -n kopiur-system \
  -f deploy/observability-values.yaml

See Observability for the full metric list, OTLP details, and a sample collector config.

The web console (optional)

Kopiur ships an optional in-cluster web console, kopiur-ui, off by default. It needs three things, and refuses to render without them:

1. installScope: cluster — impersonation and the console’s cluster-wide reads cannot be expressed in a namespaced Role.
2. An identity source — the header your authenticating proxy sets, or an explicit anonymous identity.
3. In header mode, a proxy shared secret, so the console can tell your proxy’s requests from anything else in the cluster that can reach its Service.

The chart renders the console, its Service and its roles. It does not render an Ingress, and it binds nobody unless you list them in ui.rbac.bindings, so a freshly-enabled console reports Forbidden on every screen until you bind someone — which is the correct first experience, not a broken install.

# You create the shared token first; the chart only mounts it.
kubectl -n kopiur-system create secret generic kopiur-ui-proxy-token \
  --from-literal=token="$(openssl rand -hex 32)"

helm upgrade kopiur oci://ghcr.io/home-operations/charts/kopiur -n kopiur-system \
  --set ui.enabled=true \
  --set ui.auth.userHeader=X-Forwarded-User \
  --set ui.auth.groupsHeader=X-Forwarded-Groups \
  --set ui.auth.proxySecret.existingSecret=kopiur-ui-proxy-token \
  --set 'ui.rbac.bindings[0].role=user' \
  --set 'ui.rbac.bindings[0].groups[0]=platform'

The last two lines grant the platform group full use of the console. The header names are oauth2-proxy’s. Behind Envoy Gateway, use the names from your claimToHeaders instead and also set ui.auth.groupsFormat=base64Json — Envoy base64-encodes the groups claim. With Envoy Gateway has the complete wiring.

Read Web console (kopiur-ui) before exposing it: Identity and access walks through getting a group from your IdP to a role step by step, and the page also covers the trust model, the four roles, what granting the file browser really hands over, and worked oauth2-proxy and Envoy Gateway deployments.

Upgrade / uninstall

helm upgrade kopiur oci://ghcr.io/home-operations/charts/kopiur -n kopiur-system   # does NOT touch crds/ — apply schema changes yourself (see CRD lifecycle)
helm uninstall kopiur -n kopiur-system                                             # leaves the crds/ CRDs (and your CRs) in place

Upgrading from 0.5.x needs a one-time pre-step, so the CRD move does not delete your resources. See Upgrading.

See also

• Every chart value, walked through: Helm chart values
• The optional web console, and who it lets in: Web console (kopiur-ui)
• Chart values & modes: deploy/helm/kopiur/README.md