Backup and restore

QuestDB Enterprise schedules and performs backups on the primary. The operator configures the engine and observes backups(); it does not upload data, create a Kubernetes CronJob, expose an on-demand backup API, or read the object store. Status can lag the database by roughly the manager's resync/observation interval (about two minutes).

For more information about backup and restore mechanics, please refer to the official documentation.

Before running any command, replace every <angle-bracket> value; an unreplaced placeholder can be interpreted as shell redirection.

Configure and verify backup

Before you start

Create and validate a same-namespace QuestDBObjectStore as described in Configuration. The object-store CR has no readiness status; proof comes from a consuming database.

Use a prefix reserved for this cluster:

spec:
  objectStoreRef:
    name: <store-name>
  backup:
    enabled: true
    schedule: "0 * * * *"
    timezone: UTC
    retention: 5
    root: backup/<namespace>/<name>/
    stalledAfterSeconds: 3600

Apply the cluster change. It can recreate the primary to install the new configuration, briefly interrupting writes.

Watch

Before the first run, BackupHealthy may be Unknown with reason NoBackupYet. The bounded deadline must exceed the schedule interval plus the roughly two-minute observation delay; this hourly example allows about 75 minutes:

for _ in $(seq 1 450); do
  STATUS="$(kubectl get questdbcluster <name> -n <namespace> \
    -o jsonpath='{.status.backup.lastBackup.status}')"
  REASON="$(kubectl get questdbcluster <name> -n <namespace> \
    -o jsonpath='{.status.conditions[?(@.type=="BackupHealthy")].reason}')"
  [ "$STATUS" = "completed" ] && break
  [ "$STATUS" = "failed" ] && break
  [ "$REASON" = "Stalled" ] && break
  sleep 10
done
kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{range .status.conditions[?(@.type=="BackupHealthy")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}{.status.backup.lastBackup}{"\nlastProgressAt="}{.status.backup.lastProgressAt}{"\n"}'
[ "$STATUS" = "completed" ]

Verify both the first successful run and continued recent successful runs. A failed engine run has lastBackup.status=failed and BackupHealthy=False/Failed; inspect .status.backup.lastBackup.error, database logs, Secret metadata, cloud IAM, DNS, and network access.

A stalled run is deliberately different: the engine status remains in_progress, while BackupHealthy=False/Stalled reports that progressPercent has not changed for the configured threshold. .status.backup.lastProgressAt is when the operator first observed the current run or most recently observed its percentage change. stalledAfterSeconds defaults to 3600 seconds. Setting it explicitly to 0 disables stall detection; it does not cancel the backup, mark it failed, or bound how long in_progress can remain. Diagnose and remediate the engine/store path rather than editing status.

There is no safe "force backup" command. Temporarily shortening spec.backup.schedule is a schedule/configuration change, not an on-demand backup. It can recreate pods and briefly interrupt writes; if used for a controlled test, restore the production schedule afterward.

Restore into a new cluster

Restore always creates a new QuestDBCluster. Its immutable spec.bootstrap.recovery makes the genesis primary start from backup or not start at all.

The source QuestDBObjectStore must exist in the destination cluster's namespace. A missing source CR safely withholds genesis and retries. Invalid store configuration or an engine/runtime restore problem fails recovery instead of initializing an empty database.

Before you start

• identify the source store and backup root;
• choose a destination name, PVCs, and backup/WAL prefixes that cannot collide with the source or another live cluster;
• for a multi-instance source, determine sourceInstanceName and copy it exactly; it must match the lowercase hyphen-separated pattern ^[a-z0-9]+(-[a-z0-9]+)*$;
• for PITR, confirm the retained time window before choosing a target.

For an operator-managed source, the current seed identity is normally published here:

kubectl get questdbcluster <source-name> -n <namespace> \
  -o jsonpath='{.status.replication.seed.backupInstanceName}{"\n"}'

Confirm it against the source database when possible:

SELECT backup_instance_name();

Set spec.bootstrap.recovery.sourceInstanceName when the source store holds backups from more than one instance. If omitted, the engine selects the only instance in a single-instance source and fails when selection is ambiguous. The operator cannot list the store for you.

Change

Copy the working tenant cluster's spec.image and spec.imagePullSecrets. Set <questdb-enterprise-image> to that private image and <tenant-image-pull-secret> to the pull Secret in the destination namespace. Remove the entire imagePullSecrets block only when every destination node has ambient pull access, such as an authorized EKS worker-node role.

apiVersion: questdb.io/v1alpha1
kind: QuestDBCluster
metadata:
  name: <restored-name>
  namespace: <namespace>
spec:
  image: <questdb-enterprise-image>
  imagePullSecrets:
    - name: <tenant-image-pull-secret>
  storage:
    storageClassName: <storage-class>
    size: 100Gi
  resources:
    requests:
      memory: 4Gi
    limits:
      memory: 4Gi
  objectStoreRef:
    name: <destination-store>
  bootstrap:
    recovery:
      source:
        objectStoreRef:
          name: <source-store>
          root: backup/<source-namespace>/<source-name>/
      sourceInstanceName: <source-backup-instance-name>
  backup:
    enabled: true
    schedule: "0 * * * *"
    root: backup/<namespace>/<restored-name>/
  # Optional tuning block. If set, this root is immutable.
  replication:
    root: db/<namespace>/<restored-name>/

Apply the reviewed manifest:

kubectl apply -f <restore-file.yaml>

The destination backup and replication prefixes must be distinct from the source/live cluster's prefixes. When omitted, the destination replication root defaults to the identity-scoped db/<namespace>/<restored-name>/; confirm that identity is unique.

Watch and verify

GENERATION="$(kubectl get questdbcluster <restored-name> -n <namespace> \
  -o jsonpath='{.metadata.generation}')"
for _ in $(seq 1 180); do
  OBSERVED="$(kubectl get questdbcluster <restored-name> -n <namespace> \
    -o jsonpath='{.status.observedGeneration}')"
  RECOVERED="$(kubectl get questdbcluster <restored-name> -n <namespace> \
    -o jsonpath='{.status.conditions[?(@.type=="Recovered")].status}')"
  FAILED="$(kubectl get questdbcluster <restored-name> -n <namespace> \
    -o jsonpath='{.status.conditions[?(@.type=="RecoveryFailed")].status}')"
  AVAILABLE="$(kubectl get questdbcluster <restored-name> -n <namespace> \
    -o jsonpath='{range .status.conditions[?(@.type=="Available")]}{.status} {.reason}{end}')"
  PROGRESSING="$(kubectl get questdbcluster <restored-name> -n <namespace> \
    -o jsonpath='{range .status.conditions[?(@.type=="Progressing")]}{.status} {.reason}{end}')"
  WRITE_HEALTHY="$(kubectl get questdbcluster <restored-name> -n <namespace> \
    -o jsonpath='{range .status.conditions[?(@.type=="WriteHealthy")]}{.status} {.reason}{end}')"
  if [ "$OBSERVED" = "$GENERATION" ] && \
     [ "$RECOVERED" = "True" ] && [ "$FAILED" != "True" ] && \
     [ "$AVAILABLE" = "True PrimaryReady" ] && \
     [ "$PROGRESSING" = "False Settled" ] && \
     [ "$WRITE_HEALTHY" = "True Healthy" ]; then
    break
  fi
  [ "$FAILED" = "True" ] && break
  sleep 10
done
kubectl get questdbcluster <restored-name> -n <namespace> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}'
[ "$OBSERVED" = "$GENERATION" ] && \
[ "$RECOVERED" = "True" ] && [ "$FAILED" != "True" ] && \
[ "$AVAILABLE" = "True PrimaryReady" ] && \
[ "$PROGRESSING" = "False Settled" ] && \
[ "$WRITE_HEALTHY" = "True Healthy" ]

Recovered=True means the engine reported restore completion; the other gates establish current ordinary writer readiness, including observed WAL write health. They still do not prove that you selected the intended data or guarantee free disk capacity. Before sending traffic, query the restored database and validate critical tables, minimum/maximum timestamps, expected row counts, application invariants, and storage headroom.

If it fails

Read Recovered/RecoveryFailed, then inspect the genesis pod and its recovery init container:

kubectl get pods -n <namespace> -l questdb.io/cluster=<restored-name> -o wide
kubectl describe questdbcluster <restored-name> -n <namespace>
kubectl logs -n <namespace> <restored-name>-1 -c recovery-bootstrap --tail=500
kubectl logs -n <namespace> <restored-name>-1 -c questdb --tail=500

Container names can be confirmed with kubectl get pod <restored-name>-1 -n <namespace> -o jsonpath='{.spec.initContainers[*].name}{" "}{.spec.containers[*].name}{"\n"}'.

Because spec.bootstrap is immutable, correct a wrong source, identity, root, or target in a new cluster. A live failed destination CR continues reconciling and can race cleanup or recreate its pods, so remove it before touching retained storage:

CLEANUP_READY=false
if kubectl delete questdbcluster <restored-name> -n <namespace> --timeout=5m; then
  for _ in $(seq 1 120); do
    if PODS="$(kubectl get pods -n <namespace> \
      -l questdb.io/cluster=<restored-name> -o name)" && [ -z "$PODS" ]; then
      CLEANUP_READY=true
      break
    fi
    sleep 5
  done
fi
[ "${CLEANUP_READY:-false}" = "true" ] &&
  kubectl get pvc -n <namespace> -l questdb.io/cluster=<restored-name> -o wide

Proceed only if the PVC inventory command ran successfully. Confirm the listed PVC belongs only to the failed destination and contains no needed data.

Deleting the retained destination PVC permanently destroys its

incomplete restore data. Delete only the PVC you have confirmed is disposable.

In the same shell, delete only the confirmed PVC:

[ "${CLEANUP_READY:-false}" = "true" ] &&
  kubectl delete pvc <confirmed-disposable-pvc> -n <namespace> --timeout=5m

Apply the corrected restore with a fresh cluster name, which creates a distinct fresh PVC. Do not reuse <restored-name> or its old PVC.

Point-in-time recovery (PITR)

PITR selects the newest retained backup at or before an RFC3339 target. Its granularity is the backup cadence, not continuous WAL time. The operator cannot inspect the object store to pre-validate the recoverable window.

Kubernetes accepts RFC3339 offsets and fractional seconds; the operator normalizes the target to UTC Z and truncates fractional seconds to six digits rather than rounding, preserving the at-or-before boundary.

Add this immutable block when creating the restored cluster:

spec:
  bootstrap:
    recovery:
      source:
        objectStoreRef:
          name: <source-store>
          root: backup/<source-namespace>/<source-name>/
      sourceInstanceName: <source-backup-instance-name>
      recoveryTarget:
        timestamp: "2026-06-30T14:00:00Z"

If the target is older than the retained window, supported QuestDB

Enterprise 4.0.0 fails startup with no backup timestamp at or before the target, and the operator reports RecoveryFailed=True/RestoreError. Because spec.bootstrap is immutable, delete the failed destination safely and create a fresh cluster with a valid target. :::

Confirm the source retention window before creating the immutable destination, then follow the same bounded watch, failure checks, and data validation as a normal restore.