6 High Availability setup #

Most Private Registry application components are stateless. You can run multiple replicas and distribute them across worker nodes or failure domains. This protects the application from a pod or node failure, but only when the replicas do not share the same failure domain.

The stateful parts of the deployment require separate resilience decisions:

Private Registry does not deploy or manage HA for these external endpoints and services.

Choose a model based on the kinds of failure that the deployment must tolerate:

Increasing the replica count alone does not create a resilient deployment. For example, two replicas on the same worker node are both affected when that node fails.

The example in Appendix B, Example of a Private Registry HA setup Helm chart sets three replicas for the portal, core, job service, and registry components, together with the corresponding nodeSelector and topologySpreadConstraints values. These settings are available for the Private Registry components in Appendix A, Overriding the SUSE Private Registry Helm chart.

Use the following scheduling principles:

Kubernetes does not automatically spread replicas across failure domains unless the workload includes suitable scheduling rules. For more information, see the Kubernetes pod assignment and topology spread constraints documentation.

The chart can create a pod disruption budget for each component with <COMPONENT>.podDisruptionBudget.enabled, which is disabled by default. Node-drain policy remains a cluster-level control. Define it with the platform tools used by your organization and verify that the disruption budgets do not prevent planned maintenance.

Size the worker-node pool with at least one node more than the replica count. With whenUnsatisfiable: DoNotSchedule and a pool the same size as the replica count, draining a node leaves one replica of every component unschedulable until the node returns. The registry stays available on the remaining replicas, but the deployment runs without spare capacity for the duration of the maintenance.

Figure 6.1: Private Registry HA setup #

 

Use a highly available Ingress controller or load balancer in front of the Private Registry services. The endpoint provides the redundancy; DNS only needs a stable record that resolves reliably to it, and the TLS certificate only needs to remain valid when pods or worker nodes change. The externalURL value and expose.ingress.hosts.core value must identify the same registry address.

Private Registry does not manage the external endpoint. Configure health checks, endpoint failover, certificate renewal, and DNS availability with the Ingress or load-balancer provider.

When expose.type is ingress, the chart does not deploy its own nginx proxy, and increasing nginx.replicas has no effect. Endpoint redundancy then comes entirely from the Ingress controller. Increase nginx.replicas only when expose.type is clusterIP, nodePort, or loadBalancer.

An HA deployment has the same base prerequisites as a standard deployment. For the supported Kubernetes versions, the required Helm version, and the subscription requirement, see Section 2.1, “Prerequisites”.

An HA deployment additionally requires the following:

The following table summarizes the main dependency choices:

External managed services reduce the operational work inside the Kubernetes cluster, but introduce network, credential, and provider dependencies. In-cluster stateful services keep the deployment self-contained, but you must design their replication, storage, upgrades, monitoring, and backups. Private Registry does not provide those HA implementations.

For the supported connection and storage values, see Appendix B, Example of a Private Registry HA setup Helm chart and Appendix A, Overriding the SUSE Private Registry Helm chart.

When you use file system storage, confirm that the storage class can make the volume available after a node failure. Shared ReadWriteMany access is not automatically required for every deployment, but the access mode must match the selected chart configuration and storage implementation. When you use object storage, protect the object store with its own availability, access-control, retention, and backup policies.

To configure file system storage, set the storage class and access mode in the values file. For example:

persistence:
  enabled: true
  persistentVolumeClaim:
    registry:
      storageClass: <STORAGE_CLASS_NAME>
      accessMode: ReadWriteMany
      size: 500Gi

Use ReadWriteMany only when the storage implementation supports simultaneous access from the nodes that run the registry pods. Verify that the StorageClass supports the requested access mode and provides the required durability and failure behavior before using it in production.

When the storage implementation does not support ReadWriteMany, you must set updateStrategy.type to Recreate. That strategy stops the running pods before it starts the new ones, which causes downtime during an upgrade and prevents the registry components from running more than one replica per volume. This applies even to a component that runs a single replica. With the default RollingUpdate strategy, the replacement pod is created before the running pod stops, and it stays Pending indefinitely: the ReadWriteOnce volume ties it to one node, and the topologySpreadConstraints of the component keep it off that node while the running pod is still there. Use object storage or a ReadWriteMany volume when the deployment must stay available during upgrades.

Select the storage class and the access mode before the first installation. The specification of a PersistentVolumeClaim cannot be changed afterwards, so an upgrade that modifies storageClass or accessMode for an existing release fails with spec is immutable after creation and leaves the release in a failed state. Recover from it with helm rollback. Only the size can be increased, and only when the storage class allows volume expansion. To move the registry to a different storage class or access mode, copy the data to the new volume, delete the claim, and let the chart create it again. Note that a claim created with persistence.resourcePolicy set to keep also survives helm uninstall and must be deleted explicitly. Installing the chart again with the same release name in the same namespace reuses the surviving claims and preserves their data.

Use the sizing recommendations in Chapter 2, Requirements as a starting point. Increase capacity based on artifact growth, scan activity, job logs, retention, and replication traffic.

Private Registry supports connecting to an external PostgreSQL database. You can deploy CloudNativePG (CNPG) to provide a Kubernetes-native, highly available PostgreSQL cluster. CloudNativePG runs one primary instance and one or more streaming standbys. It manages failover, rolling updates and backups declaratively through a Cluster custom resource.

Using CloudNativePG for the external database provides several advantages:

> export CNPG_VERSION=<chart-version>
> export CNPG_IMAGE=ghcr.io/cloudnative-pg/postgresql:<tag>

> helm upgrade --install cnpg oci://dp.apps.rancher.io/charts/cloudnative-pg \
  --version "$CNPG_VERSION" \
  --namespace cnpg-system --create-namespace \
  --set config.clusterWide=true \
  --set crds.create=true \
  --wait

> kubectl -n cnpg-system rollout status deployment -l app.kubernetes.io/name=cloudnative-pg
> kubectl wait --for=condition=Established crd/clusters.postgresql.cnpg.io --timeout=5m

> kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: spr-postgres
  namespace: <PRIVATE_REGISTRY_NAMESPACE>
spec:
  instances: 3
  imageName: ${CNPG_IMAGE}
  postgresUID: 26
  postgresGID: 26
  enableSuperuserAccess: false
  affinity:
    enablePodAntiAffinity: true
    podAntiAffinityType: required
    topologyKey: kubernetes.io/hostname
  bootstrap:
    initdb:
      database: registry
      owner: harbor
  postgresql:
    parameters:
      max_connections: "300"
  storage:
    size: 8Gi
    storageClass: <STORAGE_CLASS_NAME>
EOF

> kubectl -n <PRIVATE_REGISTRY_NAMESPACE> wait --for=condition=Ready cluster.postgresql.cnpg.io/spr-postgres --timeout=15m
> kubectl cnpg status spr-postgres -n <PRIVATE_REGISTRY_NAMESPACE>

database:
  type: external
  maxIdleConns: 5
  maxOpenConns: 20
  external:
    host: spr-postgres-rw.<PRIVATE_REGISTRY_NAMESPACE>.svc.cluster.local
    port: 5432
    username: harbor
    coreDatabase: registry
    existingSecret: spr-postgres-app
    sslmode: verify-full

Follow this procedure for planned maintenance, such as a node drain or a rolling upgrade:

Set persistence.resourcePolicy to keep when PVCs must remain after a Helm release is deleted, for example when a release is reinstalled as part of maintenance. This setting does not replace backups and does not protect PVCs from deletion outside Helm.

Before placing the deployment into production, test the failure of one pod and one worker node, and test the documented recovery procedure with a non-production backup.

Retention and garbage-collection settings are standing configuration, not steps in the maintenance procedure in Section 6.9, “Performing maintenance and upgrades”. Set them up once, independently of any planned maintenance, and let them run on their own schedule.

Configure image retention policies and schedule garbage collection in the Private Registry administrator interface. Garbage collection removes unreferenced image data after retention policies or artifact deletions run. Review the reclaimable storage before a production garbage-collection run and monitor storage growth afterward. The registry upload-purging settings also remove leftover data from incomplete or interrupted image uploads (as opposed to garbage collection, which removes unreferenced but otherwise complete image data); see Appendix A, Overriding the SUSE Private Registry Helm chart for the available registry.upload_purging values.

Monitor the following conditions:

Enable the chart’s Prometheus metrics and configure a ServiceMonitor when your cluster runs the Prometheus Operator:

metrics:
  enabled: true
  serviceMonitor:
    enabled: true

Create alerts for unavailable replicas, repeated pod restarts, storage capacity, failed scans, failed jobs, and dependency health. Use metrics.serviceMonitor.additionalLabels when your monitoring stack requires a label to discover the ServiceMonitor.

Ensure that the worker-node and load-balancer networks provide enough bandwidth for peak image pushes, pulls, replication, and vulnerability database updates. Monitor network throughput and connection errors during normal and peak workloads.

If the registry requires a corporate proxy, configure the proxy URL, excluded addresses, and affected components in the Helm values:

proxy:
  httpProxy: <HTTP_PROXY_URL>
  httpsProxy: <HTTPS_PROXY_URL>
  noProxy: 127.0.0.1,localhost,.local,.internal
  components:
    - core
    - jobservice
    - trivy

Include the registry services, database, cache, object storage, Kubernetes API, and monitoring endpoints in noProxy when they must not use the proxy. Verify proxy access for each component that downloads external content or connects to an external service.

HA reduces downtime during selected failures. It does not replace backups or protect against accidental deletion, data corruption, or loss of the entire cluster. Define the required recovery point objective (RPO) and recovery time objective (RTO) before selecting backup schedules and storage.

For a complete recovery plan:

For the detailed Velero procedure, see Chapter 9, Backing up and restoring with Velero. The procedure also describes expected recovery behavior, including lost Valkey or Redis sessions, interrupted tasks, internal PostgreSQL consistency requirements, and the need to preserve chart-generated secrets.

Confirm the following items before you rely on the deployment for production workloads: