Change the sizing profile

Overview

The global.suseObservability.sizing.profile value selects the resources, replica counts and storage sizes of an installation. This page applies to installations that set the profile with this value. Installations that use the sizing_values.yaml file generated by the suse-observability-values chart are not covered, and neither is the trial profile. How you change the profile depends on the direction:

Change Method

Change	Method
Within non-HA (`10-nonha`, `20-nonha`, `50-nonha`, `100-nonha`)	Helm upgrade
Within HA (`150-ha`, `250-ha`, `500-ha`, `4000-ha`)	Helm upgrade
Between non-HA and HA, in either direction	Reinstall in the same namespace and restore from a backup

Within non-HA (10-nonha, 20-nonha, 50-nonha, 100-nonha)

Helm upgrade

Within HA (150-ha, 250-ha, 500-ha, 4000-ha)

Helm upgrade

Between non-HA and HA, in either direction

Reinstall in the same namespace and restore from a backup

Test the method you plan to use in a test environment first, whether it is a Helm upgrade or a reinstall with a restore. Both methods change or remove volumes and can cause downtime or data loss. A test run shows you how long it takes, which steps apply to your installation and what to expect before you do it on a production installation.

Always take a backup before you change the profile, and verify that it can be restored.

Change the profile with a Helm upgrade

Changing the profile in the values and running helm upgrade works only when no persistent volume has to change size. Kubernetes does not allow the volumeClaimTemplates of a StatefulSet to change, so an upgrade that changes a volume size is rejected:

StatefulSet.apps "suse-observability-kafka" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', ... are forbidden

A rejected upgrade is not harmless. Helm applies part of the release before it fails, which can restart pods. Compare the volume sizes and prepare the volumes before you upgrade.

Compare the volume sizes

Render the chart with the new profile and ask the Kubernetes API server whether it accepts the result. Use the chart version you installed and the values that are in use now, and replace the namespace and release name:

helm get values suse-observability --namespace suse-observability > current-values.yaml

helm template suse-observability suse-observability/suse-observability \
  --version <CHART_VERSION> \
  --namespace suse-observability \
  --values current-values.yaml \
  --set global.suseObservability.sizing.profile=<NEW_PROFILE> > new.yaml

kubectl --namespace suse-observability apply --dry-run=server --server-side --force-conflicts -f new.yaml

Every StatefulSet that reports Forbidden: updates to statefulset spec has a volume size that differs from the one in the cluster. To see the current sizes:

kubectl --namespace suse-observability get statefulset \
  -o custom-columns='NAME:.metadata.name,SIZE:.spec.volumeClaimTemplates[*].spec.resources.requests.storage'

helm upgrade --dry-run=server does not detect this problem and succeeds even when the real upgrade is rejected.

Handle each volume that changes

Volumes can grow but cannot shrink. For every volume that differs, choose one of the following:

Volume grows

Resize the volume as described in Resizing storage: patch the PVCs, then delete the StatefulSet with --cascade=orphan and run the upgrade. This needs a storage class with allowVolumeExpansion: true. Most CSI drivers expand the volume online, without restarting the pod.

Volume would shrink

Keep the current size by setting it explicitly in your values. The chart then renders the same size as the cluster and the StatefulSet is unchanged. For example, to keep ClickHouse at its current 100Gi:

clickhouse:
  persistence:
    size: 100Gi

Use the keys in the table below.

The volume sizes per profile are in the table at the end of this page. The sizes that differ between two profiles can differ in either direction, so check each pair. For example, 150-ha gives ClickHouse 100Gi but 250-ha only 50Gi, so an upgrade from 150-ha to 250-ha shrinks ClickHouse unless you set it explicitly. A downgrade from 250-ha to 150-ha is rejected for the opposite reason on Elasticsearch, because its volumes are larger in 250-ha, and it grows ClickHouse from 50Gi to 100Gi.

Volume Values key

Volume	Values key
Elasticsearch	`elasticsearch.volumeClaimTemplate.resources.requests.storage`
ClickHouse	`clickhouse.persistence.size`
Kafka	`kafka.persistence.size`
StackGraph (HBase, non-HA)	`hbase.stackgraph.persistence.size`
HDFS data nodes (HA)	`hbase.hdfs.datanode.persistence.size`
VictoriaMetrics	`victoria-metrics-0.server.persistentVolume.size`, and `victoria-metrics-1.server.persistentVolume.size` for HA
ZooKeeper	`zookeeper.persistence.size`

Elasticsearch

elasticsearch.volumeClaimTemplate.resources.requests.storage

ClickHouse

clickhouse.persistence.size

Kafka

kafka.persistence.size

StackGraph (HBase, non-HA)

hbase.stackgraph.persistence.size

HDFS data nodes (HA)

hbase.hdfs.datanode.persistence.size

VictoriaMetrics

victoria-metrics-0.server.persistentVolume.size, and victoria-metrics-1.server.persistentVolume.size for HA

ZooKeeper

zookeeper.persistence.size

Example: Kafka when moving from `10-nonha` to `20-nonha`

Kafka is the only volume that differs between these two profiles: 60Gi in 10-nonha and 100Gi in 20-nonha.

Patch the Kafka PVC:

kubectl --namespace suse-observability patch pvc data-suse-observability-kafka-0 \
  -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'

Wait until kubectl get pvc shows the new capacity.

Delete the StatefulSet and keep its pods:

kubectl --namespace suse-observability delete statefulset suse-observability-kafka --cascade=orphan

Set the new profile in your values and run the same helm upgrade command that you used for the installation.

To go back from 20-nonha to 10-nonha, keep Kafka at 100Gi by setting kafka.persistence.size: 100Gi in your values. Without it the downgrade is rejected.

Kafka partitions

The 500-ha and 4000-ha profiles create the sts_correlate_endpoints, sts_correlate_http_trace_observations and sts_correlated_connections topics with 40 partitions instead of 10. The upgrade job creates a topic with these values only when the topic does not exist, and for an existing topic it updates the settings but never the number of partitions. This means:

Upgrading to 500-ha or 4000-ha keeps the existing topics at 10 partitions. Correlation keeps working, but it is less efficient than these profiles are sized for.
Downgrading keeps the topics at 40 partitions, which is harmless.

To get the 40 partitions after an upgrade to 500-ha or 4000-ha, increase them once the upgrade has finished and all Kafka pods are ready:

kubectl --namespace suse-observability exec suse-observability-kafka-0 -c kafka -- bash -c '
unset JMX_PORT
for topic in sts_correlate_endpoints sts_correlate_http_trace_observations sts_correlated_connections; do
  kafka-topics.sh --bootstrap-server suse-observability-kafka-headless:9092 --alter --topic "$topic" --partitions 40
  kafka-topics.sh --bootstrap-server suse-observability-kafka-headless:9092 --describe --topic "$topic" | head -n 1
done'

Each --describe line must show PartitionCount: 40. unset JMX_PORT stops the tools from opening the JMX port of the broker. Running the command for a topic that already has 40 partitions fails for that topic and changes nothing.

You can increase the number of partitions of a topic but never decrease it. For a few minutes after the change, some connections and traces may not be correlated. Stored topology, metrics, logs and traces are not affected.

Replicas

Moving between HA profiles can change the number of replicas of ClickHouse, HBase and HDFS. The scale-up of replicas needs enough free capacity in the cluster. During the upgrade, the old and the new pods can exist at the same time, and some pods stay Pending until the cluster autoscaler adds nodes. Allow several minutes for the upgrade to settle.

Downgrade from 4000-ha

Moving down from 4000-ha to another HA profile reduces the number of ClickHouse replicas from three to one, and the HDFS data nodes and HBase region servers from five to three. This page covers only ClickHouse. To leave 4000-ha without handling the HDFS data nodes and HBase region servers yourself, use the reinstall procedure instead.

4000-ha is the only profile with more than one ClickHouse replica. Helm removes the pods suse-observability-clickhouse-shard0-1 and suse-observability-clickhouse-shard0-2, but the replicas stay registered in the ClickHouse coordination service. The tables then report replicas that are never active, and statements that change data in all replicas, such as ALTER TABLE … DELETE, fail with some replicas are inactive.

When the upgrade has finished and the remaining ClickHouse pod is ready, remove the registrations of the two replicas that no longer exist. Replace the pod and replica names if your release name is not suse-observability:

kubectl --namespace suse-observability exec -i suse-observability-clickhouse-shard0-0 -c clickhouse -- bash -c \
  'clickhouse-client --user "$CLICKHOUSE_ADMIN_USER" --password "$CLICKHOUSE_ADMIN_PASSWORD" --multiquery' <<'SQL'
SYSTEM DROP REPLICA 'suse-observability-clickhouse-shard0-1' FROM TABLE otel.otel_migrations;
SYSTEM DROP REPLICA 'suse-observability-clickhouse-shard0-1' FROM TABLE otel.otel_resources;
SYSTEM DROP REPLICA 'suse-observability-clickhouse-shard0-1' FROM TABLE otel.otel_traces;
SYSTEM DROP REPLICA 'suse-observability-clickhouse-shard0-1' FROM TABLE otel.otel_traces_field_values;
SYSTEM DROP REPLICA 'suse-observability-clickhouse-shard0-1' FROM TABLE otel.otel_traces_trace_id_ts;
SYSTEM DROP REPLICA 'suse-observability-clickhouse-shard0-2' FROM TABLE otel.otel_migrations;
SYSTEM DROP REPLICA 'suse-observability-clickhouse-shard0-2' FROM TABLE otel.otel_resources;
SYSTEM DROP REPLICA 'suse-observability-clickhouse-shard0-2' FROM TABLE otel.otel_traces;
SYSTEM DROP REPLICA 'suse-observability-clickhouse-shard0-2' FROM TABLE otel.otel_traces_field_values;
SYSTEM DROP REPLICA 'suse-observability-clickhouse-shard0-2' FROM TABLE otel.otel_traces_trace_id_ts;
SQL

The statements are safe to run on the remaining pod: ClickHouse refuses to drop the replica of the pod that you run them on. If your version has different tables, run the statement for each table that SELECT table FROM system.replicas WHERE database = 'otel' returns.

Check the result with the same client:

SELECT table, total_replicas, active_replicas FROM system.replicas WHERE database = 'otel'

Every table must show 1 for both columns.

The volume claims of the two removed replicas, data-suse-observability-clickhouse-shard0-1 and data-suse-observability-clickhouse-shard0-2, remain in the cluster. Delete them when you no longer need the data to free the storage.

Move between non-HA and HA

The non-HA and HA profiles use different StatefulSets, replica counts and storage layouts, so there is no in-place upgrade between them. Instead, remove the installation, install it again with the target profile in the same namespace, and restore the data from a backup. This works in both directions and does not need a new namespace or cluster.

The installation is unavailable from the moment you remove it until the restore has finished. Data that it receives after the backup is lost, including data that the new installation receives before the restore.

This procedure requires backup CLI version 0.12.0 or newer. Download the latest version as described in Download the backup CLI.

Before you start

Backups must be enabled for every data store and stored in object storage (S3 or Azure Blob Storage), as described in Enable backups. Do not use the Kubernetes persistent volume storage option: its volumes are deleted in step 2.
Keep your values.yaml. You install again with the same values and change only the profile.
If Argo CD or another GitOps tool manages the installation, turn off automatic sync for the whole procedure and remove and install the application through that tool. Otherwise it can recreate the old StatefulSets while you delete the volumes.
Do the removal, installation and restore in one sitting. Restore VictoriaMetrics within one day of the installation in step 3: after that, the hourly backup of the new installation overwrites the backup that you restore.

Step 1: Back up the installation

Most scheduled backups run once a day. To restore the current state, start a backup of every data store now:

kubectl --namespace suse-observability create job --from=cronjob/suse-observability-backup-sg-v2 manual-backup-sg
kubectl --namespace suse-observability create job --from=cronjob/suse-observability-clickhouse-full-backup manual-backup-clickhouse

For Elasticsearch, run the snapshot policy through a port-forward, and stop the port-forward when you have finished:

kubectl --namespace suse-observability port-forward svc/suse-observability-elasticsearch-master 9200:9200 &
curl -XPOST localhost:9200/_slm/policy/auto-sts-backup/_execute

Note the snapshot_name that the call returns.

VictoriaMetrics has no on-demand backup. It backs up from a sidecar on the hourly schedule, which you can change with victoria-metrics-0.backup.scheduled.hourly (and victoria-metrics-1 for HA). Wait for the next run.

The jobs and the snapshot call only start the backups. Wait until the jobs have completed and, for Elasticsearch, until the snapshot state is SUCCESS. Then list the backups of each data store with its list command, as described in Restore backups, and note the name of the newest Elasticsearch snapshot, ClickHouse backup, VictoriaMetrics backup and StackGraph v2 archive. Check that each one was created after you started this step. For VictoriaMetrics the newest backup is <bucket>/victoria-metrics-0-latest, which every hourly run updates in place.

Step 2: Remove the installation

Remove the Helm release and then the volumes. helm uninstall keeps the volume claims of the StatefulSets, and the new installation would otherwise reuse them with their old sizes and data.

helm uninstall suse-observability --namespace suse-observability
kubectl --namespace suse-observability get pvc

Delete every volume claim of the installation. Their names all contain suse-observability. Check the list first if other applications share the namespace:

kubectl --namespace suse-observability get pvc -o name | grep suse-observability | xargs kubectl --namespace suse-observability delete

Secrets that you created yourself, such as the license, API key and admin password secrets, are not removed and are used again by the new installation.

Step 3: Install with the target profile

Install again with the same chart version and values, with global.suseObservability.sizing.profile set to the new profile and these backup settings:

Keep the bucketName (the container on Azure) and s3Prefix values of every data store unchanged, so that the new installation reads the old backups.
Remove the volume sizes that you set for an earlier Helm upgrade, unless you want to keep them. The profile then decides the sizes.
Set backup.elasticsearch.enabled: false until the restore has finished. The new installation then does not take Elasticsearch snapshots of its own, which could replace your backup under the snapshot retention rules. The snapshot repository stays registered, so the restore still works.
Keep clickhouse.backup.enabled: true. When it is disabled, the backup service that the restore needs is removed.
Keep victoria-metrics-0.backup.minHistorySeconds (and the victoria-metrics-1 key) at its default, not 0. For one day after the installation it stops the hourly backup from overwriting the backup in victoria-metrics-0-latest.

Wait until all pods are ready.

Step 4: Restore every data store

Restore the backups that you noted in step 1. Use the names, not --latest: a scheduled backup of the new, empty installation could otherwise be the newest one.

sts-backup elasticsearch restore --namespace suse-observability --snapshot <SNAPSHOT> --yes
sts-backup clickhouse restore --namespace suse-observability --snapshot <BACKUP> --yes
sts-backup victoria-metrics restore --namespace suse-observability --archive <ARCHIVE> --yes
sts-backup stackgraph-v2 restore --namespace suse-observability --archive <ARCHIVE> --yes

Use the stackgraph-v2 subcommand. The other StackGraph subcommand looks for v1 backups. All commands are described in Restore backups.

If a restore is interrupted, finish it with the check-and-finalize command of that data store. A new restore command is refused while the previous one holds its restore lock.

Step 5: Set the number of Elasticsearch replicas

Skip this step if the number of Elasticsearch nodes does not change.

The restored indices keep the replica count of the installation you backed up. Set it to match the new profile, otherwise Elasticsearch stays yellow on a single node, or keeps no copy of the data on three nodes. The Elasticsearch pods are new, so open a new port-forward as in step 1.

For an HA profile, which has three Elasticsearch nodes:

curl -XPUT 'localhost:9200/sts*,.ds-sts*/_settings?expand_wildcards=all' \
  -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":1}}'

For a non-HA profile use "number_of_replicas":0.

Step 6: Finish and check

Remove backup.elasticsearch.enabled: false from your values and run helm upgrade with the same chart version and values.
Open the UI and compare the topology, monitors and installed StackPacks with what you saw before.
Check that Elasticsearch is green and that the sts_k8s_logs data stream exists: curl localhost:9200/_data_stream/sts_k8s_logs.
Check that all pods are ready.

The StackGraph restore brings back settings, StackPacks, permissions and service tokens. The admin password and API key come from your values.

Volume sizes per profile

The sizes in the table are the defaults of the chart. They can change between chart versions; check the output of helm template for the version you install.

Volume	10-nonha	20-nonha	50-nonha	100-nonha	150-ha	250-ha	500-ha	4000-ha
Elasticsearch	50Gi	50Gi	50Gi	100Gi	200Gi	250Gi	250Gi	250Gi
ClickHouse	50Gi	50Gi	50Gi	50Gi	100Gi	50Gi	50Gi	50Gi
Kafka	60Gi	100Gi	100Gi	100Gi	100Gi	100Gi	400Gi	400Gi
StackGraph (non-HA)	50Gi	50Gi	100Gi	100Gi	-	-	-	-
HDFS data nodes (HA)	-	-	-	-	250Gi	250Gi	250Gi	1000Gi
VictoriaMetrics	50Gi	50Gi	50Gi	50Gi	250Gi	250Gi	250Gi	400Gi
ZooKeeper	8Gi	8Gi	8Gi	8Gi	8Gi	8Gi	8Gi	8Gi

Volume

10-nonha

20-nonha

50-nonha

100-nonha

150-ha

250-ha

500-ha

4000-ha

Elasticsearch

50Gi

100Gi

200Gi

250Gi

ClickHouse

50Gi

100Gi

50Gi

Kafka

60Gi

100Gi

400Gi

StackGraph (non-HA)

50Gi

100Gi

HDFS data nodes (HA)

250Gi

1000Gi

VictoriaMetrics

50Gi

250Gi

400Gi

ZooKeeper

8Gi

The number of replicas of the data services also differs: Elasticsearch and Kafka have 3 replicas in every HA profile, ClickHouse has 3 replicas in 4000-ha, and HDFS has 5 data nodes and HBase 5 region servers in 4000-ha.

Change the sizing profile

Overview

Change the profile with a Helm upgrade

Compare the volume sizes

Handle each volume that changes

Example: Kafka when moving from 10-nonha to 20-nonha

Kafka partitions

Replicas

Downgrade from 4000-ha

Move between non-HA and HA

Before you start

Step 1: Back up the installation

Step 2: Remove the installation

Step 3: Install with the target profile

Step 4: Restore every data store

Step 5: Set the number of Elasticsearch replicas

Step 6: Finish and check

Volume sizes per profile

Example: Kafka when moving from `10-nonha` to `20-nonha`