Affinity and Tolerations Values Configuration

Affinity and tolerations control pod scheduling behavior in SUSE® Observability. They help optimize resource utilization and ensure high availability by controlling where pods are scheduled.

Global tolerations and node affinity apply to every SUSE® Observability workload, including infrastructure components, hooks, and backup and restore jobs. Pod affinity and pod anti-affinity have narrower scopes. You can override all global settings for individual components, see Precedence of global and component settings.

Available Configuration Options

Node Affinity

You can use node affinity to schedule pods to specific nodes or instance groups, such as EC2 nodes deployed to the same availability zone.

affinity:
  # Node Affinity settings - applied to all components when configured
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - us-west-2a

Pod Anti-Affinity

You can use pod anti-affinity to schedule replicas of data services to different nodes to ensure high availability. By default, the scheduling is required (hard anti-affinity) and topologyKey is kubernetes.io/hostname.

affinity:
  podAntiAffinity:
    # Enable required pod anti-affinity (true = hard, false = soft)
    requiredDuringSchedulingIgnoredDuringExecution: true
    # Topology key for pod anti-affinity
    topologyKey: "kubernetes.io/hostname"

Tolerations

Use tolerations to schedule SUSE® Observability pods on tainted nodes, for example a dedicated node pool.

global.suseObservability.tolerations is available from SUSE® Observability 2.13.0.

 
global:
  suseObservability:
    tolerations:
    - key: dedicated
      operator: Equal
      value: observability
      effect: NoSchedule

Precedence of global and component settings

The precedence rules described here apply from SUSE® Observability 2.13.0. See the version-specific upgrade instructions for the changes compared to earlier versions.

Affinity and tolerations can be set globally and for individual components. The most specific setting wins. All keys in this section are values of the suse-observability chart, as used in the recommended global.suseObservability method.

Workload Settings, most specific first

Application components

stackstate.components.<component>.split.<type> (only for receiver and correlate, when split.enabled is true), stackstate.components.<component>, stackstate.components.all, global.suseObservability

Backup and restore jobs

stackstate.components.backup (stackstate.components.configurationBackup for configuration backups), stackstate.components.all, global.suseObservability

HBase

hbase.<component>, hbase.all, global.suseObservability

Other infrastructure components

The component’s own setting, for example kafka.affinity, clickhouse.backup.tolerations or victoria-metrics-0.server.affinity, then global.suseObservability

Tolerations are resolved as a whole list. The first setting that defines tolerations replaces all less specific ones; the lists are not combined. Repeat any global tolerations that the component still needs. Set tolerations: [] to remove all inherited tolerations from a component.

global:
  suseObservability:
    tolerations:
    - key: dedicated
      operator: Equal
      value: observability
      effect: NoSchedule
stackstate:
  components:
    api:
      tolerations:
      - key: dedicated
        operator: Equal
        value: observability
        effect: NoSchedule
      - key: api-only
        operator: Exists
    ui:
      tolerations: []

Affinity is resolved separately for nodeAffinity, podAffinity and podAntiAffinity. A component that sets only one of them keeps the other two from less specific settings. Within a field, the most specific value replaces the less specific ones completely. For example, kafka.affinity.nodeAffinity replaces the global nodeAffinity instead of adding terms to it, so repeat any global node selector terms that Kafka still needs. On a component or subchart, set a field to {} to remove the inherited value. Setting {} in global.suseObservability.affinity does not remove the anti-affinity of the infrastructure components; to relax it for all of them, set global.suseObservability.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution to false, see Soft PodAntiAffinity for HA Deployment.

global.suseObservability.affinity.podAntiAffinity is a simplified setting that controls the anti-affinity of the infrastructure components on HA profiles. At component and subchart level, podAntiAffinity takes the standard Kubernetes format and replaces that generated anti-affinity.

kafka:
  affinity:
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 10
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: zookeeper
clickhouse:
  affinity:
    podAntiAffinity: {}

In this example, if global.suseObservability.affinity.nodeAffinity is set, Kafka keeps it. With an HA profile Kafka also keeps its pod anti-affinity, and it adds the podAffinity. ClickHouse keeps the global nodeAffinity but has no pod anti-affinity.

global.suseObservability.affinity.podAffinity applies only to application components, not to infrastructure components.

Elasticsearch uses elasticsearch.nodeAffinity for node affinity and elasticsearch.antiAffinity (hard or soft) for pod anti-affinity. When they are not set, Elasticsearch uses the global node affinity and, with an HA profile, hard or soft anti-affinity according to global.suseObservability.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution.

If pods stay in Pending after you set affinity or tolerations, check for leftover empty values such as tolerations: [] or nodeAffinity: {} in your values.yaml. They remove the inherited settings. For HA profiles with fewer nodes than replicas, set global.suseObservability.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution to false (see Soft PodAntiAffinity for HA Deployment), or set podAntiAffinity: {} on the component, for example kafka.affinity.podAntiAffinity: {}.

Behavior

Node Affinity

  • When configured: Applied to all components, with or without a sizing profile

Pod Anti-Affinity

  • When configured: The required pod anti-affinity of the sizing profile is applied only when sizing.profile ends with -ha (High Availability profiles)

  • HA Profiles: 150-ha, 250-ha, 500-ha, 4000-ha

  • Components affected: All stateful data services including Clickhouse, Kafka, Zookeeper, VictoriaMetrics, StackGraph, Elasticsearch

Example configurations

Basic Node Affinity (same Availability Zone) + hard PodAntiAffinity for HA Deployment

Assume you set sizing.profile to one of the HA profiles. The values below configure nodeAffinity to schedule components of SUSE® Observability to the us-west-2a zone and enforce scheduling the replicas of the same data services to different hosts (affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution=true by default).

affinity:
  # Schedule all pods to nodes in the same AZ
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - us-west-2a

Soft PodAntiAffinity for HA Deployment

If enforcing scheduling of data services to different nodes is not desirable, yo can configure podAntiAffinity in "soft" mode:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution: false

Usage

  • Recommended method

  • Legacy method (Deprecated)

The global.suseObservability configuration method is available starting from version 2.8.0. For earlier versions, use the legacy method.

Add affinity configuration directly to your values.yaml file under global.suseObservability.affinity:

global:
  suseObservability:
    # Your existing configuration
    license: "YOUR-LICENSE-KEY"
    baseUrl: "https://observability.example.com"
    sizing:
      profile: "150-ha"
    adminPassword: "your-password"

    # Affinity configuration
    affinity:
      # Node affinity - schedule all pods to nodes in the same AZ
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - us-west-2a

      # Pod anti-affinity for HA profiles (infrastructure components)
      podAntiAffinity:
        # true = hard anti-affinity, false = soft anti-affinity
        requiredDuringSchedulingIgnoredDuringExecution: true
        topologyKey: "kubernetes.io/hostname"

Then install or upgrade with a single command:

helm upgrade --install \
  --namespace suse-observability \
  --create-namespace \
  --values values.yaml \
  suse-observability \
  suse-observability/suse-observability
For HA profiles, pod anti-affinity is automatically applied to all stateful data services. You only need to configure it if you want to change the default behavior.

For more details on migration from the legacy method, see the migration guide.

This method is deprecated. For new installations, use the recommended method above. For existing installations using this method, see the migration guide to transition to the new configuration format.

==== Step 1: Create Your Affinity Values file

Create a separate values file with your desired affinity configuration. For example, save the following as suse-observability-values-values.yaml:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - us-west-2a
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution: true

==== Step 2: Generate Affinity template values

Run the following command to generate the affinity values template:

export VALUES_DIR=.
helm template \
  --set license='<your license>' \
  --set baseUrl='<suse-observability-base-url>' \
  --set sizing.profile='<sizing.profile>' \
  --values suse-observability-values-values.yaml \
  suse-observability-values \
  suse-observability/suse-observability-values --output-dir $VALUES_DIR

==== Step 3: Use generated values in Helm installation

Include the generated affinity values in your Helm installation:

helm upgrade \
  --install \
  --namespace suse-observability \
  --values $VALUES_DIR/suse-observability-values/templates/baseConfig_values.yaml \
  --values $VALUES_DIR/suse-observability-values/templates/sizing_values.yaml \
  --values $VALUES_DIR/suse-observability-values/templates/affinity_values.yaml \
  suse-observability \
  suse-observability/suse-observability