Telemetry gateway

Overview

The telemetry gateway is an OpenTelemetry Collector component in the SUSE® Observability Agent that accepts metrics and traces pushed over OTLP from application-embedded OpenTelemetry SDKs. It enriches the data with Kubernetes metadata and forwards it to the SUSE® Observability platform.

Use the telemetry gateway when your workloads are instrumented with an OTel SDK (Java, Go, Python, Node.js, etc.) and export telemetry using OTLP. It is the recommended integration path for natively instrumented services.

If your workloads expose a Prometheus / OpenMetrics endpoint instead, see Scraping OpenMetrics with ServiceMonitor and PodMonitor.

Enable

The telemetry gateway is disabled by default. Enable it in your agent Helm values alongside the OTel master switch:

otel:
  enabled: true
  telemetryGateway:
    enabled: true

After the Helm upgrade, a Deployment named suse-observability-agent-otel-telemetry-gateway is created in the agent namespace, along with a ClusterIP Service, ClusterRole, and ServiceAccount.

Configure your SDKs

Point your application SDKs at the telemetry gateway Service. By default, the Service is suse-observability-agent-otel-telemetry-gateway in the suse-observability-agent namespace.

Set the following environment variables on each application pod. Most OTel SDKs pick these up automatically at startup without any code changes.

Log ingestion via OTLP is not yet supported. Set OTEL_LOGS_EXPORTER=none to avoid exporting logs. If a workload sends OTLP logs to the gateway anyway, they are written to the gateway pod logs for debug visibility and are not ingested as SUSE® Observability logs.

gRPC (port 4317)

OTEL_EXPORTER_OTLP_ENDPOINT=http://suse-observability-agent-otel-telemetry-gateway.suse-observability-agent.svc.cluster.local:4317
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=none
OTEL_SERVICE_NAME=<your-service-name>
OTEL_RESOURCE_ATTRIBUTES=k8s.pod.uid=$(POD_UID)

HTTP (port 4318)

OTEL_EXPORTER_OTLP_ENDPOINT=http://suse-observability-agent-otel-telemetry-gateway.suse-observability-agent.svc.cluster.local:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=none
OTEL_SERVICE_NAME=<your-service-name>
OTEL_RESOURCE_ATTRIBUTES=k8s.pod.uid=$(POD_UID)

Inject POD_UID via the Kubernetes downward API:

env:
  - name: POD_UID
    valueFrom:
      fieldRef:
        fieldPath: metadata.uid

Attribute ownership (security enforcement)

The collector overwrites the following attributes on every received payload, regardless of what the client sends. These attributes are used for RBAC and tenancy decisions, so the collector queries the Kubernetes API for authoritative values instead of relying on application-provided values.

  • k8s.cluster.name — set from the agent Helm release configuration

  • k8s.namespace.name — set from the Kubernetes API using the source pod’s identity

Attributes that applications must set:

  • service.name — the logical name of your service

  • service.version — the deployed version

  • deployment.environment — e.g. production or staging

  • k8s.pod.uid — recommended; inject via the downward API (see above) for reliable pod association. The collector also falls back to k8s.pod.ip and connection source address, but downward API injection is the most reliable method and should be preferred.

Span metrics

The gateway automatically generates span metrics (request rate, error rate, duration histograms) from incoming traces using the span_metrics connector. The generated metrics use the otel_span namespace and follow OTel semantic conventions:

  • Attribute span.name

  • Histogram name otel_span.duration

  • Counter without _total suffix

These metrics are forwarded to SUSE® Observability alongside your application metrics and can be used to build golden-signal dashboards without any additional instrumentation.

Trace sampling

The gateway uses the upstream OpenTelemetry tail-sampling processor. By default, it buffers spans for approximately 10 seconds after first receiving a trace, then decides whether to export the spans received so far. Span metrics are generated before sampling, so sampling does not bias the gateway’s request rates, error rates, or duration histograms. The platform does not apply another tail-sampling stage to received traces.

The default span budget is 500 spans per second per gateway pod, allocated in this order:

Policy Matches Budget allocation

Errors

At least one span has OTel status ERROR

33%

Slow traces

Time from the earliest span start to the latest span end exceeds one second

33%

Remaining traces

All other traces

34%

These percentages allocate span budgets; they are not percentages of traces retained. A trace is dropped if its spans exceed its matching policy’s remaining budget, including error traces. Unused capacity is not shared between policies, and an exhausted policy does not fall through to the next policy. Late spans of retained traces bypass a new budget check, so this is not a strict limit on exporter throughput.

Tune sampling in the agent Helm values: The waiting-period, capacity, and cache settings require Agent Helm chart version 1.7.27 or later.

otel:
  telemetryGateway:
    traceSampling:
      maxTotalSpansPerSecond: 500
      decisionWait: 10s
      numTraces: 50000
      decisionCache:
        sampledCacheSize: 0
        nonSampledCacheSize: 0

decisionWait controls the waiting period; numTraces controls trace-entry capacity per pod. The cache sizes count remembered keep and drop decisions; zero disables a cache. Choose cache sizes substantially larger than numTraces to retain decisions longer, and allow for their memory use. Longer waiting periods and larger buffers or caches require more memory. The sampling policies remain fixed.

Limitations

  • Late spans: A span arriving after the decision cannot change it, even if it contains an error. Decision caches are disabled by default. Without caches, late spans inherit the decision while the trace entry remains in memory; after eviction, they are evaluated again and can receive a different decision, producing incomplete traces.

  • Buffer capacity: Each gateway pod defaults to 50,000 trace entries (numTraces). This counts pending traces and, when decision caches are disabled, remembered decisions, not spans or bytes; memory usage depends on span count and size. When full, the oldest entry is evicted. If its decision wait has not finished, its buffered spans are lost. Increasing the export budget does not increase buffer capacity.

  • Restarts: Buffered spans and remembered decisions are held in memory and are lost if a gateway pod terminates unexpectedly.

  • SDK sampling: Spans dropped by an application SDK never reach the gateway. SDK head sampling can therefore hide errors from the tail sampler and reduce the traffic represented by span metrics.

  • Multiple gateways: All spans of a trace must reach the same sampling pod, including traces spanning services or clusters. See Scaling.

Operational monitoring

The gateway exposes standard OpenTelemetry Collector self-metrics on its internal Prometheus endpoint (:8888). These include signals such as exporter failures, refused data, process memory, and pipeline throughput. The gateway scrapes its own endpoint with a built-in prometheus/self receiver and forwards those self-metrics to the platform for operational visibility.

Monitor tail-sampler buffer occupancy, traces dropped before a decision, late-span age, and sampled versus rejected decisions alongside process memory.

Network policies

The chart does not install NetworkPolicy resources. If your cluster enforces network policies, configure:

  • Ingress to the agent namespace on port 4317 (gRPC) or 4318 (HTTP) from application namespaces.

  • Egress from application pods to the agent namespace.

To allow a specific namespace to send telemetry to the gateway:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-otlp-to-gateway
  namespace: suse-observability-agent
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: otel-telemetry-gateway
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: my-app-namespace
      ports:
        - port: 4317
        - port: 4318

To block a namespace from sending telemetry, deny its egress to the gateway namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-otlp-egress
  namespace: restricted-namespace
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchExpressions:
              - key: kubernetes.io/metadata.name
                operator: NotIn
                values:
                  - suse-observability-agent

Scaling

Horizontal scaling with correct tail sampling is currently not supported by the agent Helm chart. Keep otel.telemetryGateway.replicaCount: 1 for tail sampling. The gateway Service does not route by trace ID: with multiple replicas, spans of one trace can reach different samplers and receive inconsistent decisions.

Supporting multiple sampling pods requires a collector tier in front that uses the OpenTelemetry load-balancing exporter to route by trace ID. The chart does not deploy that tier. Each replica also has its own sampling budget.

The chart creates a PodDisruptionBudget at two or more replicas, but this does not preserve trace buffers or sampling decisions. Rolling updates can also split traces between old and new pods, even with a configured replica count of one.

Resource limits and requests can be tuned in the Helm values:

otel:
  telemetryGateway:
    resources:
      requests:
        cpu: 250m
        memory: 1Gi
      limits:
        cpu: 1
        memory: 2Gi

Overload protection

The gateway applies memory_limiter before enrichment and trace sampling. It helps protect the process from OOM by rejecting data above the soft memory limit, derived from an 80% hard limit and a 10% spike allowance. Under sustained overload, data can be lost if senders cannot retry successfully. Tail sampling limits trace export volume; it does not limit incoming traffic or guarantee that the trace buffer fits in memory.

Span-metric generation also has a cardinality guard. otel.telemetryGateway.spanMetrics.aggregationCardinalityLimit limits the number of unique span-metric aggregation series held by the gateway. The default protects memory from high-cardinality spans, while larger clusters can tune it when needed:

otel:
  telemetryGateway:
    spanMetrics:
      aggregationCardinalityLimit: 5000

Product integration contract

This section is for SUSE product teams that want their Helm charts to export push-based OTLP telemetry through the telemetry gateway when the SUSE® Observability Agent supports it.

Capability detection

Before enabling auto-export by default, check for the marker CRD in Helm:

{{- if .Capabilities.APIVersions.Has "observability.suse.com/v1/suseobservabilityagents" }}

The agent installs this CRD when at least one product-facing OTel integration path is active, such as Prometheus scraping or the telemetry gateway. Treat it as a capability signal only. It does not identify the agent namespace, Service name, protocol, port, or NetworkPolicy reachability.

Endpoint

The default gateway Service name is suse-observability-agent-otel-telemetry-gateway: the agent Helm release name, suse-observability-agent, plus the -otel-telemetry-gateway suffix. Product charts should construct this endpoint from configurable release name, namespace, protocol, and port values so custom agent installations are supported.

Opt-in/override

Product charts should keep telemetry export configurable even when the marker CRD exists. Allow users to disable export, override the endpoint, and choose OTLP gRPC or HTTP/protobuf where the product supports both.

Attribute ownership

The collector owns k8s.cluster.name and k8s.namespace.name (see Attribute ownership). Products must not set or rely on user-supplied values for these. Products must set service.name, service.version, and deployment.environment. Products should inject k8s.pod.uid via the downward API — this is the most reliable pod-association method; the collector falls back to pod IP and connection source address.

Log ingestion

Log signals are not yet supported for SUSE® Observability log ingestion. Products must not export logs to the gateway at this time; any logs sent to the gateway are written to the gateway pod logs for debug visibility only.