This is unreleased documentation for Admission Controller 1.39-dev.

Policy evaluation timeout and memory limit protection

This feature is available starting from SUSE Security Admission Controller v1.5.0 for per-Policy-Server timeout, and from v1.29.0 for per-Policy timeout.

Policy evaluation timeout protection is a security feature of the Policies and Policy Server. Its purpose is to limit the amount of time a request evaluation can take.

Purpose

You can write Admission Controller policies using both traditional programming languages (like Go, Rust and others) or using the special query language Rego. Both approaches have merits so a goal of Admission Controller is to let the policy authors choose the best tool for their needs.

When using a traditional Turing-complete language, it’s possible to have issues like:

The policy evaluation timeout protection feature terminates the evaluation of a request after a predefined period of time. This ensures the Policy Server always has compute resources available to process incoming requests.

Limitations

Currently, policy evaluation timeout protection is capable of interrupting most long running evaluations. There are certain edge cases not yet handled. This includes invoking a sleep instruction from within a policy, and deadlocks.

A future release of Policy Server will address these scenarios.

Finally, the policy evaluation timeout affects all the policies hosted by a Policy Server instance. Currently, there’s no way to tune policy evaluation timeout on a per-policy basis.

Configuration

All values of the per-Policy evaluation timeouts, the per-Policy-Server evaluation timeout, and the webhook timeouts are validated so they all are within acceptable values of each other. For example, it is not possible to set a Policy evaluation timeout value that is higher than the Kubernetes' webhook timeout.

Per Policy

Starting with Admission Controller v1.29.0, every Admission Controller Policy can set its own timeout value via its spec.timeoutEvalSeconds attribute. This is not to be confused with spec.timeoutSeconds, used for the Webhook timeout (see section below).

The spec.timeoutEvalSeconds is fine-grained, allowing per-Policy tuning of their evaluation timeout.

This setting takes precedence over the global timeout evaluation configuration per-Policy-Server.

For example, to set a longer evaluation timeout for a specific policy:

apiVersion: policies.kubewarden.io/v1
kind: ClusterAdmissionPolicy
metadata:
  annotations:
    io.kubewarden.policy.category: Secrets
    io.kubewarden.policy.severity: medium
  name: env-variable-secrets-scanner
spec:
  module: registry://ghcr.io/kubewarden/policies/env-variable-secrets-scanner:v1.0.6
  settings: {}
  timeoutEvalSeconds: 10 // Set evaluation timeout
  mutating: false
  rules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      resources: ["pods"]
      operations: ["CREATE"]
    - apiGroups: [""]
      apiVersions: ["v1"]
      resources: ["replicationcontrollers"]
      operations: ["CREATE", "UPDATE"]
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
      operations: ["CREATE", "UPDATE"]
    - apiGroups: ["batch"]
      apiVersions: ["v1"]
      resources: ["jobs", "cronjobs"]
      operations: ["CREATE", "UPDATE"]

Per Policy Server

Starting with Admission Controller v1.5.0, Policy Servers come with a configurable evaluation timeout, enabled by default. Interruption of a request evaluation takes place after 2 seconds. This configuration affects all policies scheduled in that Policy Server. The per-Policy configurable spec.timeoutEvalSeconds timeout has precedence over this per-Policy-Server setting.

You can tune this behavior by using these environment variables:

  • KUBEWARDEN_DISABLE_TIMEOUT_PROTECTION: This disables policy evaluation entirely. Any assigned value turns off the feature.

  • KUBEWARDEN_POLICY_TIMEOUT: This sets a different timeout value. The value is in seconds with a default value of 2.

When using the PolicyServer Kubernetes Custom Resource Definition, you can set these environment variables as follows:

# A Policy Server that has policy evaluation timeout disabled
apiVersion: policies.kubewarden.io/v1
kind: PolicyServer
metadata:
  name: no-policy-timeout
spec:
  env:
    - name: KUBEWARDEN_DISABLE_TIMEOUT_PROTECTION
      value: "true"
---

# A Policy Server that has policy evaluation timeout enabled,
# with a 3 seconds timeout value
apiVersion: policies.kubewarden.io/v1
kind: PolicyServer
metadata:
  name: custom-policy-timeout
spec:
  env:
    - name: KUBEWARDEN_POLICY_TIMEOUT
      value: "3"

Policy memory limit

A policy with a bug, or a malicious policy, can use all the memory of the Policy Server host process. The policy memory limit controls the quantity of linear memory that each policy can use during an evaluation.

The Policy Server applies this limit by default. If you do not configure the limit, each policy can use a maximum of 256 MiB of linear memory. The Policy Server binary applies this default value. Thus the limit is active on all Policy Servers, including the default Policy Server.

The limit applies to all policy runtimes: waPC, Rego, and WASI. You cannot set a different limit for one policy.

You can configure this behavior with these policy-server flags, or with the matching environment variables:

  • --policy-memory-limit / KUBEWARDEN_POLICY_MEMORY_LIMIT: The maximum amount of linear memory, in MiB, that a policy can use. The default value is 256. The value must be more than 0.

  • --disable-memory-limit / KUBEWARDEN_DISABLE_MEMORY_LIMIT: This flag disables the policy memory limit. The environment variable accepts only true or false. The default value is false.

When using the PolicyServer Kubernetes Custom Resource Definition, you can set these environment variables as follows:

# A Policy Server where policies can use up to 512 MiB of linear memory
apiVersion: policies.kubewarden.io/v1
kind: PolicyServer
metadata:
  name: custom-policy-memory-limit
spec:
  env:
    - name: KUBEWARDEN_POLICY_MEMORY_LIMIT
      value: "512"
---

# A Policy Server that has the policy memory limit disabled
apiVersion: policies.kubewarden.io/v1
kind: PolicyServer
metadata:
  name: no-policy-memory-limit
spec:
  env:
    - name: KUBEWARDEN_DISABLE_MEMORY_LIMIT
      value: "true"

If a policy needs more than 256 MiB of linear memory, increase --policy-memory-limit on the PolicyServer that hosts the policy. If a policy uses more memory than the limit, the evaluation fails. The Policy Server continues to run.

The value 0 is not valid for --policy-memory-limit. If you set the value to 0, the Policy Server shows an error and does not start. To disable the limit, use --disable-memory-limit.

kwctl run and kwctl bench accept the same two flags. Thus you can measure the memory use of a policy on your local machine before you deploy the policy.

The WASI runtime also limits the size of the in-memory stdout and stderr pipes. These pipes transfer data between the host and a WASI policy.

A WASI policy reads the admission request and the settings from stdin. The policy writes the validation response, or the mutation response, to stdout. The policy writes only diagnostic messages to stderr.

The limit for stdout is 64 MiB. The limit for stderr is 8 MiB. If a policy writes more data than the limit, the write operation fails and the evaluation fails. The Policy Server host process continues to run.

This limit is different from the policy memory limit.

Comparison with Kubernetes Dynamic Admission Controller timeout

Admission Controller is a webhook implementation of the Kubernetes Dynamic Admission Controller.

Internally, the Kubernetes API server makes an HTTP request to Admission Controller’s Policy Server describing an event that’s about to happen. After the HTTP request, Kubernetes API Server waits for an answer.. However, the Kubernetes API server doesn’t wait forever. After a certain amount of time it considers the request to have timed out.

Quoting the official Kubernetes documentation:

Because webhooks add to API request latency, they should evaluate as quickly as possible. timeoutSeconds allows configuring how long the API server should wait for a webhook to respond before treating the call as a failure.

If the timeout expires before the webhook responds, the webhook call will be ignored or the API call will be rejected based on the failure policy.

The timeout value must be between 1 and 30 seconds.

The timeout for an admission webhook defaults to 10 seconds.

That means that, regardless of the policy evaluation timeout feature, each Kubernetes admission request is subject to a timeout.

Every Admission Controller Policy can set its own timeout value via the timeoutSeconds attribute of the ClusterAdmissionPolicy and AdmissionPolicy custom resources. By default, the timeout value is 10 seconds.

All the Kubernetes admission requests made toa Policy Server are subject to two different timeouts:

  • The Kubernetes API server timeout value. Set to 10 seconds by default, tunable on a per-Policy basis via the dedicated spec.timeoutSeconds attribute on the Policy Custom Resource.

  • The policy evaluation timeout. Set in the Policy Server via environment variables or per-Policy via the spec.timeoutEvalSeconds attribute on Policy Custom Resource.

Now you can examine the following scenarios to better understand the differences between Kubernetes' Webhook timeout and Admission Controller’s policy evaluation timeout.

Admission Controller policy evaluation timeout is disabled

Assume you have a Policy Server that has the policy evaluation timeout feature turned off, and no policy scheduled on it has set their spec.timeoutEvalSeconds field. This Policy Server is hosting a policy affected by a bug which causes it to enter an infinite loop during evaluation.

The Kubernetes API server sends an admission request for evaluation by this buggy policy. As a result, the policy evaluation enters an infinite loop. Meanwhile, the Kubernetes API server is waiting for a response.

After 10 seconds, Kubernetes' webhook timeout takes place, and the request is handled according to the webhook’s failure policy.

Now the Policy Server has computational resources stuck in this infinite loop. Over time, with more admission requests triggering the bugged policy, the Policy Server runs out of computational resources. It’s unable to respond to the Kubernetes API server. This is equal to a Denial Of Service (DOS) attack on the Policy Server. A policy that leaks memory, or that uses a memory spike during an evaluation, can cause the same kind of Denial Of Service through memory exhaustion. See Policy memory limit for the mitigation.

Admission Controller policy evaluation timeout is enabled

Assume a scenario where the same Policy Server now has the policy evaluation timeout feature enabled, either globally in the Policy Server, or in the Policy via the policy spec.timeoutEvalSeconds, and the policy evaluation timeout is 2 seconds.

The Kubernetes API server sends an admission request for evaluation by this buggy policy. As a result, policy evaluation enters an infinite loop. Meanwhile, the Kubernetes API server is waiting for a response.

After two seconds, Admission Controller’s policy evaluation timeout feature interrupts the policy evaluation and produces a rejection response. The response contains a message explaining that rejection happened because the policy evaluation didn’t complete in time.

Setting Admission Controller’s policy evaluation timeout to a value as high as the Kubernetes' webhook timeout isn’t a good choice.

While the policy evaluation is still interrupted, reducing the chances of a DOS attack, the final rejection response isn’t produced by the Policy Server. The rejection comes from the Kubernetes API server with the webhook timeout.

As a result, it’s harder for users, and Kubernetes operators, to detect these slow/buggy policies. The only proof of the policy evaluation interruption is in Policy Server logs and trace events.