Using Pod Security Admission with Kubewarden

Pod Security Policies (PSP) are removed since the Kubernetes 1.25 release. They’re replaced by the Pod Security Admission (PSA).

PSA simplifies securing the Pods in Kubernetes clusters.

PSA has three profiles (described in Pod Security Standards) for namespaces:

privileged, providing the widest range of permissions
baseline, to prevent new privilege escalations
restricted, restricted to harden Pods

A PSA controller performs actions on violation detection. The actions are: enforce, audit, and warn. They can be configured.

At the time of writing, with Kubernetes 1.28, the PSA controller has the following limitations:

No mutation capabilities
Higher level objects (like Deployment, Job) are evaluated only when the audit or warn modes are enabled

Kubewarden can be used to integrate a PSA profile to avoid these limitations.

You could use Kubewarden to replace the old PSP configuration as shown in PSP migration. However, the goal of this article is to show how Kubewarden can complement the new PSA.

Example

In this example we’re creating a namespace and applying restrictive PSA policies:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.25
EOF

This PSA profile doesn’t allow creating containers that run their application as the root user. When defining this container:

the runAsNonRoot attribute must be set to true
the runAsUser one can’t be set to 0.

So, the following resource won’t reach its desired state:

kubectl command configuring a resource with the highlighted runAsUser: 0

shell kubectl apply -n my-namespace -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 1 selector: matchLabels: app: template-nginx template: metadata: labels: app: template-nginx spec: containers: - name: nginx image: nginx:1.14.2 securityContext: runAsNonRoot: true // highlight-next-line runAsUser: 0 allowPrivilegeEscalation: false capabilities: drop: - "ALL" seccompProfile: type: "RuntimeDefault" ports: - containerPort: 80 EOF

If we check the deployment, we can see the PSA prevents the pod creation:

kubectl get deploy -n my-namespace nginx-deployment -o json | jq ".status.conditions[] | select(.reason == \"FailedCreate\")"
{
  "lastTransitionTime": "2022-10-28T19:09:56Z",
  "lastUpdateTime": "2022-10-28T19:09:56Z",
  "message": "pods \"nginx-deployment-5f98b4db8c-2m96l\" is forbidden: violates PodSecurity \"restricted:v1.25\": runAsUser=0 (container \"nginx\" must not set runAsUser=0)",
  "reason": "FailedCreate",
  "status": "True",
  "type": "ReplicaFailure"
}

You can fix this by removing the runAsUser: 0 from the container definition:

kubectl command configuring a resource without runAsUser: 0

shell kubectl apply -n my-namespace -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 1 selector: matchLabels: app: template-nginx template: metadata: labels: app: template-nginx spec: containers: - name: nginx image: nginx:1.14.2 securityContext: runAsNonRoot: true allowPrivilegeEscalation: false capabilities: drop: - "ALL" seccompProfile: type: "RuntimeDefault" ports: - containerPort: 80 EOF

Now PSA allows an attempt at pod creation but it still fails.

kubectl get pods -n my-namespace
NAME                                READY   STATUS                       RESTARTS   AGE
nginx-deployment-57d8568bbb-h4bx7   0/1     CreateContainerConfigError   0          47s

It’s because the container definition didn’t specify a user to be used when starting a program inside the container. The default is to run as the root user if this is the case. That’s not allowed by the runAsNonRoot directive:

kubectl get pods -n my-namespace nginx-deployment-57d8568bbb-h4bx7 -o json | jq ".status.containerStatuses"
[
  {
    "image": "nginx:1.14.2",
    "imageID": "",
    "lastState": {},
    "name": "nginx",
    "ready": false,
    "restartCount": 0,
    "started": false,
    "state": {
      "waiting": {
        "message": "container has runAsNonRoot and image will run as root (pod: \"nginx-deployment-57d8568bbb-8mvkc_my-namespace(add7bcc5-3d23-43d0-94e9-6e78f887a53f)\", container: nginx)",
        "reason": "CreateContainerConfigError"
      }
    }
  }
]

This is where Kubewarden can help. You can use the user-group-policy policy to mutate the Deployment definition. This configures a default user for containers omitting that information.

You need the Kubewarden stack in the Kubernetes cluster for this example. See the QuickStart for more details.

It’s possible to enforce a user ID range, for example, 1000—2000 and 4000—5000:

kubectl command enforcing a user id range

kubectl apply -f - <<EOF
apiVersion: policies.kubewarden.io/v1
kind: ClusterAdmissionPolicy
metadata:
  name: user-group-psp
spec:
  policyServer: default
  module: registry://ghcr.io/kubewarden/policies/user-group-psp:latest
  rules:
  - apiGroups: ["", "apps"]
    apiVersions: ["v1"]
    resources: ["pods", "deployments"]
    operations:
    - CREATE
    - UPDATE
  mutating: true
  settings:
    run_as_user:
      rule: "MustRunAs"
      overwrite: false
// highlight-start
      ranges:
        - min: 1000
          max: 2000
        - min: 4000
          max: 5000
// highlight-end
    run_as_group:
      rule: "RunAsAny"
    supplemental_groups:
      rule: "RunAsAny"
EOF

Check the policy is active before continuing:

kubectl get clusteradmissionpolicy.policies.kubewarden.io/user-group-psp

When the policy is active, re-create the deployment:

kubectl command recreating the deployment

kubectl delete deployment -n my-namespace nginx-deployment && \
kubectl apply -n my-namespace -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: template-nginx
  template:
    metadata:
      labels:
        app: template-nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        securityContext:
          runAsNonRoot: true
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - "ALL"
          seccompProfile:
            type: "RuntimeDefault"

        ports:
        - containerPort: 80
EOF

Now the deployment is mutated by Kubewarden’s policy which allows the Pod to be started. The container defined inside the Pod has a default runAsUser value:

kubectl get pods -n my-namespace nginx-deployment-57d8568bbb-nv8fj -o json | jq ".spec.containers[].securityContext"
{
  "allowPrivilegeEscalation": false,
  "capabilities": {
    "drop": [
      "ALL"
    ]
  },
  "runAsNonRoot": true,
  "runAsUser": 1000,
  "seccompProfile": {
    "type": "RuntimeDefault"
  }
}

The Kubewarden integration can do more in this scenario. It can check the value of the runAsUser provided.

This resource is rejected by the Kubewarden policy from earlier:

kubectl command to show resource rejection

kubectl apply -n my-namespace -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment2
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: template-nginx
  template:
    metadata:
      labels:
        app: template-nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        securityContext:
          runAsNonRoot: true
// highlight-next-line
          runAsUser: 7000
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - "ALL"
          seccompProfile:
            type: "RuntimeDefault"
        ports:
        - containerPort: 80
EOF

It’s rejected because the runAsUser value is set to 7000, which is outside the ranges allowed by the policy:

kubectl get deploy -n my-namespace nginx-deployment -o json | jq ".status.conditions[] | select(.reason == \"FailedCreate\")"
{
  "lastTransitionTime": "2022-10-28T19:22:04Z",
  "lastUpdateTime": "2022-10-28T19:22:04Z",
  "message": "admission webhook \"clusterwide-user-group-psp.kubewarden.admission\" denied the request: User ID outside defined ranges",
  "reason": "FailedCreate",
  "status": "True",
  "type": "ReplicaFailure"
}

Summary

PSA provides an easy way to secure Kubernetes clusters. The main goal of PSA is simplicity and it doesn’t have the power and flexibility of the earlier PSP.

Using Kubewarden together with PSA helps fill this gap.