Using CEL Expressions in Kyverno Policies

Using CEL Expressions in Kyverno Policies

Kyverno, in simple terms, is a policy engine for Kubernetes that can be used to describe policies and validate resource requests against those policies. It allows us to create policies for our Kubernetes cluster on different levels. It enables us to validate, change, and create resources based on our defined policies.

A Kyverno policy is a collection of rules. Whenever we receive an API request to our Kubernetes cluster, we validate it with a set of rules.

A policy consists of different clauses, such as:

  • Match: It selects the resources to be included in a rule.
  • Exclude: It selects a subset of the resources from the match block which should be excluded from a rule.

Match and Exclude are used to select resources, users, user groups, service accounts, namespaced roles, and cluster-wide roles.

  • Validate: It validates the properties of the new resource, and it is created if it matches what is declared in the rule.
  • Mutate: It modifies matching resources.
  • Generate: It creates additional resources.
  • Verify Images: It verifies container image signatures using Cosign and Notary.

Refer to Selecting Resources for more information.

Each rule can contain only a single validate, mutate, generate, or verifyImages child declaration.

In this post, I will show you how to write CEL expressions in Kyverno policies for resource validation. Common Expression Language (CEL) was first introduced to Kubernetes for the validation rules for CustomResourceDefinitions, and then it was used by Kubernetes ValidatingAdmissionPolicies in 1.26.

CEL Expressions in validate rules

Creating a policy to disallow host paths for Deployments

The below policy ensures no hostPath volumes are in use for Deployments.

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5  name: disallow-host-path
 6spec:
 7  validationFailureAction: Enforce
 8  background: false
 9  rules:
10    - name: host-path
11      match:
12        any:
13        - resources:
14            kinds:
15              - Deployment
16      validate:
17        cel:
18          expressions:
19            - expression: "!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))"
20              message: "HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset."
21EOF

spec.rules.validate.cel contains CEL expressions that use the Common Expression Language (CEL) to validate the request. If an expression evaluates to false, the validation check is enforced according to the spec.validationFailureAction field.

Now, let’s try deploying an app that uses a hostPath:

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5  name: nginx
 6spec:
 7  replicas: 2
 8  selector:
 9    matchLabels:
10      app: nginx
11  template:
12    metadata:
13      labels:
14        app: nginx
15    spec:
16      containers:
17      - name: nginx-server
18        image: nginx
19        volumeMounts:
20          - name: udev
21            mountPath: /data
22      volumes:
23      - name: udev
24        hostPath:
25          path: /etc/udev
26EOF

We can see that our policy is enforced. Great!

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Deployment/default/nginx was blocked due to the following policies 

disallow-host-path:
  host-path: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath
    must be unset.

Creating a policy to check StatefulSet Namespaces

The below policy ensures that any StatefulSet is created in the production Namespace

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5  name: check-statefulset-namespace
 6spec:
 7  validationFailureAction: Enforce
 8  background: false
 9  rules:
10    - name: statefulset-namespace
11      match:
12        any:
13        - resources:
14            kinds:
15              - StatefulSet
16      validate:
17        cel:
18          expressions:
19            - expression: "namespaceObject.metadata.name == 'production'"
20              message: "The StatefulSet must be created in the 'production' namespace."
21EOF

Let’s try creating a StatefulSet in the default Namespace.

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: StatefulSet
 4metadata:
 5  name: bad-statefulset
 6spec:
 7  replicas: 1
 8  selector:
 9    matchLabels:
10      app: app
11  template:
12    metadata:
13      labels:
14        app: app
15    spec:
16      containers:
17      - name: container2
18        image: nginx
19EOF

As expected, the Statefulset creation is blocked because it violates the rule

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource StatefulSet/default/bad-statefulset was blocked due to the following policies 

check-statefulset-namespace:
  statefulset-namespace: The StatefulSet must be created in the 'production' namespace.

Let’s create a Statefulset in the production Namespace.

 1kubectl apply -f - << EOF
 2apiVersion: apps/v1
 3kind: StatefulSet
 4metadata:
 5  name: good-statefulset
 6  namespace: production
 7spec:
 8  replicas: 1
 9  selector:
10    matchLabels:
11      app: app
12  template:
13    metadata:
14      labels:
15        app: app
16    spec:
17      containers:
18      - name: container2
19        image: nginx
20EOF

The StatefulSet is successfully created. Great!

statefulset.apps/good-statefulset created

In the previous two examples, we have used object in CEL expressions which refers to the incoming object and namespaceObject which refers to the Namespace that the incoming object belongs to.

Some other useful variables that we can use in CEL expressions are

  1. oldObject: The existing object. The value is null for CREATE requests.
  2. authorizer: It can be used to perform authorization checks.
  3. authorizer.requestResource: A shortcut for an authorization check configured with the request resource (group, resource, (subresource), namespace, name).

CEL Preconditions in Kyverno Policies

The below policy ensures the hostPort field is set to a value between 5000 and 6000 for pods whose metadata.name set to nginx

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5  name: disallow-host-port-range
 6spec:
 7  validationFailureAction: Enforce
 8  background: false
 9  rules:
10    - name: host-port-range
11      match:
12        any:
13        - resources:
14            kinds:
15              - Pod
16      celPreconditions:
17          - name: "first match condition in CEL"
18            expression: "object.metadata.name.matches('nginx')"
19      validate:
20        cel:
21          expressions:
22          - expression: "object.spec.containers.all(container, !has(container.ports) || container.ports.all(port, !has(port.hostPort) || (port.hostPort >= 5000 && port.hostPort <= 6000)))"
23            message: "The only permitted hostPorts are in the range 5000-6000."
24EOF

spec.rules.celPreconditions are CEL expressions. All celPreconditions must be evaluated to true for the resource to be evaluated. Therefore, any Pod with nginx in its metadata.name will be evaluated.

Let’s try deploying an Apache server with hostPort set to 80.

 1kubectl apply -f - <<EOF
 2apiVersion: v1
 3kind: Pod
 4metadata:
 5  name: apache
 6spec:
 7  containers:
 8  - name: apache-server
 9    image: httpd
10    ports:
11    - containerPort: 8080
12      hostPort: 80
13EOF

You’ll see that it’s successfully created because the validation rule wasn’t applied on the new Pod as it doesn’t satisfy the celPreconditions. That’s exactly what we need.

Pod/apache created

Let’s try deploying an Nginx server with hostPort set to 80.

 1kubectl apply -f - <<EOF
 2apiVersion: v1
 3kind: Pod
 4metadata:
 5  name: nginx
 6spec:
 7  containers:
 8  - name: nginx-server
 9    image: nginx
10    ports:
11    - containerPort: 8080
12      hostPort: 80
13EOF

Since the new Pod satisfies the celPreconditions, the validation rule will be applied. As a result, the creation of the Pod will be blocked as it violates the rule.

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/default/nginx was blocked due to the following policies 

disallow-host-port-range:
  host-port-range: The only permitted hostPorts are in the range 5000-6000.

Parameter Resources in Kyverno Policies

The below policy ensures the deployment replicas are less than a specific value. This value is defined in a parameter resource.

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5  name: check-deployment-replicas
 6spec:
 7  validationFailureAction: Enforce
 8  background: false
 9  rules:
10    - name: deployment-replicas
11      match:
12        any:
13        - resources:
14            kinds:
15              - Deployment
16      validate:
17        cel:
18          paramKind: 
19            apiVersion: rules.example.com/v1
20            kind: ReplicaLimit
21          paramRef:
22            name: "replica-limit-test.example.com"
23            parameterNotFoundAction: "Deny"
24          expressions:
25            - expression: "object.spec.replicas <= params.maxReplicas"
26              messageExpression:  "'Deployment spec.replicas must be less than ' + string(params.maxReplicas)"
27EOF

The cel.paramKind and cel.paramRef specify the resource used to parameterize this policy. For this example, it is configured by ReplicaLimit custom resources.

The ReplicaLimit could be as follows:

1kubectl apply -f - <<EOF
2apiVersion: rules.example.com/v1
3kind: ReplicaLimit
4metadata:
5  name: "replica-limit-test.example.com"
6maxReplicas: 3
7EOF

Here’s the corresponding custom resource definition:

 1kubectl apply -f - <<EOF
 2apiVersion: apiextensions.k8s.io/v1
 3kind: CustomResourceDefinition
 4metadata:
 5  name: replicalimits.rules.example.com
 6spec:
 7  group: rules.example.com
 8  names:
 9    kind: ReplicaLimit
10    plural: replicalimits
11  scope: Namespaced
12  versions:
13    - name: v1
14      served: true
15      storage: true
16      schema:
17        openAPIV3Schema:
18          type: object
19          properties:
20            apiVersion:
21              type: string
22            kind:
23              type: string
24            metadata:
25              type: object
26            maxReplicas:
27              type: integer
28EOF

Now, let’s try deploying an app with five replicas.

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5  name: nginx
 6spec:
 7  replicas: 5
 8  selector:
 9    matchLabels:
10      app: nginx
11  template:
12    metadata:
13      labels:
14        app: nginx
15    spec:
16      containers:
17      - name: nginx-server
18        image: nginx
19EOF

As expected, the deployment creation will be blocked because it violates the rule.

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Deployment/default/nginx was blocked due to the following policies 

check-deployment-replicas:
  deployment-replicas: Deployment spec.replicas must be less than 3

Let’s try deploying an app with two replicas.

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5  name: nginx
 6spec:
 7  replicas: 2
 8  selector:
 9    matchLabels:
10      app: nginx
11  template:
12    metadata:
13      labels:
14        app: nginx
15    spec:
16      containers:
17      - name: nginx-server
18        image: nginx
19EOF

The deployment is created successfully. Great!

deployment.apps/nginx created

CEL Variables in Kyverno Policies

If an expression grows too complicated, or part of the expression is reusable and computationally expensive to evaluate, We can extract some parts of the expressions into variables. A variable is a named expression that can be referred later as variables in other expressions.

The order of variables is important because a variable can refer to other variables defined before it. This ordering prevents circular references.

The below policy enforces that image repo names match the environment defined in its Namespace. It enforces that all containers of deployment have the image repo match the environment label of its Namespace except for “exempt” deployments or any containers that do not belong to the “example.com” organization (e.g., common sidecars). For example, if the Namespace has a label of {“environment”: “staging”}, all container images must be either staging.example.com/* or do not contain “example.com” at all, unless the deployment has {“exempt”: “true”} label.

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5  name: image-matches-namespace-environment.policy.example.com
 6spec:
 7  validationFailureAction: Enforce
 8  background: false
 9  rules:
10    - name: image-matches-namespace-environment
11      match:
12        any:
13        - resources:
14            kinds:
15              - Deployment
16      validate:
17        cel:
18          variables:
19            - name: environment
20              expression: "'environment' in namespaceObject.metadata.labels ? namespaceObject.metadata.labels['environment'] : 'prod'"
21            - name: exempt
22              expression: "has(object.metadata.labels) && 'exempt' in object.metadata.labels && object.metadata.labels['exempt'] == 'true'"
23            - name: containers
24              expression: "object.spec.template.spec.containers"
25            - name: containersToCheck
26              expression: "variables.containers.filter(c, c.image.contains('example.com/'))"
27          expressions:
28            - expression: "variables.exempt || variables.containersToCheck.all(c, c.image.startsWith(variables.environment + '.'))"
29              messageExpression: "'only ' + variables.environment + ' images are allowed in namespace ' + namespaceObject.metadata.name"
30EOF

Let’s start with creating a Namespace that has a label of environment: staging

1kubectl apply -f - <<EOF
2apiVersion: v1
3kind: Namespace
4metadata:
5  name: staging-ns
6  labels:
7    environment: staging
8EOF

And then create a deployment whose image is example.com/nginx in the staging-ns Namespace.

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5  name: deployment-fail
 6  namespace: staging-ns
 7spec:
 8  replicas: 1
 9  selector:
10    matchLabels:
11      app: app
12  template:
13    metadata:
14      labels:
15        app: app
16    spec:
17      containers:
18      - name: container2
19        image: example.com/nginx
20EOF

As expected, the deployment creation will be blocked since its image must be staging.example.com/nginx

Let’s try setting the deployment image to staging.example.com/nginx instead

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5  name: deployment-pass
 6  namespace: staging-ns
 7spec:
 8  replicas: 1
 9  selector:
10    matchLabels:
11      app: app
12  template:
13    metadata:
14      labels:
15        app: app
16    spec:
17      containers:
18      - name: container2
19        image: staging.example.com/nginx
20EOF

The deployment is created successfully. Great!

deployment.apps/deployment-pass created

Auto-Gen Rules for CEL Expressions

Since Kubernetes has many higher-level controllers that directly or indirectly manage Pods: Deployment, DaemonSet, StatefulSet, Job, and CronJob resources, it’d be inefficient to write a policy that targets Pods and every higher-level controller. Kyverno solves this issue by supporting the automatic generation of policy rules for higher-level controllers from a rule written exclusively for a Pod.

Check the autogen rules for more information.

For example, when creating a validation policy like below, which disallows latest image tags, the policy applies to all resources capable of generating Pods.

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5  name: disallow-latest-tag
 6spec:
 7 validationFailureAction: Enforce
 8  rules:
 9  - name: disallow-latest-tag
10    match:
11      any:
12      - resources:
13          kinds:
14          - Pod
15    validate:
16        cel:
17          expressions:
18            - expression: "object.spec.containers.all(container, !container.image.contains('latest'))"
19              message: "Using a mutable image tag e.g. 'latest' is not allowed."
20EOF

Once the policy is created, these other resources can be shown in auto-generated rules which Kyverno adds to the policy under the status object.

 1status:
 2  autogen:
 3    rules:
 4    - exclude:
 5        resources: {}
 6      generate:
 7        clone: {}
 8        cloneList: {}
 9      match:
10        any:
11        - resources:
12            kinds:
13            - DaemonSet
14            - Deployment
15            - Job
16            - StatefulSet
17            - ReplicaSet
18            - ReplicationController
19        resources: {}
20      mutate: {}
21      name: autogen-disallow-latest-tag
22      validate:
23        cel:
24          expressions:
25          - expression: object.spec.template.spec.containers.all(container, !container.image.contains('latest'))
26            message: Using a mutable image tag e.g. 'latest' is not allowed.
27    - exclude:
28        resources: {}
29      generate:
30        clone: {}
31        cloneList: {}
32      match:
33        any:
34        - resources:
35            kinds:
36            - CronJob
37        resources: {}
38      mutate: {}
39      name: autogen-cronjob-disallow-latest-tag
40      validate:
41        cel:
42          expressions:
43          - expression: object.spec.jobTemplate.spec.template.spec.containers.all(container,
44              !container.image.contains('latest'))
45            message: Using a mutable image tag e.g. 'latest' is not allowed.

Let’s try creating an nginx deployment with the latest tag.

 1kubectl apply -f - << EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5  name: nginx-deployment
 6  labels:
 7    app: nginx
 8spec:
 9  replicas: 3
10  selector:
11    matchLabels:
12      app: nginx
13  template:
14    metadata:
15      labels:
16        app: nginx
17    spec:
18      containers:
19      - name: nginx
20        image: nginx:latest
21EOF

As expected the deployment creation is blocked.

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Deployment/default/nginx-deployment was blocked due to the following policies 

disallow-latest-tag:
  autogen-disallow-latest-tag: Using a mutable image tag e.g. 'latest' is not allowed.

Conclusion

This blog post explains how to use CEL expressions in Kyverno policies to validate resources covering all the features introduced in Kubernetes ValidatingAdmissionPolicies. Stay tuned for our next post, where we’ll show you how to generate Kubernetes ValidatingAdmissionPolicies from Kyverno policies.

Last modified April 08, 2024 at 8:29 AM PST: Refactor links (#1205) (5060f3d)