Generating Kubernetes ValidatingAdmissionPolicies from Kyverno Policies
In the previous blog post, we discussed writing Common Expression Language (CEL) expressions in Kyverno policies for resource validation. CEL was first introduced to Kubernetes for the Validation rules for CustomResourceDefinitions, and then it was used by Kubernetes ValidatingAdmissionPolicies in 1.26.
ValidatingAdmissionPolicies offer a declarative, in-process alternative to validating admission webhooks.
ValidatingAdmissionPolicies use the Common Expression Language (CEL) to declare the validation rules of a policy. Validation admission policies are highly configurable, enabling policy authors to define policies that can be parameterized and scoped to resources as needed by cluster administrators.
This post will show you how to generate Kubernetes ValidatingAdmissionPolicies and their bindings from Kyverno policies.
Prerequisite
Generating Kubernetes ValidatingAdmissionPolicies require the following:
- A cluster with Kubernetes 1.26 or higher.
- Enable the
ValidatingAdmissionPolicyfeature gate. - Enable the
admissionregistration.k8s.io/v1beta1API for v1.28 and v1.29. OR Enable theadmissionregistration.k8s.io/v1alpha1API for v1.26 and v1.27. - Set the
--generateValidatingAdmissionPolicyflag in the Kyverno admission controller. - Grant the admission controller service account the required permissions to generate ValidatingAdmissionPolicies and their bindings.
In this post, we will use the beta version of Kubernetes 1.29.
Installation & Setup
- Create a local cluster
1kind create cluster --image "kindest/node:v1.28.0" --config - <<EOF
2kind: Cluster
3apiVersion: kind.x-k8s.io/v1alpha4
4featureGates:
5 ValidatingAdmissionPolicy: true
6runtimeConfig:
7 admissionregistration.k8s.io/v1beta1: true
8 admissionregistration.k8s.io/v1alpha1: true
9nodes:
10 - role: control-plane
11 - role: worker
12EOF
- Add the Kyverno Helm repository.
1helm repo add kyverno https://kyverno.github.io/kyverno/
2helm repo update
- Create a new file that overrides the values in the chart.
1cat << EOF > new-values.yaml
2features:
3 generateValidatingAdmissionPolicy:
4 enabled: true
5
6admissionController:
7 rbac:
8 clusterRole:
9 extraResources:
10 - apiGroups:
11 - admissionregistration.k8s.io
12 resources:
13 - validatingadmissionpolicies
14 - validatingadmissionpolicybindings
15 verbs:
16 - create
17 - update
18 - delete
19 - list
20EOF
- Deploy Kyverno
1helm install kyverno kyverno/kyverno -n kyverno --create-namespace --version v3.1.4 --values new-values.yaml
We are now ready to generate Kubernetes ValidatingAdmissionPolicies from Kyverno policies.
Generating Kubernetes ValidatingAdmissionPolicies
In this section, we will create a Kyverno policy that ensures no hostPath volumes are in use for Deployments, and then we will have a look at the generated ValidatingAdmissionPolicy and its binding. Finally, we will create a Deployment that violates the policy.
Let’s start with creating the Kyverno policy.
1kubectl apply -f - <<EOF
2apiVersion: kyverno.io/v1
3kind: ClusterPolicy
4metadata:
5 name: disallow-host-path
6spec:
7 validationFailureAction: Enforce
8 background: false
9 rules:
10 - name: host-path
11 match:
12 any:
13 - resources:
14 kinds:
15 - Deployment
16 validate:
17 cel:
18 expressions:
19 - expression: "!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))"
20 message: "HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset."
21EOF
You can check whether a ValidatingAdmissionPolicy is generated or not from the Kyverno policy status.
1$ kubectl get cpol disallow-host-path -o jsonpath='{.status}'
2
3{
4 "autogen":{
5
6 },
7 "conditions":[
8 {
9 "lastTransitionTime":"2023-09-12T11:42:13Z",
10 "message":"Ready",
11 "reason":"Succeeded",
12 "status":"True",
13 "type":"Ready"
14 }
15 ],
16 "ready":true,
17 "rulecount":{
18 "generate":0,
19 "mutate":0,
20 "validate":1,
21 "verifyimages":0
22 },
23 "validatingadmissionpolicy":{
24 "generated":true,
25 "message":""
26 }
27}
Let’s try getting the ValidatingAdmissionPolicy and its binding.
1$ kubectl get validatingadmissionpolicy
2NAME VALIDATIONS PARAMKIND AGE
3disallow-host-path 1 <unset> 8m12s
4
5$ kubectl get validatingadmissionpolicybindings
6NAME POLICYNAME PARAMREF AGE
7disallow-host-path-binding disallow-host-path <unset> 8m30s
You may notice that the ValidatingAdmissionPolicy and the ValidatingAdmissionPolicyBinding share the same name as the Kyverno policy they originate from, with the binding having a “-binding” suffix.
Let’s have a look at the ValidatingAdmissionPolicy and its binding in detail.
1$ kubectl get validatingadmissionpolicy disallow-host-path -o yaml
2apiVersion: admissionregistration.k8s.io/v1beta1
3kind: ValidatingAdmissionPolicy
4metadata:
5 creationTimestamp: "2023-09-12T11:42:13Z"
6 generation: 1
7 labels:
8 app.kubernetes.io/managed-by: kyverno
9 name: disallow-host-path
10 ownerReferences:
11 - apiVersion: kyverno.io/v1
12 kind: ClusterPolicy
13 name: disallow-host-path
14 uid: e540d96b-c683-4380-a84f-13411384241a
15 resourceVersion: "11294"
16 uid: 9f3e0161-d010-4a6f-bd28-bf9c87151795
17spec:
18 failurePolicy: Fail
19 matchConstraints:
20 matchPolicy: Equivalent
21 namespaceSelector: {}
22 objectSelector: {}
23 resourceRules:
24 - apiGroups:
25 - apps
26 apiVersions:
27 - v1
28 operations:
29 - CREATE
30 - UPDATE
31 resources:
32 - deployments
33 scope: '*'
34 validations:
35 - expression: '!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume,
36 !has(volume.hostPath))'
37 message: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath
38 must be unset.
39 variables: null
40status:
41 observedGeneration: 1
42 typeChecking: {}
1$ kubectl get validatingadmissionpolicybindings disallow-host-path-binding -o yaml
2apiVersion: admissionregistration.k8s.io/v1beta1
3kind: ValidatingAdmissionPolicyBinding
4metadata:
5 creationTimestamp: "2023-09-12T11:42:13Z"
6 generation: 1
7 labels:
8 app.kubernetes.io/managed-by: kyverno
9 name: disallow-host-path-binding
10 ownerReferences:
11 - apiVersion: kyverno.io/v1
12 kind: ClusterPolicy
13 name: disallow-host-path
14 uid: e540d96b-c683-4380-a84f-13411384241a
15 resourceVersion: "11292"
16 uid: 2fec35c3-8a8c-42a7-8a02-a75e8882a01e
17spec:
18 policyName: disallow-host-path
19 validationActions:
20 - Deny
Now, let’s try deploying an app that uses a hostPath:
1kubectl apply -f - <<EOF
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5 name: nginx
6spec:
7 replicas: 2
8 selector:
9 matchLabels:
10 app: nginx
11 template:
12 metadata:
13 labels:
14 app: nginx
15 spec:
16 containers:
17 - name: nginx-server
18 image: nginx
19 volumeMounts:
20 - name: udev
21 mountPath: /data
22 volumes:
23 - name: udev
24 hostPath:
25 path: /etc/udev
26EOF
As expected, the deployment creation is rejected by the API server and not by the Kyverno admission controller.
1The deployments "nginx" is invalid: ValidatingAdmissionPolicy 'disallow-host-path' with binding 'disallow-host-path-binding' denied request: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset.
If either the ValidatingAdmissionPolicy or the binding is deleted/updated for some reason, the controller is responsible for reverting it.
Let’s try deleting the ValidatingAdmissionPolicy.
1$ kubectl delete validatingadmissionpolicy disallow-host-path
2validatingadmissionpolicy.admissionregistration.k8s.io "disallow-host-path" deleted
3
4$ kubectl get validatingadmissionpolicy
5NAME VALIDATIONS PARAMKIND AGE
6disallow-host-path 1 <unset> 11s
In addition, you can update the Kyverno policy, and the controller will re-generate the ValidatingAdmissionPolicy accordingly. For example, you can change the Kyverno policy to match statefulsets too.
patch.yaml:
1spec:
2 rules:
3 - name: host-path
4 match:
5 any:
6 - resources:
7 kinds:
8 - Deployment
9 - StatefulSet
10 validate:
11 cel:
12 expressions:
13 - expression: "!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))"
14 message: "HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset."
1kubectl patch cpol disallow-host-path --type merge --patch-file patch.yaml
The ValidatingAdmissionPolicy will be updated to match StatefulSets too.
1apiVersion: admissionregistration.k8s.io/v1beta1
2kind: ValidatingAdmissionPolicy
3metadata:
4 creationTimestamp: "2023-09-12T12:54:48Z"
5 generation: 2
6 labels:
7 app.kubernetes.io/managed-by: kyverno
8 name: disallow-host-path
9 ownerReferences:
10 - apiVersion: kyverno.io/v1
11 kind: ClusterPolicy
12 name: disallow-host-path
13 uid: e540d96b-c683-4380-a84f-13411384241a
14 resourceVersion: "29208"
15 uid: 9325e2b7-9131-4ff4-9e56-244129cb625e
16spec:
17 failurePolicy: Fail
18 matchConstraints:
19 matchPolicy: Equivalent
20 namespaceSelector: {}
21 objectSelector: {}
22 resourceRules:
23 - apiGroups:
24 - apps
25 apiVersions:
26 - v1
27 operations:
28 - CREATE
29 - UPDATE
30 resources:
31 - deployments
32 - statefulsets
33 scope: '*'
34 validations:
35 - expression: '!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume,
36 !has(volume.hostPath))'
37 message: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath
38 must be unset.
39 variables: null
40status:
41 observedGeneration: 2
42 typeChecking: {}
Conclusion
In this blog, we discussed how to generate Kubernetes ValidatingAdmissionPolicies from Kyverno policies. You can use CEL expressions in Kyverno policies to validate resources through either the Kyverno engine or the API server. In the next blog, we will discuss how to generate BackgroundScan reports for ValidatingAdmissionPolicies.