Scaling Kyverno

Scaling Kyverno

Kyverno supports scaling in multiple dimensions, both vertical as well as horizontal. It is important to understand when to scale, how to scale, and what the effects of that scaling will have on its operation. See the sections below to understand these topics better.

Because Kyverno is an admission controller with many capabilities and due to the variability with respect to environment type, size, and composition of Kubernetes clusters, the amount of processing performed by Kyverno can vary greatly. Sizing a Kyverno installation based solely upon Node or Pod count is often not appropriate to accurately predict the amount of resources it will require.

For example, a large production cluster hosting 60,000 Pods yet with no Kyverno policies installed which match on Pod has no bearing on the resources required by Kyverno. Because webhooks are dynamically managed by Kyverno according to the policies installed in the cluster, no policies which match on Pod results in no information about Pods being sent by the API server to Kyverno and, therefore, reduced processing load.

However, any policies which match on a wildcard ("*") will result in Kyverno being forced to process every operation (CREATE, UPDATE, DELETE, and CONNECT) on every resource in the cluster. Even if the policy logic itself is simple, only a single, simple policy written in such a manner and installed in a large cluster can and will have significant impact on the resources required by Kyverno.

Vertical Scale

Vertical scaling refers to increasing the resources allocated to existing Pods, which amounts to resource requests and limits.

We recommend conducting tests in your own environment to determine real-world utilization in order to best set resource requests and limits, but as a best practice we also recommend not setting CPU limits.

Horizontal Scale

Horizontal scaling refers to increasing the number of replicas of a given controller. Kyverno supports multiple replicas for each of its controllers, but the effect of multiple replicas is handled differently according to the controller. See the high availability section for more details.

Scale Testing

Admission Controller

Kyverno admission controller benchmarks are measured using k6 with the xk6-kubernetes extension. k6’s shared-iterations executor distributes a fixed number of pod CREATE requests across a pool of concurrent virtual users (VUs). All latency values are end-to-end iteration duration measured from the k6 client (API server + webhook round-trip). KWOK fake nodes are used so pod scheduling overhead does not inflate admission latency numbers.

CEL-Based Policies (v1.18.1, policies.kyverno.io/v1)

Tested on Akamai Dedicated g6-dedicated-32 (32 vCPU / 64 GB RAM), KinD 3-worker cluster, KWOK fake nodes, Kyverno v1.18.1 (Helm chart 3.8.1). Load generated from a separate g6-dedicated-16 instance. Latency reported as avg / p95 / p99. Memory and CPU metrics were not captured in this run; use the kyverno_admission_review_duration_seconds Prometheus metric for runtime monitoring.

The vpol-pss scenario installs 16 ValidatingPolicy resources mirroring the Pod Security Standards restricted profile. The combined scenario activates one ValidatingPolicy and one MutatingPolicy simultaneously, exercising both webhook paths per admission request.

replicas	# policies	Policy Type	Scenario	Subject	VUs / Iterations	avg	p95	p99	fail%
1	16	ValidatingPolicy	vpol-pss	Pods	50 / 5,000	57ms	103ms	182ms	0%
1	16	ValidatingPolicy	vpol-pss	Pods	200 / 10,000	156ms	316ms	487ms	0%
1	16	ValidatingPolicy	vpol-pss	Pods	500 / 10,000	366ms	866ms	1,247ms ⚠	0%
3	16	ValidatingPolicy	vpol-pss	Pods	50 / 5,000	57ms	95ms	106ms	0%
3	16	ValidatingPolicy	vpol-pss	Pods	200 / 10,000	145ms	237ms	284ms	0%
3	16	ValidatingPolicy	vpol-pss	Pods	500 / 10,000	321ms	536ms	649ms	0%
1	1	MutatingPolicy	mpol-complex	Pods	200 / 10,000	152ms	221ms	240ms	0%
1	1	MutatingPolicy	mpol-complex	Pods	500 / 10,000	252ms	378ms	427ms	0%
3	1	MutatingPolicy	mpol-complex	Pods	200 / 10,000	152ms	220ms	244ms	0%
3	1	MutatingPolicy	mpol-complex	Pods	500 / 10,000	256ms	386ms	460ms	0%
1	2	combined	vpol + mpol	Pods	200 / 10,000	155ms	295ms	454ms	0%
1	2	combined	vpol + mpol	Pods	500 / 10,000	384ms	777ms	1,113ms ⚠	0%
3	2	combined	vpol + mpol	Pods	200 / 10,000	149ms	241ms	289ms	0%
3	2	combined	vpol + mpol	Pods	500 / 10,000	310ms	491ms	567ms	0%

⚠ p99 > 1 s at single-replica under extreme load (500 concurrent creates). The KyvernoAdmissionHighLatency PrometheusRule alert fires at p99 > 1 s sustained for 5 minutes. With 3 replicas all scenarios remain below 650 ms p99 at 500 VU. Full results including simple/moderate complexity variants and per-VU-level breakdowns are available in docs/perf-testing/v1.18.1/results/summary.md.

Reports Controller

The following table shows the resource consumption (memory and CPU) and objects sizes in etcd of increased workloads. The test was conducted where we installed Kyverno policies to audit the Kubernetes pod security standards using 16 policies. Subsequently, we created workloads and scheduled them on the fake KWOK nodes to measure total size of policy reports in etcd. KWOK is a toolkit that enables setting up a cluster of thousands of Nodes in seconds. For more details on these tests, refer to the testing documentation for the report controller.

# policyreports	total etcd size	CPU (max)	memory (max)
1284	373 MB	3709m	99Mi
2524	381 MB	3729m	102Mi
3572	390 MB	3721m	110Mi
4879	398 MB	3981m	118Mi
6099	418 MB	4014m	124Mi
7319	431 MB	4034m	136Mi
10254	483 MB	4273m	152Mi

AdmissionReview Reference

API requests, operations, and activities which match corresponding Kyverno rules result in an AdmissionReview request getting sent to admission controllers like Kyverno. The number and frequency of these requests may vary greatly depending on the amount and type of activity in the cluster. The following table below is provided to give a sense of how many minimum AdmissionReview requests may result from common operations. These figures only refer to the minimum number and, in actuality, the final count will almost certainly be greater but varies depending on things like finalizers and other controllers in the cluster.

Operation	Resource	Config	ARs
CREATE	Pod		1
DELETE	Pod		3
CREATE	Deployment	replicas=1	3
UPDATE	Deployment	Change image	8
DELETE	Deployment	replicas=1	7
CREATE	Deployment	replicas=2	4
UPDATE	Deployment	Change image	13
DELETE	Deployment	replicas=2	10
CREATE	Job	restartPolicy=Never, backoffLimit=4	3
DELETE	Job		4
CREATE	CronJob	schedule=“_/1 _ * * *”	4 (3 per invocation)
DELETE	CronJob	schedule=“_/1 _ * * *”, 2 completed	9
CREATE	ConfigMap		1
EDIT	ConfigMap		1
DELETE	ConfigMap		1

These figures were captured using K3d v5.4.9 on Kubernetes v1.26.2 and Kyverno 1.10.0-alpha.2 with a 3-replica admission controller. When testing against KinD, there may be one less DELETE AdmissionReview for Pod-related operations.

Kubernetes api-server and etcd resource footprint

In clusters with many Kyverno policy resources, the resource footprint of some core Kubernetes components may be affected, in particular the kube-apiserver and etcd.

For example, if you create several thousand Kyverno policy resources, double check that the kube-apiserver pods have head room to increase its memory allocations, otherwise the cluster may crash entirely.