etcdserver: request is too large

[chen-keinan]

It has been observed that in some cases the report produced by trivy-operator is hitting the default etcd request limit and fails.

The reason for report getting too big is due to amount of vulnerabilities (found in image) and it associated data stored in the report.

Workaround for this issue is to tune etcd request limit

There are three potential solution for this issue:

• Reduce the amount of vulnerability data ,fields , stored in the report. the report refer to Aqua AVD so any data that can be found in reference can be omitted from report Reduce the amount of vulnerability fields stored in the vulnerabilities report #442
• The report stored a lot of repetitive data , maybe we need to restructure the report to reduce the amount of repetitive data which will end out with smaller report Remove vulnerability report repetitive data #443
• use EndpointSlice , sharding, and split the report to smaller reports use EndpointSlice for splitting the vulnerability report to smaller reports #445

[chary1112004]

@chen-keinan we also see the issue in the log. Do you have any update to resolve this issue?

{"level":"error","ts":"2024-01-17T00:43:52Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-7c4d69b694","namespace":"trivy-system"},"namespace":"trivy-system","name":"scan-vulnerabilityreport-7c4d69b694","reconcileID":"7a07aa93-7c7e-4530-9c54-8d3f0d10f24b","error":"etcdserver: request is too large","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

[chen-keinan]

not really , its not an easy one, the only workaround I can think of is increasing the the request limit in etcd

[chary1112004]

not really , its not an easy one, the only workaround I can think of is increasing the the request limit in etcd

For increasing the request limit in etcd, I think it only works for kubernetes installed self nodes, not kubernetes provided by cloud provider since master nodes are managed by cloud provider.

[AndriiBurenko]

@chen-keinan Hi! I have the same problem. We use AWS EKS and can't change request limit in etcd. Do you have any update to resolve this issue?

We use:
Trivy-Operator version (helm-chart): 0.19.1
Kubernetes version (use kubectl version): 1.28.0

[lindsaygrace]

@chen-keinan Its a breaking change, but perhaps there can be a Vulnerability resource that trivy-operator maintains for each unique discovered vulnerability. That VulnerabilityReport references and includes some key details. This way fields that take up a lot of space like the extra links are kept to their own resource.

[chen-keinan]

@lindsaygrace could be , another option is to compress the reported data and encode it (save it to crd body), but it will not be human readable.

[Orrimp]

not really , its not an easy one, the only workaround I can think of is increasing the the request limit in etcd

We have our own Kubernetes Cluster but increasing the request limit could create another bunch of problems. The increased lag would compromise the stability of the cluster. Therefore we need to decrease the amount of information in the the report. I like the idea of @lindsaygrace to store each found vulnerability seperataly. We have multiple identical CVEs in os and library (java)

[alanrichman]

@chen-keinan We have the Trivy Operator deployed on a handful of GKE clusters and are encountering this issue as well. Ideally we would love to see a way to persist the data outside of the cluster to avoid etcd size limits entirely, but we are open to any other suggestions for a workaround.

[cthtrifork]

@lindsaygrace could be , another option is to compress the reported data and encode it (save it to crd body), but it will not be human readable.

If there is a way to hook into the serializer, we could enable a config flag for this and just encode the data in base64 (encryption does not seem necessary). If you could point me in the right direction, I could take a look at it.

[afdesk]

this issue still appeares in tests: https://github.com/aquasecurity/trivy-operator/actions/runs/12045796081/job/33585312426?pr=2305

[jemag]

Issue went stale, but are policyreports still considered?
#2003
Would be nice to use something that is more general to the k8s ecosystem and used already by other projects.

[afdesk]

Issue went stale, but are policyreports still considered? #2003 Would be nice to use something that is more general to the k8s ecosystem and used already by other projects.

I've reopened the issue

[oha95]

a slightly messy solution is to clone the project and modify the vulnerability report before saving it in order to reduce the size by removing the vulnerabilities from the lowest to the highest, for example add the following code:

// Maximum size for vulnerability reports to avoid "etcdserver: request is too large" errors
const MaxReportSize = 1 * 1024 * 1024 // 1MB in bytes

// LimitVulnerabilitiesSize limits the number of vulnerabilities in the report data
// by removing the lowest severity vulnerabilities first to ensure the report
// stays under the maximum size limit (1MB by default)
func LimitVulnerabilitiesSize(reportData *v1alpha1.VulnerabilityReportData) {
	// First, sort vulnerabilities by severity (highest to lowest)
	vulnerabilities := Vulnerabilities(reportData.Vulnerabilities)
	sort.Sort(BySeverity{vulnerabilities})
	
	// Store the sorted vulnerabilities back
	reportData.Vulnerabilities = vulnerabilities
	
	// Check if the current size exceeds our limit
	for {
		// Marshal the report to check its size
		reportBytes, err := json.Marshal(reportData)
		if err != nil || len(reportBytes) <= MaxReportSize {
			break
		}
		
		// If report is too large, remove vulnerabilities with lowest severity
		vulnsLen := len(reportData.Vulnerabilities)
		if vulnsLen == 0 {
			break
		}
		
		// Find the first index to remove (starting from the end - lowest severity)
		removeIndex := vulnsLen - 1
		severity := reportData.Vulnerabilities[removeIndex].Severity
		
		// Find how many vulnerabilities with this severity we should remove
		// Remove up to 25% of the vulnerabilities with this severity at once
		countToRemove := 0
		for i := removeIndex; i >= 0; i-- {
			if reportData.Vulnerabilities[i].Severity == severity {
				countToRemove++
			} else {
				break
			}
		}
		
		// Remove at least one but no more than 25% at once
		removeCount := max(1, countToRemove/4)
		newSize := vulnsLen - removeCount
		
		// Update the vulnerabilities list
		reportData.Vulnerabilities = reportData.Vulnerabilities[:newSize]
		
		// Update the summary counts
		reportData.Summary = vulnerabilitySummary(reportData.Vulnerabilities)
	}
}

// Helper function to get max of two integers
func max(a, b int) int {
	if a > b {
		return a
	}
	return b
}

and call this function in “pkg/vulnerabilityreport/io.go” at the beginning of the function “createOrUpdate”, after which simply compile the code and build the image and use it in the helm chart "image"

[itaysk]

converting this to a discussino as this isn't actionable right now

[remram44]

A system-breaking defect is not an issue if it "isn't actionable right now"?

You post that right after someone provided a workaround that you didn't comment on, too.

[itaysk]

Issues are reserved for actionable items: confirmed issues that has an agreed upon resolution and a plan for implementation. This particular issue is more of a discussion: some user reports, and some suggested workarounds. there are multiple ways to address this and when there's commitment to work on one, an issue would be created. for example, see #2571

[Nemric]

Hi,
I did follow this pr : #2545 that became this commit 454d8c8
I came here because my home lab only have a small control plane, and trivy operator fill near all its resources
well, just want to mention that this great work looks like a really suitable solution :)

[nickinkaty]

@chen-keinan I was wondering if it was possible to configure etcd location to store reports?

[mblaschke-daimlertruck]

can confirm the issue for AKS (Azure Kubernetes service) with no possibility to increase etcd request size.

[afdesk]

you can try to use alternateReportStorage:

trivy-operator/deploy/helm/values.yaml

Lines 825 to 835 in 10390b5

	# -- alternateReportStorage is the flag to enable alternate storage for all trivy reports (crds) in the form json files inside of a persistent volume
	alternateReportStorage:
	enabled: false
	mountPath: "/mnt/data/trivy-operator"
	volumeName: "trivy-operator-pvc"
	storage: "10Gi"
	# -- storageClassName for the PVC (optional, uses cluster default if not specified)
	storageClassName: ""
	podSecurityContext:
	runAsUser: 10000
	fsGroup: 10000

it's a community driven feature for such cases.