[Dataset quality] degradedDocs rule

[yngrdyn]

Closes #179173.

Rule type

A dedicated stack rule type was created datasetQuality.degradedDocs.

This new rule is aggregated by default using _index and could be further configured by the user (e.g. user can also aggregate by cloud.provider).

A new rule type was needed to be created since there is no actual way to aggregate all documents in a dataStream if we use a DataView like logs-*-*. Inside datasStream documents there is no indication about the dataStream where they belong to, instead we just have _index which contains backingIndexName instead of actual index.

It's important to note, that this rule type is also visible from Observability > Alerts, which is useful specially for serverless.

Screen.Recording.2025-04-09.at.16.44.56.mov

Role Based Access-Control (RBAC)

RBAC for dataset quality alerts is defined within dataQuality kibana feature.
We have three privileges defined:

1. all: This privilege now contains a subFeature manage_rules that will allow for more granularity on alerting level. It's by default assigned to all but can be disabled.
2. read: This privilege is only related to serverless (when we don't have yet custom roles).
   https://github.com/user-attachments/assets/70ed5bde-bf45-4024-b448-228799fcaf71
3. none: This privilege is only relevant for stateful (in serverless we don't have custom roles).

🎥 Demo

Serverless

all privileges

Screen.Recording.2025-04-09.at.14.30.38.mov

read privileges

Screen.Recording.2025-04-09.at.14.28.31.mov

Stateful

all privileges

Screen.Recording.2025-04-09.at.15.01.24.mov

all privileges with subFeature disabled

Screen.Recording.2025-04-09.at.13.03.31.mov

none privileges (Stateful)

Screen.Recording.2025-04-09.at.14.45.26.mov

How to test?

1. run synthrace scenario degraded_logs in live mode

node scripts/synthtrace degraded_logs --live

2. Open dataset quality page (/app/management/data/data_quality)
3. Select synth.3 dataset (/app/management/data/data_quality/details?pageState=(dataStream:logs-synth.3-default)
4. Click on Actions and then select Create rule
5. Fill out the alert form
6. Go to Observability > Alerts or Stack management > Alerts (/app/observability/alerts)

Release note

Adds the Create alert rule action to dataset quality page and dataset quality details. This allows you to generate an alert when the percentage of degraded docs on the chart crosses a certain threshold.

[yngrdyn]

I didn't test that locally, but do we expect this to work for remote clusters as well? I would guess yes, but just to make sure.

@flash1293 yes, it works for remote clusters

Screen.Recording.2025-06-19.at.09.57.57.mov

[denar50]

Changes related to @elastic/security-detection-engine LGTM. Code review only.

[yngrdyn]

The rule creation form should block the user from moving to the next step if the required fields are not filled out/configured.

Thanks for catching this @fkanout! I was missing to add the validation to errors object c8da5f0.

Screen.Recording.2025-06-19.at.12.27.25.mov

[fkanout]

@yngrdyn, thanks for handling the points I shared. LGTM

[flash1293]

Works as expected, approving because I think it's good for merge but there are some improvements we coud/should make.

Somehow the red highlighted area does not include the very last bucket:

Not sure whether we can control that, but in the "Document summary" of the alert, it doesn't show the grouping fields which are obviously very important:

Should we add that to the summary table? cc @LucaWintergerst

The "Reason" does list the grouping fields, but it doesn't give the field names

[kibanamachine]

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/15820549948

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: d3f24c9

Failed CI Steps

• FTR Configs #119

Test Failures

• [job] [logs] FTR Configs #119 / event_log Event Log public API Legacy Ids "before all" hook for "should support search event by ids and legacyIds"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
alerting	220	221	+1
apm	1925	1926	+1
cases	1022	1023	+1
datasetQuality	443	738	+295
discover	1322	1323	+1
embeddableAlertsTable	408	409	+1
infra	1443	1444	+1
ml	2406	2407	+1
monitoring	631	632	+1
observability	1311	1312	+1
observabilityAIAssistantApp	321	322	+1
observabilityShared	199	200	+1
securitySolution	7670	7671	+1
slo	1183	1184	+1
stackAlerts	292	293	+1
synthetics	1247	1248	+1
timelines	144	145	+1
transform	694	695	+1
triggersActionsUi	876	877	+1
uptime	773	774	+1
total			+314

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/rule-data-utils	183	185	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
datasetQuality	254.3KB	418.0KB	+163.6KB
observability	1.3MB	1.3MB	+162.0B
total			+163.8KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
datasetQuality	20.6KB	23.8KB	+3.3KB
observability	93.6KB	93.6KB	+43.0B
total			+3.3KB

Unknown metric groups

API count

id	before	after	diff
@kbn/rule-data-utils	196	198	+2

async chunk count

id	before	after	diff
datasetQuality	13	16	+3

ESLint disabled line counts

id	before	after	diff
datasetQuality	19	21	+2

miscellaneous assets size

id	before	after	diff
datasetQuality	0.0B	178.3KB	+178.3KB

References to deprecated APIs

id	before	after	diff
datasetQuality	1	2	+1

Total ESLint disabled count

id	before	after	diff
datasetQuality	21	23	+2

History

• 💚 Build #310642 succeeded 211bc70
• 💚 Build #310369 succeeded 1fe4f92
• 💛 Build #310198 was flaky 98b7953
• 💛 Build #310011 was flaky a42813d
• 💔 Build #309880 failed 49853c9
• 💚 Build #309763 succeeded 8b4ec47

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts You might need to backport the following PRs to 8.19: - [Response Ops][Reporting] Scheduled Reports (#221028)

Manual backport

To create the backport manually run:

node scripts/backport --pr 216026

Questions ?

Please refer to the Backport tool documentation

[yngrdyn]

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation