add new field `related_integrations` to the post build by Mikaayenson · Pull Request #2060 · elastic/detection-rules

Mikaayenson · 2022-06-27T20:35:00Z

Issues

Resolves #2057

Summary

Adds the new rule field related_integrations to the rule post build.
Sets the value to None as a default.

detection_rules/rule.py

…rm-build-process

terrancedejesus

LGTM.

…rm-build-process

…ocess' of github.com:elastic/detection-rules into 2057-add-related_integrationsto-post-transform-build-process

terrancedejesus · 2022-07-05T19:43:06Z

Attempting to identify what potential routes are available for us here, this is what I discussed with Mika. @brokensound77 thoughts?

For package/integration

As discussed we focus on event.dataset and the field associated with that. We should be able to pull these from self.data.ast similar to how fields is pulled.

Versioning

The only clear linkage I am aware of is comparing stack versions from the manifest of each package to that our our own packages.yml and finding the least common compatible version through iteration. Instead of making network calls for this we can build a local JSON reference file that is built via separate command. A simple process would be...

Our rules backport to earliest supported stack version and packages.yml is referenced
Packages.yml contains the semantic version of the supported stack so we know the rule is validated and backported to the appropriate branches in Detection Rules, we can rely on this to then ask the question "what is the earliest supported package:integration that supports this stack version"
We create a separate dev command to download packages from elastic-package (https://github.com/elastic/package-storage/tree/production/packages), specifically prod branch and builds a local dictionary file for reference. We run this command during "prep for X.X branch process" or every release we do.
This JSON file is used to match our earliest supported branch for the rule to the kibana.version of each package/version/manifest.yml.
Our logic determines, if this rule's earliest support (stack based) is within the same window as the kibana.version from the manifest.yml then it is compatible.
We iterate through every package version until we find one that is not compatible in which we know the package version before is what we want and is the earliest compatible.

…rm-build-process

brokensound77 · 2022-07-06T03:48:23Z

We should definitely create a new py file called integrations and store the logic there to build the local manifests (for any referenced integrations. (we will also eventually store the integrations schema code there as well).

Parsing should come from the query AST event.dataset (and we can unit test against the manifests for those as well for package and integration).

Version compatibility should be:

get min_stack_version
find lowest manifest for that integration and use the integration version

brokensound77 · 2022-07-06T03:49:18Z

Before continuing too far, I would merge #2059 to verify a true diff and avoid divergent errors from code duplication

…of dictionaries, started work on integrations py

terrancedejesus · 2022-07-06T20:13:16Z

Adding some current progress for this before I turn my attention to a separate and potentially more urgent issue.

At this time, add_related_integrations will call get_packaged_integrations and return a list of dictionaries based on the parsed data from event.dataset in the ast field as shown below.

[{'package': 'azure', 'integration': 'activitylogs'}, {'package': 'azure', 'integration': 'auditlogs'}, {'package': 'o365', 'integration': 'audit'}]

Next steps are to build out the integrations.py file, IntegrationPackages class and additional methods to load a integration-manifests.yaml file locally during instantiation as well as load the current package version. These will then be used during the find_least_compatible_version method call to find the least compatible version and return that as a string. This should then create finalized dictionary object with the correct and expected format.

A separate method is being built, build_integrations_manifest, to build the integration-manifests.yaml file and save it for use where we may add a simple dev command to do the process ad-hoc.

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

…rm-build-process

detection_rules/rule.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

…rm-build-process

brokensound77 · 2022-08-08T16:53:54Z

detection_rules/integrations.py

+        for kibana_compat_vers in re.sub(r"\>|\<|\=|\^", "", manifest["conditions"]["kibana.version"]).split(" || "):
+            if compare_versions(kibana_compat_vers, current_stack_version):
+                return version
+    raise Exception(f"no compatible version for integration {package}:{integration}")


going forward, we may just want to return None and not set the integration rather than raising an exception, but we can punt this for now

adjusted in recent commit. Method expects Union[str, None]. If no compatible version, it will print instead of raise the error and then return None. Had to adjust rule.add_related_integrations code so when policy templates are checked, it only does it if version exists or it would error.

brokensound77

Left a few comments on some previous suggestions that got missed, otherwise LGTM. Thanks for the patience with reviews!!

…nstead of raise error only

detection_rules/integrations.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* add new field `related_integrations` to the post build * add exception for endpoint `integration` * Skip rules without related integrations * lint * refactor related_integrations to TOMLRuleContents class * update to reflect required_fields updates * add todo * add new line for linting * related_integrations updates, get_packaged_integrations returns list of dictionaries, started work on integrations py * build_integrations_manifest command completed * initial test completed for post-building related_integrations * removed get_integration_manifest method from rule, removed global integrations path * moved integration related methods to integrations.py and fixed flake issues * adjustments for PipedQuery from eql sequence rules and packages with no integration * adjusted github client import for integrations.py * Update detection_rules/devtools.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/devtools.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * added integration manifest schema, made adjustments * Update detection_rules/integrations.py * Update detection_rules/rule.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/rule.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/rule.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * removed get_integrations_package to consolidate code * removed type list return * adjusted import flake errors * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * adjusted indentation error * adjusted rule.get_packaged_integrations to account for kql.ast.OrExpr if event.dataset is not set * Update detection_rules/devtools.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/devtools.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * adjusted find_least_compatible_version in integrations.py * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * fixed flake issues * adjusted get_packaged_integrations * iterate the ast for literal event.dataset values * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * Update detection_rules/integrations.py Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> * made small adjustments to address errors during build manifests command * addressing integrations.find_least_compatible method to return None instead of raise error only * Update detection_rules/integrations.py Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com> Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com> Co-authored-by: Terrance DeJesus <terrance.dejesus@elastic.co> Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com> (cherry picked from commit 7d973a3)

add new field related_integrations to the post build

2811249

Mikaayenson added python Internal python for the repository v8.4.0 labels Jun 27, 2022

Mikaayenson requested review from brokensound77 and terrancedejesus as code owners June 27, 2022 20:35

Mikaayenson assigned Mikaayenson and terrancedejesus Jun 27, 2022

github-actions bot added the backport: auto label Jun 27, 2022

Mikaayenson added 3 commits June 27, 2022 16:47

add exception for endpoint integration

1381e11

Skip rules without related integrations

9d43b5b

lint

21b1a8e

Mikaayenson commented Jun 27, 2022

View reviewed changes

detection_rules/rule.py Outdated Show resolved Hide resolved

Mikaayenson marked this pull request as draft June 28, 2022 20:36

refactor related_integrations to TOMLRuleContents class

0f4f88e

Mikaayenson marked this pull request as ready for review June 29, 2022 01:29

Merge branch 'main' into 2057-add-related_integrationsto-post-transfo…

9982dc2

…rm-build-process

terrancedejesus approved these changes Jun 29, 2022

View reviewed changes

Mikaayenson and others added 5 commits June 30, 2022 15:49

Merge branch 'main' into 2057-add-related_integrationsto-post-transfo…

62c8aa2

…rm-build-process

update to reflect required_fields updates

6ae5318

Merge branch '2057-add-related_integrationsto-post-transform-build-pr…

2dbff89

…ocess' of github.com:elastic/detection-rules into 2057-add-related_integrationsto-post-transform-build-process

add todo

f4e39f6

add new line for linting

7d25987

Merge branch 'main' into 2057-add-related_integrationsto-post-transfo…

fa8fc84

…rm-build-process

terrancedejesus added 2 commits July 6, 2022 12:08

added changes from required_fields PR

f53f9d7

related_integrations updates, get_packaged_integrations returns list …

d818c57

…of dictionaries, started work on integrations py

build_integrations_manifest command completed

6dfde72

terrancedejesus and others added 7 commits August 4, 2022 09:25

Update detection_rules/integrations.py

d57f5dc

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

adjusted find_least_compatible_version in integrations.py

684292d

Update detection_rules/integrations.py

ef58a09

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

fixed flake issues

1e6efa1

Merge branch 'main' into 2057-add-related_integrationsto-post-transfo…

ca3a082

…rm-build-process

adjusted get_packaged_integrations

ee8ac61

iterate the ast for literal event.dataset values

986f19b

brokensound77 reviewed Aug 4, 2022

View reviewed changes

detection_rules/rule.py Show resolved Hide resolved

brokensound77 reviewed Aug 4, 2022

View reviewed changes

detection_rules/rule.py Show resolved Hide resolved

brokensound77 reviewed Aug 4, 2022

View reviewed changes

detection_rules/rule.py Show resolved Hide resolved

terrancedejesus and others added 7 commits August 4, 2022 18:43

Update detection_rules/integrations.py

1dcbff8

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Update detection_rules/integrations.py

fd26283

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Update detection_rules/integrations.py

e6230cb

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Update detection_rules/integrations.py

481d233

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

made small adjustments to address errors during build manifests command

0de0c45

Merge branch 'main' into 2057-add-related_integrationsto-post-transfo…

825132a

…rm-build-process

Merge branch 'main' into 2057-add-related_integrationsto-post-transfo…

565b45b

…rm-build-process

brokensound77 reviewed Aug 8, 2022

View reviewed changes

brokensound77 approved these changes Aug 8, 2022

View reviewed changes

addressing integrations.find_least_compatible method to return None i…

abeccd0

…nstead of raise error only

Mikaayenson commented Aug 8, 2022

View reviewed changes

detection_rules/integrations.py Outdated Show resolved Hide resolved

Update detection_rules/integrations.py

2537d07

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

terrancedejesus merged commit 7d973a3 into main Aug 8, 2022

terrancedejesus deleted the 2057-add-related_integrationsto-post-transform-build-process branch August 8, 2022 17:44

Mikaayenson mentioned this pull request Aug 8, 2022

[Bug] related_integration added to all rule versions #2234

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add new field `related_integrations` to the post build#2060

add new field `related_integrations` to the post build#2060
terrancedejesus merged 64 commits intomainfrom
2057-add-related_integrationsto-post-transform-build-process

Mikaayenson commented Jun 27, 2022 •

edited

Loading

Uh oh!

Uh oh!

terrancedejesus left a comment

Uh oh!

terrancedejesus commented Jul 5, 2022 •

edited

Loading

Uh oh!

brokensound77 commented Jul 6, 2022

Uh oh!

brokensound77 commented Jul 6, 2022

Uh oh!

terrancedejesus commented Jul 6, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brokensound77 Aug 8, 2022

Uh oh!

terrancedejesus Aug 8, 2022

Uh oh!

brokensound77 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Mikaayenson commented Jun 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issues

Summary

Uh oh!

Uh oh!

terrancedejesus left a comment

Choose a reason for hiding this comment

Uh oh!

terrancedejesus commented Jul 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

For package/integration

Versioning

Uh oh!

brokensound77 commented Jul 6, 2022

Uh oh!

brokensound77 commented Jul 6, 2022

Uh oh!

terrancedejesus commented Jul 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brokensound77 Aug 8, 2022

Choose a reason for hiding this comment

Uh oh!

terrancedejesus Aug 8, 2022

Choose a reason for hiding this comment

Uh oh!

brokensound77 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Mikaayenson commented Jun 27, 2022 •

edited

Loading

terrancedejesus commented Jul 5, 2022 •

edited

Loading

terrancedejesus commented Jul 6, 2022 •

edited

Loading