[Security Solution][Detections] Related Integrations & Required Fields Feedback & Fixes

[spong]

Summary

Addressing the following feedback from #132847:

• On the Rule Management page package name is used instead of package title when the package has only 1 integration:
• move integrations_popover to related_integrations directory
• update useInstalledIntegrations to always return array of installedIntegration
• move useInstalledIntegrations to related_integrations directory
• Slight update to copy in Rules Table popover
• Ok to use Rule Details UI within Rules Table popover content
• Sort integrations alphabetically
• Update left padding on version mis-match tooltip
• [Security Solution] Incorrect icon is displaying for numeric type under Required fields #133291
• [Security Solution]Dash/none should be shown under rule detail for Related integration and Required Field #133269
• [Security Solution] Red banner error is displaying when navigate to integration URL from related integration #133600
• Add Kibana Advanced Setting for disabling integrations badge on Rules Table

• Adds tests
  • useInstalledIntegrations hook
  • relatedIntegrations utils
  • IntegrationDescription
• Add loaders where necessary since there can now be API delay
  • May hold off on loaders as transition from no installed integrations -> installed integrations isn't too bad as-is

Updated integrations popover content on Rules Table to match Rule Details design

In discussions with @banderror reviewing the different integration states (uninstalled, installed, enabled, and agents enrolled), we are now capturing the distinction between Installed and Enabled so that we don't confuse users when a package is installed but the integration isn't enabled/configured. I also added tooltips for clarifying each state and what action the user should perform to ensure compatibility. In collab with @yiyangliu9286 @jethr0null (comments below) -- we've consolidated to a single Installed: enabled badge, and updated Uninstalled to Not installed as well.

Tooltips

Not installed

Installed

Installed: enabled

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
  • Collaborating with docs teams on this dedicated docs issue: [DOCS] Update documentation to include the new Rule fields: Related Integrations, Required Fields and Setup security-docs#2015
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[yiyangliu9286]

I agree with the badge colour usage here to be consistent with what we use in ML Job enabled/disabled UI. On the different statuses itself though I wonder if it makes sense to combine the Installed and Enabled as Installed: enabled rather than two badges since the Tooltip content is similar. So we have 3 different status in total.

1. Uninstalled Show tooltip:

2. Installed Show tooltip:

3. Installed: enabled Show tooltip
   Integration is installed. An integration policy with the required configuration exists. Ensure Elastic Agents are assigned this policy to ingest compatible events.

[jethr0null]

@spong I’m on board with @yiyang’s suggestions as well. My only other thought/question is whether it makes sense to use Available instead of Uninstalled to avoid any ambiguity (e.g. uninstalled being interpreted as “was once installed and has since been removed”. I don't feel strongly about changing this if everyone else agrees that uninstalled makes more sense.

[spong]

Thanks for the feedback @yiyangliu9286 and @jethr0null! 🙂

I've combined the Installed and Enabled badges, updated Uninstalled to Not installed (from our other discussion), and updated the badge colors such that Installed is the green success, and Not installed is the muted gray. I've updated the images in the description, so please check those out to confirm these changes 👍 .

[banderror]

While testing locally, found a few bugs. Will post these right away and continue testing and reviewing the code.

Version mismatch icon's color is not readable

When the dark mode is off:

Links include a semver requirement instead of a target version

Looks like it happens for installed integrations, and works properly for those that are not installed when there's a version mismatch. Both pages are affected.

ndjson for repro:

{"id":"6cc39c80-da3a-11ec-9fce-65c1a0bee900","updated_at":"2022-05-23T01:48:23.422Z","updated_by":"elastic","created_at":"2022-05-23T01:48:20.940Z","created_by":"elastic","name":"Rule with Related Integrations, Required Fields, and Setup Guide","tags":["Test"],"interval":"5m","enabled":false,"description":"See Related Integrations, Required Fields, and Setup Guide on the Rule Details and Rule Management pages.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":".siem-signals-default","meta":{"from":"5m"},"rule_name_override":"message","timestamp_override":"event.ingested","author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"2c66bf23-6ae9-4eb2-859e-446bea181ae0","max_signals":10000,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":7,"exceptions_list":[],"immutable":false,"related_integrations":[{"package":"system","version":"~1.6.4"},{"package":"aws","integration":"cloudfront","version":"~1.11.0"},{"package":"aws","integration":"cloudtrail","version":"~1.11.0"},{"package":"aws","integration":"route53","version":"~1.11.0"},{"package":"unknown","integration":"unknown","version":"^1.2.3"},{"package":"unknown2","version":"4.5.6"}],"required_fields":[{"ecs":true,"name":"event.code","type":"keyword"},{"ecs":true,"name":"message","type":"match_only_text"},{"ecs":false,"name":"winlog.event_data.AttributeLDAPDisplayName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AttributeValue","type":"keyword"},{"ecs":false,"name":"winlog.event_data.ShareName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.RelativeTargetName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AccessList","type":"keyword"}],"setup":"## Config\n\nThe 'Audit Detailed File Share' audit policy must be configured (Success Failure).\nSteps to implement the logging policy with with Advanced Audit Configuration:\n\n```\nComputer Configuration > \nPolicies > \nWindows Settings > \nSecurity Settings > \nAdvanced Audit Policies Configuration > \nAudit Policies > \nObject Access > \nAudit Detailed File Share (Success,Failure)\n```\n\nThe 'Audit Directory Service Changes' audit policy must be configured (Success Failure).\nSteps to implement the logging policy with with Advanced Audit Configuration:\n\n```\nComputer Configuration > \nPolicies > \nWindows Settings > \nSecurity Settings > \nAdvanced Audit Policies Configuration > \nAudit Policies > \nDS Access > \nAudit Directory Service Changes (Success,Failure)\n```\n","type":"query","language":"kuery","index":["logs-*"],"query":"host.name:*","filters":[],"throttle":"no_actions","actions":[]}
{"id":"6cc39c80-da3a-11ec-9fce-65c1a0bee901","updated_at":"2022-05-23T01:48:23.422Z","updated_by":"elastic","created_at":"2022-05-23T01:48:20.940Z","created_by":"elastic","name":"Rule with Related Integrations","tags":["Test"],"interval":"5m","enabled":false,"description":"See Related Integrations on the Rule Details and Rule Management pages.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":".siem-signals-default","meta":{"from":"5m"},"rule_name_override":"message","timestamp_override":"event.ingested","author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"2c66bf23-6ae9-4eb2-859e-446bea181ae1","max_signals":10000,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":7,"exceptions_list":[],"immutable":false,"related_integrations":[{"package":"unknown","version":"4.5.6"},{"package":"unknown","integration":"unknown","version":"^1.2.3"},{"package":"system","version":"~1.6.4"},{"package":"aws","integration":"cloudfront","version":"~1.11.0"},{"package":"aws","integration":"cloudtrail","version":"~1.11.0"},{"package":"aws","integration":"route53","version":"~123.456.789"},{"package":"azure","integration":"activitylogs","version":"~99.99.99"},{"package":"azure","integration":"platformlogs","version":"100.x.x"}],"required_fields":[],"setup":"","type":"query","language":"kuery","index":["logs-*"],"query":"host.name:*","filters":[],"throttle":"no_actions","actions":[]}
{"id":"6cc39c80-da3a-11ec-9fce-65c1a0bee902","updated_at":"2022-05-23T01:48:23.422Z","updated_by":"elastic","created_at":"2022-05-23T01:48:20.940Z","created_by":"elastic","name":"Rule with Required Fields","tags":["Test"],"interval":"5m","enabled":false,"description":"See Required Fields on the Rule Details page.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":".siem-signals-default","meta":{"from":"5m"},"rule_name_override":"message","timestamp_override":"event.ingested","author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"2c66bf23-6ae9-4eb2-859e-446bea181ae2","max_signals":10000,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":7,"exceptions_list":[],"immutable":false,"related_integrations":[],"required_fields":[{"ecs":true,"name":"event.code","type":"keyword"},{"ecs":true,"name":"message","type":"match_only_text"},{"ecs":false,"name":"winlog.event_data.AttributeLDAPDisplayName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AttributeValue","type":"keyword"},{"ecs":false,"name":"winlog.event_data.ShareName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.RelativeTargetName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AccessList","type":"keyword"}],"setup":"","type":"query","language":"kuery","index":["logs-*"],"query":"host.name:*","filters":[],"throttle":"no_actions","actions":[]}
{"id":"6cc39c80-da3a-11ec-9fce-65c1a0bee903","updated_at":"2022-05-23T01:48:23.422Z","updated_by":"elastic","created_at":"2022-05-23T01:48:20.940Z","created_by":"elastic","name":"Rule with Setup Guide","tags":["Test"],"interval":"5m","enabled":false,"description":"See Setup Guide on the Rule Details page.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":".siem-signals-default","meta":{"from":"5m"},"rule_name_override":"message","timestamp_override":"event.ingested","author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"2c66bf23-6ae9-4eb2-859e-446bea181ae3","max_signals":10000,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":7,"exceptions_list":[],"immutable":false,"related_integrations":[],"required_fields":[],"setup":"## Config\n\nThe 'Audit Detailed File Share' audit policy must be configured (Success Failure).\nSteps to implement the logging policy with with Advanced Audit Configuration:\n\n```\nComputer Configuration > \nPolicies > \nWindows Settings > \nSecurity Settings > \nAdvanced Audit Policies Configuration > \nAudit Policies > \nObject Access > \nAudit Detailed File Share (Success,Failure)\n```\n\nThe 'Audit Directory Service Changes' audit policy must be configured (Success Failure).\nSteps to implement the logging policy with with Advanced Audit Configuration:\n\n```\nComputer Configuration > \nPolicies > \nWindows Settings > \nSecurity Settings > \nAdvanced Audit Policies Configuration > \nAudit Policies > \nDS Access > \nAudit Directory Service Changes (Success,Failure)\n```\n","type":"query","language":"kuery","index":["logs-*"],"query":"host.name:*","filters":[],"throttle":"no_actions","actions":[]}
{"id":"6cc39c80-da3a-11ec-9fce-65c1a0bee904","updated_at":"2022-05-23T01:48:23.422Z","updated_by":"elastic","created_at":"2022-05-23T01:48:20.940Z","created_by":"elastic","name":"Rule without the new fields","tags":["Test"],"interval":"5m","enabled":false,"description":"See no Related Integrations, Required Fields, and Setup Guide on the Rule Details and Rule Management pages.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":".siem-signals-default","meta":{"from":"5m"},"rule_name_override":"message","timestamp_override":"event.ingested","author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"2c66bf23-6ae9-4eb2-859e-446bea181ae4","max_signals":10000,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":7,"exceptions_list":[],"immutable":false,"type":"query","language":"kuery","index":["logs-*"],"query":"host.name:*","filters":[],"throttle":"no_actions","actions":[]}
{"exported_count":5,"exported_rules_count":5,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":0,"exported_exception_list_item_count":0,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0}

[spong]

While testing locally, found a few bugs. Will post these right away and continue testing and reviewing the code.

Version mismatch icon's color is not readable

When the dark mode is off:

Links include a semver requirement instead of a target version

Looks like it happens for installed integrations, and works properly for those that are not installed when there's a version mismatch. Both pages are affected.

Thanks @banderror! Should be resolved as of b0ba541. I've updated the icon color to be warning as it is in the fleet UI, ensure the left-margin was being applied (it was set, but was being overriden by the EuiIconTip), and lastly fixed the version calculation and started adding our tests from the test plan.

There might still be another bug in here with the version calculation, so will finish adding the remaining tests and push those next 👍

[banderror]

Almost there! Reviewed most of the changes and have some suggestions, so please check the comments I left @spong.

Still have to carefully scan the getInstalledRelatedIntegrations logic and review the test coverage. Will do it tomorrow morning.

[banderror]

Can we have an integration_details.ts file and then move this mock data to integration_details.mock.ts?

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 238949c
• Storybooks Preview
• Webpack Bundle Analyzer

Failed CI Steps

• FTR Configs #35

Test Failures

• [job] [logs] FTR Configs #35 / saved objects tagging - functional tests table listing "after all" hook in "table listing"
• [job] [logs] FTR Configs #35 / saved objects tagging - functional tests table listing "before all" hook in "table listing"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	3081	3087	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	5.1MB	5.1MB	+3.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	250.5KB	250.6KB	+82.0B

History

• 💔 Build #50024 failed 6217ca3
• 💛 Build #50018 was flaky 82f58bb
• 💚 Build #49964 succeeded 6da880b
• 💚 Build #49471 succeeded 6737fb4
• 💚 Build #49392 succeeded 4dd34da

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong

[banderror]

Last comments! Finished reviewing the code changes and tested it once again.

Couldn't find any new bugs, and the two bugs I reported above are fixed! 🙌

[banderror]

When I disabled this setting and navigated to the Rules page without doing a page reload, it still worked (the column got hidden). Can we set requiresPageReload: false here?

[spong]

Yeah we can change to false -- I had tested in multiple tabs and noticed the opposite behavior so had set as true, but this appears to be a cache issue.

[banderror]

In which case version can be an empty string?

[spong]

It's the error case when semver.valid(semver.coerce(i.version)) returns null, so we will navigate to just the latest package version (redirected by fleet) in those instances. That said, as we saw, we still need to include the integrationName if there is once so it doesn't just drop the user off on the base integration page.

[banderror]

I'm thinking - if the version requirement is satisfied, shouldn't we use the installed version? Here in the code, it would be match.package_version:

      const targetVersion = versionSatisfied
        ? match.package_version
        : semver.valid(semver.coerce(i.version)) ?? '';

i.version here is the related integration's version which is ~1.2.3 etc.

So to me, it looks like a bug, but it works as expected locally and there are some unit tests that seem to cover this code path, so I'm not sure about my sanity 😂

[banderror]

Nit: Could this do the same?

  return integrationDetails.sort((a, b) => {
    const aTitle = a.integration_title ?? a.package_title;
    const bTitle = b.integration_title ?? b.package_title;
    return aTitle.localeCompare(bTitle);
  });

Also, if we always set integration_title in IntegrationDetails, could be even simpler:

  return integrationDetails.sort((a, b) => {
    return a.integration_title.localeCompare(b.integration_title);
  });

Then, in getIntegrationLink we wouldn't need to do ?? in const integrationTitle = integration.integration_title ?? integration.package_title;

[spong]

Yeah I think we can always set integrationTitle to clean this up and then keep integrationName undefined if it's a single integration package (i.e. system package).

[banderror]

Synced on the comments with @spong and decided to address them in a follow-up PR.
🚀 🚀 🚀

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.3	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 133050

Questions ?

Please refer to the Backport tool documentation