Skip to content

[sentinel_one] Initial Release for the SentinelOne Package#3232

Merged
andrewkroh merged 5 commits intoelastic:mainfrom
vinit-chauhan:package_sentinel_one
May 26, 2022
Merged

[sentinel_one] Initial Release for the SentinelOne Package#3232
andrewkroh merged 5 commits intoelastic:mainfrom
vinit-chauhan:package_sentinel_one

Conversation

@vinit-chauhan
Copy link
Copy Markdown
Contributor

What does this PR do?

  • Generated the skeleton of the SentinelOne integration package.
  • Added a data stream.
  • Added data collection logic to the data stream.
  • Added the ingest pipeline for the data stream.
  • Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files
  • Added dashboards and visualizations.
  • Added test for pipeline for the data stream.
  • Added system test cases for the data stream.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.17.0 || ^8.0.0).

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/sentinel_one directory.
  • Run the following command to run tests.

elastic-package test

Screenshots

image
image
image
image
image
image

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Apr 29, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-05-25T22:26:30.056+0000

  • Duration: 17 min 5 sec

Test stats 🧪

Test Results
Failed 0
Passed 31
Skipped 0
Total 31

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@vinit-chauhan vinit-chauhan added enhancement New feature or request New Integration Issue or pull request for creating a new integration package. labels Apr 29, 2022
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Apr 29, 2022

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (5/5) 💚
Files 100.0% (5/5) 💚 3.538
Classes 100.0% (5/5) 💚 3.538
Methods 100.0% (70/70) 💚 11.542
Lines 98.348% (2560/2603) 👍 9.032
Conditionals 100.0% (0/0) 💚

@vinit-chauhan vinit-chauhan changed the title [SentinelOne] Initial Release for the SentinelOne Package [sentinel_one] Initial Release for the SentinelOne Package Apr 29, 2022
@elasticmachine
Copy link
Copy Markdown

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@andrewkroh andrewkroh requested a review from a team May 2, 2022 19:25
@@ -0,0 +1,20 @@
- name: data_stream.dataset
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these constant_keyword fields not have values?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, neither of the connectors has any values for the data_stream.dataset.

Comment thread packages/sentinel_one/data_stream/activity/fields/fields.yml Outdated
Comment thread packages/sentinel_one/data_stream/activity/fields/fields.yml Outdated
Comment thread packages/sentinel_one/data_stream/agent/fields/fields.yml Outdated
Comment thread packages/sentinel_one/data_stream/alert/fields/fields.yml Outdated
Comment thread packages/sentinel_one/data_stream/alert/fields/fields.yml Outdated
Comment thread packages/sentinel_one/data_stream/threat/fields/fields.yml Outdated
Comment thread packages/sentinel_one/data_stream/threat/fields/fields.yml Outdated
@vinit-chauhan vinit-chauhan force-pushed the package_sentinel_one branch from 250251c to afebf16 Compare May 5, 2022 14:49
@andrewkroh
Copy link
Copy Markdown
Member

Please add event.created to each of the data streams.

@andrewkroh
Copy link
Copy Markdown
Member

Some of the fields are duplicated. Soon we will be validating for this (see elastic/package-spec#309). Can you please address the duplicates.

packages/sentinel_one/data_stream/activity
packages/sentinel_one/data_stream/activity/fields/ecs.yml:35 - host.os.family : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/activity/fields/agent.yml:124 - host.os.family : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/activity/fields/ecs.yml:33 - host.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/activity/fields/agent.yml:119 - host.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/activity/fields/agent.yml:110 - host.ip : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/activity/fields/ecs.yml:31 - host.ip : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert
packages/sentinel_one/data_stream/alert/fields/agent.yml:110 - host.ip : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/ecs.yml:33 - host.ip : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/ecs.yml:1 - container.id : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/agent.yml:61 - container.id : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/ecs.yml:5 - container.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/agent.yml:76 - container.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/ecs.yml:37 - host.os.family : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/agent.yml:124 - host.os.family : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/agent.yml:153 - host.os.version : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/ecs.yml:39 - host.os.version : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/ecs.yml:3 - container.image.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/agent.yml:66 - container.image.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/agent.yml:159 - host.type : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/ecs.yml:41 - host.type : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/agent.yml:119 - host.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/alert/fields/ecs.yml:35 - host.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent
packages/sentinel_one/data_stream/agent/fields/agent.yml:119 - host.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/ecs.yml:37 - host.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/agent.yml:136 - host.os.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/ecs.yml:39 - host.os.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/ecs.yml:31 - host.id : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/agent.yml:105 - host.id : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/ecs.yml:15 - host.domain : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/agent.yml:93 - host.domain : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/ecs.yml:43 - host.os.version : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/agent.yml:153 - host.os.version : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/ecs.yml:33 - host.ip : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/agent.yml:110 - host.ip : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/ecs.yml:35 - host.mac : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/agent/fields/agent.yml:114 - host.mac : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat
packages/sentinel_one/data_stream/threat/fields/ecs.yml:17 - host.domain : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/agent.yml:93 - host.domain : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/agent.yml:61 - container.id : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/ecs.yml:1 - container.id : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/agent.yml:105 - host.id : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/ecs.yml:33 - host.id : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/ecs.yml:39 - host.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/agent.yml:119 - host.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/agent.yml:76 - container.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/ecs.yml:5 - container.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/agent.yml:110 - host.ip : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/ecs.yml:35 - host.ip : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/agent.yml:114 - host.mac : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/ecs.yml:37 - host.mac : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/agent.yml:136 - host.os.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/ecs.yml:41 - host.os.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/agent.yml:66 - container.image.name : [Duplicate field (2 times).]
packages/sentinel_one/data_stream/threat/fields/ecs.yml:3 - container.image.name : [Duplicate field (2 times).]

"id": "13491234512345",
"ip": "81.2.69.143",
"mac": [
"42-X1-0X-9X-X0-0X"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will fail validation due to the X. I assume this was anonymized. Some safe values to use are the ranges listed in https://www.rfc-editor.org/rfc/rfc7042.html#section-2.1.2.

For example if change _dev/build/build.yml to use ECS 8.3, remove the duplicate non-ECS definition of host.mac, then run elastic-package test pipeline you will see an error:

[0] parsing field value failed: field "host.mac"'s value, 42-X1-0X-9X-X0-0X, does not match the expected pattern: ^[A-F0-9]{2}(-[A-F0-9]{2}){5,}$

@jamiehynds jamiehynds mentioned this pull request May 11, 2022
15 tasks
@vinit-chauhan
Copy link
Copy Markdown
Contributor Author

Hey @andrewkroh, I have updated the PR as per your comments. Please have a look and let me know if there are any changes.

"ingested": "2022-05-09T12:54:52Z",
"kind": "event",
"original": "{\"agentDetectionInfo\":{\"machineType\":\"string\",\"name\":\"string\",\"osFamily\":\"string\",\"osName\":\"string\",\"osRevision\":\"string\",\"siteId\":\"123456789123456789\",\"uuid\":\"string\",\"version\":\"3.x.x.x\"},\"alertInfo\":{\"alertId\":\"123456789123456789\",\"analystVerdict\":\"string\",\"createdAt\":\"2018-02-27T04:49:26.257525Z\",\"dnsRequest\":\"string\",\"dnsResponse\":\"string\",\"dstIp\":\"0.0.0.0\",\"dstPort\":\"1234\",\"dvEventId\":\"string\",\"eventType\":\"string\",\"hitType\":\"Events\",\"incidentStatus\":\"string\",\"indicatorCategory\":\"string\",\"indicatorDescription\":\"string\",\"indicatorName\":\"string\",\"loginAccountDomain\":\"string\",\"loginAccountSid\":\"string\",\"loginIsAdministratorEquivalent\":\"string\",\"loginIsSuccessful\":\"string\",\"loginType\":\"string\",\"loginsUserName\":\"string\",\"modulePath\":\"string\",\"moduleSha1\":\"aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d\",\"netEventDirection\":\"string\",\"registryKeyPath\":\"string\",\"registryOldValue\":\"string\",\"registryOldValueType\":\"string\",\"registryPath\":\"string\",\"registryValue\":\"string\",\"reportedAt\":\"2018-02-27T04:49:26.257525Z\",\"source\":\"string\",\"srcIp\":\"0.0.0.0\",\"srcMachineIp\":\"0.0.0.0\",\"srcPort\":\"string\",\"tiIndicatorComparisonMethod\":\"string\",\"tiIndicatorSource\":\"string\",\"tiIndicatorType\":\"string\",\"tiIndicatorValue\":\"string\",\"updatedAt\":\"2018-02-27T04:49:26.257525Z\"},\"containerInfo\":{\"id\":\"string\",\"image\":\"string\",\"labels\":\"string\",\"name\":\"string\"},\"kubernetesInfo\":{\"cluster\":\"string\",\"controllerKind\":\"string\",\"controllerLabels\":\"string\",\"controllerName\":\"string\",\"namespace\":\"string\",\"namespaceLabels\":\"string\",\"node\":\"string\",\"pod\":\"string\",\"podLabels\":\"string\"},\"ruleInfo\":{\"description\":\"string\",\"id\":\"string\",\"name\":\"string\",\"scopeLevel\":\"string\",\"severity\":\"Low\",\"treatAsThreat\":\"UNDEFINED\"},\"sourceParentProcessInfo\":{\"commandline\":\"string\",\"fileHashMd5\":\"5d41402abc4b2a76b9719d911017c592\",\"fileHashSha1\":\"aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d\",\"fileHashSha256\":\"2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824\",\"filePath\":\"string\",\"fileSignerIdentity\":\"string\",\"integrityLevel\":\"unknown\",\"name\":\"string\",\"pid\":\"12345\",\"pidStarttime\":\"2018-02-27T04:49:26.257525Z\",\"storyline\":\"string\",\"subsystem\":\"unknown\",\"uniqueId\":\"string\",\"user\":\"string\"},\"sourceProcessInfo\":{\"commandline\":\"string\",\"fileHashMd5\":\"5d41402abc4b2a76b9719d911017c592\",\"fileHashSha1\":\"aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d\",\"fileHashSha256\":\"2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824\",\"filePath\":\"string\",\"fileSignerIdentity\":\"string\",\"integrityLevel\":\"unknown\",\"name\":\"string\",\"pid\":\"12345\",\"pidStarttime\":\"2018-02-27T04:49:26.257525Z\",\"storyline\":\"string\",\"subsystem\":\"unknown\",\"uniqueId\":\"string\",\"user\":\"string\"},\"targetProcessInfo\":{\"tgtFileCreatedAt\":\"2018-02-27T04:49:26.257525Z\",\"tgtFileHashSha1\":\"aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d\",\"tgtFileHashSha256\":\"2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824\",\"tgtFileId\":\"string\",\"tgtFileIsSigned\":\"string\",\"tgtFileModifiedAt\":\"2018-02-27T04:49:26.257525Z\",\"tgtFileOldPath\":\"string\",\"tgtFilePath\":\"string\",\"tgtProcCmdLine\":\"string\",\"tgtProcImagePath\":\"string\",\"tgtProcIntegrityLevel\":\"unknown\",\"tgtProcName\":\"string\",\"tgtProcPid\":\"12345\",\"tgtProcSignedStatus\":\"string\",\"tgtProcStorylineId\":\"string\",\"tgtProcUid\":\"string\",\"tgtProcessStartTime\":\"2018-02-27T04:49:26.257525Z\"}}",
"type": "string"
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh May 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[0] parsing field value failed: field "event.type"'s value "string" is not one of the allowed values (access, admin, allowed, change, connection, creation, deletion, denied, end, error, group, indicator, info, installation, protocol, start, user)

And it should be an array.

See #3016 (comment) for testing info.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the time being we are not sure of the possible values of the mapped field as we are not able to generate the alerts. Once we have the alerts available we can update event.type accordingly.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't leave this with an invalid value so I moved the original value into sentinel_one.alert.info.event_type. After we have samples then we can set event.type and event.category based on this value.

Don't set event.type with the raw evenType value because it won't conform to ECS's
allowed values. After we have real alert samples then this value can be used to inform
the event.category and event.type values.
@andrewkroh andrewkroh force-pushed the package_sentinel_one branch from 58357d6 to 196fbb5 Compare May 25, 2022 22:26
@andrewkroh andrewkroh merged commit 52cc3b4 into elastic:main May 26, 2022
@vinit-chauhan vinit-chauhan deleted the package_sentinel_one branch April 1, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:sentinel_one SentinelOne New Integration Issue or pull request for creating a new integration package.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants