feat: DNS and IP Database Processing

[jshlbrd]

Description

• Adds a Close method to the Applicator and BatchApplicator interfaces
• Adds an internal package for supporting IP address enrichment from multiple enrichment database providers (internal/ip/database)
• Adds an IP address database processor that applies enrichment to IP addresses from local databases
• Adds a DNS processor

Motivation and Context

Currently the team at Brex runs DNS and IP address enrichment in a separate sub-system of our threat detection platform, but we identified an opportunity to migrate this into Substation. IP address enrichment is the most complex of these for two reasons:

• it relies on having the ability to retrieve and load enrichment databases into the system at runtime
• we don't know what enrichment database providers users may use

The project's use of lazy loading, configuration through environment variables, and the ability to contextually retrieve files from local disk, HTTP(S), and AWS S3 solve many of these issues. As a starting point, I've added support for the free MaxMind and IP2Location databases.

The handling of open files in the IPDatabase processor required the addition of a Close method to each applicator interface; this allows each processor to define it's own teardown process (in the case of IPDatabase, it's closing the open database readers).

How Has This Been Tested?

The changes were integration tested using the two new examples files. I can provide a link to a privately hosted MaxMind database so that the reviewer(s) can check that the code is working as expected.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[shellcromancer]

style: rename to ipDatabases after the interface change?

[shellcromancer]

Looks great!