Skip to content

fix: process.IPDatabase Errors, condition.IP Type#44

Merged
jshlbrd merged 7 commits intomainfrom
jshlbrd/fix/ipdb-loading
Dec 13, 2022
Merged

fix: process.IPDatabase Errors, condition.IP Type#44
jshlbrd merged 7 commits intomainfrom
jshlbrd/fix/ipdb-loading

Conversation

@jshlbrd
Copy link
Contributor

@jshlbrd jshlbrd commented Dec 10, 2022

Description

  • Fixes an error in the IPDatabase processor that occurs when concurrency is enabled
  • Refactors the IPDatabase Factory to follow our documented design pattern
  • Adds new type check to IP inspector for checking if an IP address is any valid type
  • Adds missing IPDatabase and DNS processor libsonnet configs from feat: DNS and IP Database Processing #39
  • Adds new libsonnet pattern for looking up public IP addresses in IP enrichment databases

Motivation and Context

Discovered a bug in the IPDatabase processor when doing lower-level integration prep work -- if concurrency is active, then opening and accessing the database can randomly fail, but opening the database then putting it into the map fixes the problem.

While working on this I noticed two other things:

  • the original PR for the IP database feature doesn't follow our design pattern for creating factories -- surprised I didn't catch this during development, the use of environment variables should have been a clue that something wasn't quite right with the solution
  • the project has to support checking if an IP address is valid before sending it to the IPDatabase processor -- this processor may return an error indicating that the input is not a valid IP address, so we have to give users a way to handle that use case

How Has This Been Tested?

Generated a list of random IP addresses using IPVOID and tested different concurrency settings using the file app (SUBSTATION_CONCURRENCY=1, SUBSTATION_CONCURRENCY=2, SUBSTATION_CONCURRENCY=6). Unrelated to this change, using high concurrency settings (2+) when loading files from remote locations (HTTP) caused massive slowdown in runtime. We should encourage users to do local cacheing of files instead of pulling them from remote sources on every invocation, even if we don't provide tools to do the cacheing.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@jshlbrd jshlbrd changed the title fix: process.IPDatabase Errors, condition.IP Type fix!: process.IPDatabase Errors, condition.IP Type Dec 12, 2022
@jshlbrd jshlbrd marked this pull request as ready for review December 12, 2022 18:29
@jshlbrd jshlbrd requested a review from a team as a code owner December 12, 2022 18:29
@jshlbrd jshlbrd changed the title fix!: process.IPDatabase Errors, condition.IP Type fix: process.IPDatabase Errors, condition.IP Type Dec 13, 2022
@jshlbrd jshlbrd merged commit a2cf347 into main Dec 13, 2022
@jshlbrd jshlbrd deleted the jshlbrd/fix/ipdb-loading branch December 13, 2022 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants