Fingerprint ingest processor

[danhermann]

Adds a fingerprint processor that computes hashes of document content for content fingerprinting use cases.

E.g.:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "fingerprint": {
          "fields": ["user"]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "user": {
          "last_name": "Smith",
          "first_name": "John",
          "date_of_birth": "1980-01-15",
          "is_active": true
        }
      }
    }
  ]
}

Which produces:

"_source" : {
  "fingerprint" : "WbSUPW4zY1PBPehh2AA/sSxiRjw=",
  "user" : {
    "last_name" : "Smith",
    "first_name" : "John",
    "date_of_birth" : "1980-01-15",
    "is_active" : true
  }
}

Supports any number of document fields, nested document content, any hash from [MD5, SHA-1, SHA-256, SHA-512], and a per-processor salt.

Closes #53578 though it addresses only content fingerprinting and not anonymization use cases.

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[danhermann]

@graphaelli, could you or a member of your team look this over to verify that it provides the functionality that the APM use case requires?

[danhermann]

@elasticmachine run elasticsearch-ci/bwc

[graphaelli]

Thanks @danhermann! @elastic/apm-server - can you provide feedback here?

[mark-vieira]

@elasticmachine update branch

[axw]

@danhermann thank you, this will do nicely for APM.

We'll need MD5 (which I see is available) for now, but later I would like to migrate to something faster, such as Murmurhash3 (non-cryptographic would be fine). Any reason why we couldn't add that later? Doesn't look like it to me, just wanted to double check.

[danhermann]

@danhermann thank you, this will do nicely for APM.

We'll need MD5 (which I see is available) for now, but later I would like to migrate to something faster, such as Murmurhash3 (non-cryptographic would be fine). Any reason why we couldn't add that later? Doesn't look like it to me, just wanted to double check.

Yes, adding support for the murmur3 hash should be doable.

[dakrone]

This looks great @danhermann, I left a few really minor comments, but I think we need to add regular docs also right?

[dakrone]

Streams are super great, but they aren't quite as fast compared to foreach clauses yet, and since this is performance sensitive code, what do you think about doing:

Suggested change

	var setList = set.stream().sorted().collect(Collectors.toList());
	for (int k = setList.size() - 1; k >= 0; k--) {
	values.push(setList.get(k));
	}
	var setList = new ArrayList<?>(set);
	setList.sort(Comparator.naturalOrder());
	for (Object thing : setList) {
	values.push(thing);
	}

?

(I might be missing something about why you're using a for loop indexed rather than foreach, as I think the foreach compiles to the same bytecode, but I could totally be wrong)

[danhermann]

The for loop iterates backward over the sorted list to push the values onto the stack. That means that the values are processed in sorted order after being popped off the stack which is not strictly necessary since only a stable hash is required, but having the values in sorted order is certainly nicer when debugging.

[danhermann]

I did replace the usage of streams both here and below.

[danhermann]

Thanks for the review, @dakrone. I'll open a separate PR with the docs for this one shortly.

[danhermann]

@dakrone, I've updated this with all your suggestions except the indexed for loop. Let me know if you're ok with the reverse iteration so elements from the stack are processed in sorted order.

[danhermann]

@elasticmachine update branch

[danhermann]

@elasticmachine run elasticsearch-ci/2

[dakrone]

LGTM, I'm totally cool with the reverse ordering, thanks for iterating!

[danhermann]

cc: @elastic/es-ui in case Kibana auto-complete needs to be updated with this new processor.