Add "fingerprint" ingest processor

[fbaligand]

Hi,

Since GDPR regulation, anonymization is become an important consideration.
That's why it would be great to add "fingerprint" ingest processor, with similar feature than Logstash "fingerprint" filter:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[danhermann]

@fbaligand, a few comments on work we've done or considered in ES that's related to your suggestion for a fingerprint ingest processor--

You mention both content fingerprinting and content anonymization (esp. in a GDPR context) as use cases for such a processor. While both scenarios commonly perform some type of hashing on the document content, there are important differences between the two use cases.

For content fingerprinting, we want to produce a single, stable token useful either for comparison or as a unique identifier that is based on the document content. This is commonly done by taking the SHA-256 or similar hash across all the relevant fields of the document (and this is what the Logstash fingerprint filter does). Notably, we want to make it easy to go from document content to the fingerprint token.

With content anonymization, we also want to produce from one or more sensitive fields within the document stable token(s) useful for comparison or aggregation. The big difference is that we want it to be very hard to derive the original content from the token, especially in the context of GDPR or HIPAA where there is an obligation to safeguard personally identifying or other sensitive information. In such conditions, a simple hash such as SHA-256 could be susceptible to a brute force attack, especially when the set of possible input values is relatively small or when some of the input values are known such as patient names for a particular doctor.

That susceptibility to brute force attacks can be eliminated by using a hash with a secret key. We did some work on such a processor in #31087 but it was ultimately pulled before release because there was no mechanism to ensure that the secret key was consistent across all ingest nodes. Without a guarantee of key consistency across nodes, different hashes for the same input data could be produced on different ingest nodes which would make the anonymized tokens useless for comparison or aggregation. We have some ideas about how to ensure consistency of secret keys across ingest nodes, but it is not a trivially easy problem to solve in the context of a distributed system such as ES.

[fbaligand]

First, thank you very much for your detailed answer.

I understand that it is not easy to provide a hash mechanism strong enough for anonymization use case.
And I understand that SHA2 is not really strong enough for anonymization use case.

That said, on a more global point of view, I think that it is very important to have a homogeneous filter/process offer between Logstash and Ingest Pipelines.

I mean that I would find very sad to have fingerprint filter in Logstash and fingerprint processor in ingest pipeline with different options and behavior.

That's why a first step would be to provide fingerprint processor with same features and behavior than Logstash.

And later, maybe a nice idea to have a strong anonymization hash will emerge :)

[ypid-geberit]

From my understanding, we are talking about pseudonymization in the GDPR context here because even with the more suitable version of fingerprinting with a secret key (HMAC), users could still be re-identified with the knowledge of the secret key. Maybe add a way to rotate this key at defined intervals? This would provide strong pseudonymization properties after the rotation period. Note that this is just an idea. I have not checked if others are using such a design and what possible downsides are.

That's why a first step would be to provide fingerprint processor with same features and behavior than Logstash.

Not sure that you would really want that as the Logstash fingerprint filter is currently non-stable/none-deterministic for nested fields. Ref: logstash-plugins/logstash-filter-fingerprint#41