duplicate terms in name affecting scoring

[missinglink]

It seems as though having duplicate tokens in a name is causing elasticsearch to score the result higher (this is due to how the TF/IDF scoring works)

While it's impossible to have exact duplicate names (this is taken care of by pelias/model in a 'post' step), it is possible to have two terms which are very similar such as this:

https://www.openstreetmap.org/way/432890745

I'm opening this issue so I don't forget, we can either try to solve this during import or during search.

A query such as /v1/search?text=whole foods market, NY illustrates the issue (although there may be other things at play here)

[orangejulius]

Oh yeah, this is a good one. It's kinda inherent to the scoring mechanisms of Elasticsearch.

Maybe we can solve this by modifying the scoring settings of Elasticsearch, which is easier and more flexible in newer versions. We might be able to split altnames into separate fields in a clever way as well. But I think this sort of case will be pretty hard, though not impossible, to solve.

[missinglink]

Last night I went into a deep-dive on tuning the BM25 algorithm, I suspect that we could reduce k1 to either 0 or near 0 to resolve this issue.

It's a big topic in itself so I'll start a discussion over on pelias/schema

[edit] I think maybe we actually want to set k1=1?

[orangejulius]

Since we've decided this issue is mostly unsolvable without major changes to how we index data in Elasticsearch, let's talk mitigation:

In this particular case, one name is a prefix of the other. Is it worth it to detect that and skip the shorter version?

In the other main case I've noticed with OSM data, the names only differ by whitespace. That one might be trickier, but we could probably still do it.