Skip to content

Allow different decay values depending on the score function#91195

Merged
benwtrent merged 6 commits intoelastic:mainfrom
matiassalles99:feature/allow-different-decay-values-per-score-function
Nov 2, 2022
Merged

Allow different decay values depending on the score function#91195
benwtrent merged 6 commits intoelastic:mainfrom
matiassalles99:feature/allow-different-decay-values-per-score-function

Conversation

@matiassalles99
Copy link
Copy Markdown
Contributor

@matiassalles99 matiassalles99 commented Oct 31, 2022

Tackles #78887

Currently, when setting a decay value on a score function, you can only set a value between 0 and 1 exclusively, independently of the score function you're using.

// elasticsearch/server/src/main/java/org/elasticsearch/index/query/functionscore/DecayFunctionBuilder.java
protected DecayFunctionBuilder(String fieldName, Object origin, Object scale, Object offset, double decay) {
        if (fieldName == null) {
            throw new IllegalArgumentException("decay function: field name must not be null");
        }
        if (scale == null) {
            throw new IllegalArgumentException("decay function: scale must not be null");
        }
        if (decay <= 0 || decay >= 1.0) {
            throw new IllegalStateException("decay function: decay must be in range 0..1!");
        }
// ....

This makes sense with gauss and exp score functions, since they use ln(decay) in their formula, and ln(0) does not exist.
But when using linear functions the decay should be allowed to be 0, since the formula would not result in a division by 0.
image

I'm open to any suggestions on how to tackle it better or test it better

@elasticsearchmachine elasticsearchmachine added v8.6.0 needs:triage Requires assignment of a team area label external-contributor Pull request authored by a developer outside the Elasticsearch team labels Oct 31, 2022
@bpintea bpintea added :Search/Search Search-related issues that do not fall into other categories and removed needs:triage Requires assignment of a team area label labels Oct 31, 2022
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-search (Team:Search)

@elasticsearchmachine elasticsearchmachine added the Team:Search Meta label for search team label Oct 31, 2022
@benwtrent benwtrent self-requested a review November 1, 2022 15:20
Copy link
Copy Markdown
Member

@benwtrent benwtrent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the change is mostly good. It clarifies the mathematical restrictions within the validations.

I mentioned one change required. We should only have a separate decay validator.

Additionally, It would be good to add some test lines to ScoreFunctionBuilderTests:

public void testIllegalArguments() {
expectThrows(IllegalArgumentException.class, () -> new RandomScoreFunctionBuilder().seed(null));
expectThrows(IllegalArgumentException.class, () -> new ScriptScoreFunctionBuilder((Script) null));
expectThrows(IllegalArgumentException.class, () -> new FieldValueFactorFunctionBuilder((String) null));
expectThrows(IllegalArgumentException.class, () -> new FieldValueFactorFunctionBuilder("").modifier(null));
expectThrows(IllegalArgumentException.class, () -> new GaussDecayFunctionBuilder(null, "", "", ""));
expectThrows(IllegalArgumentException.class, () -> new GaussDecayFunctionBuilder("", "", null, ""));
expectThrows(IllegalArgumentException.class, () -> new GaussDecayFunctionBuilder("", "", null, "", randomDouble()));
expectThrows(IllegalArgumentException.class, () -> new LinearDecayFunctionBuilder(null, "", "", ""));
expectThrows(IllegalArgumentException.class, () -> new LinearDecayFunctionBuilder("", "", null, ""));
expectThrows(IllegalArgumentException.class, () -> new LinearDecayFunctionBuilder("", "", null, "", randomDouble()));
expectThrows(IllegalArgumentException.class, () -> new ExponentialDecayFunctionBuilder(null, "", "", ""));
expectThrows(IllegalArgumentException.class, () -> new ExponentialDecayFunctionBuilder("", "", null, ""));
expectThrows(IllegalArgumentException.class, () -> new ExponentialDecayFunctionBuilder("", "", null, "", randomDouble()));
}

Something along the lines

expectThrows(IllegalArgumentException.class, () -> new ExponentialDecayFunctionBuilder("", "", "", "", 0.0));
expectThrows(IllegalArgumentException.class, () -> new ExponentialDecayFunctionBuilder("", "", "", "", 1.0));
expectThrows(IllegalArgumentException.class, () -> new GaussDecayFunctionBuilder("", "", "", "", 0.0));
expectThrows(IllegalArgumentException.class, () -> new GaussDecayFunctionBuilder("", "", "", "", 1.0));
expectThrows(IllegalArgumentException.class, () -> new LinearDecayFunctionBuilder("", "", "", "", 1.0));
expectThrows(IllegalArgumentException.class, () -> new LinearDecayFunctionBuilder("", "", "", "", -1.0));
// should not throw
LinearDecayFunctionBuilder("", "", "", "", 0.0);

/**
* Validate properties, override this function if you have different validation rules per score function
*/
protected void validateProperties(String fieldName, Object scale, double decay){
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could simply be a validateDecay function. The null checks will always be universal.

But, I think this idea of a protected/Overridable function is a good one.

Copy link
Copy Markdown
Contributor Author

@matiassalles99 matiassalles99 Nov 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benwtrent Thanks for the comments! will work on it 💪

@benwtrent benwtrent added the >bug label Nov 1, 2022
@benwtrent benwtrent self-assigned this Nov 1, 2022
@benwtrent
Copy link
Copy Markdown
Member

jenkins test this please

Copy link
Copy Markdown
Member

@benwtrent benwtrent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once CI is ✅ I will give it a merge.

Thank you for your contribution!

@benwtrent
Copy link
Copy Markdown
Member

@matiassalles99 please run ./gradlew spotlessApply locally and commit the changes. This will apply correct formatting. There are some formatting complaints by CI.

One day maybe we will just auto apply formatting via a workflow. But not yet :).

@matiassalles99
Copy link
Copy Markdown
Contributor Author

Sure!

@benwtrent
Copy link
Copy Markdown
Member

@elasticmachine update branch

@benwtrent
Copy link
Copy Markdown
Member

jenkins test this please

@benwtrent benwtrent merged commit f2f9a1f into elastic:main Nov 2, 2022
@matiassalles99 matiassalles99 deleted the feature/allow-different-decay-values-per-score-function branch November 2, 2022 15:09
@benwtrent
Copy link
Copy Markdown
Member

🎉 🎉 🎉 🎉
Congratulations @matiassalles99 on your first pull-request to Elasticsearch! Thank you for the contribution!!
🎉 🎉 🎉 🎉

weizijun added a commit to weizijun/elasticsearch that referenced this pull request Nov 3, 2022
* main: (1300 commits)
  update c2id/c2id-server-demo docker image to support ARM (elastic#91144)
  Allow legacy index settings on legacy indices (elastic#90264)
  Skip prevoting if single-node discovery (elastic#91255)
  Chunked encoding for snapshot status API (elastic#90801)
  Allow different decay values depending on the score function (elastic#91195)
  Fix handling indexed envelopes crossing the dateline in mvt API (elastic#91105)
  Ensure cleanups succeed in JoinValidationService (elastic#90601)
  Add overflow behaviour test for RecyclerBytesStreamOutput (elastic#90638)
  More actionable error for ancient indices (elastic#91243)
  Fix APM configuration file delete (elastic#91058)
  Clean up handshake test class (elastic#90966)
  Improve H3#hexRing logic and add H3#areNeighborCells method (elastic#91140)
  Restrict direct use of `ApplicationPrivilege` constructor (elastic#91176)
  [ML] Allow NLP truncate option to be updated when span is set (elastic#91224)
  Support multi-intersection for FieldPermissions (elastic#91169)
  Support intersecting multi-sets of queries with DocumentPermissions (elastic#91151)
  Ensure TermsEnum action works correctly with API keys (elastic#91170)
  Fix NPE in auditing authenticationSuccess for non-existing run-as user (elastic#91171)
  Ensure PKI's delegated_by_realm metadata respect run-as (elastic#91173)
  [ML] Update API documentation for anomaly score explanation (elastic#91177)
  ...

# Conflicts:
#	x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java
#	x-pack/plugin/rollup/src/main/java/org/elasticsearch/xpack/downsample/RollupShardIndexer.java
#	x-pack/plugin/rollup/src/main/java/org/elasticsearch/xpack/downsample/TransportRollupIndexerAction.java
#	x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/v2/RollupActionSingleNodeTests.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>bug external-contributor Pull request authored by a developer outside the Elasticsearch team :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team v8.6.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants