[ML] Allow NLP truncate option to be updated when span is set

[davidkyle]

Models preconfigured with the tokenization options truncate: NONE and span: X where X is > 0 error when updating the truncate option. For example

POST _ml/trained_models/model/_infer
{
  "docs": [..]
  "inference_config": {
    "question_answering": {
      "question": "Who moved my cheese?",
      "tokenization" : {
        "bert": {
          "truncate": "second"         <-- override the existing truncate option
        }
      }
    }
  }
}

Returns the error

[span] must not be provided when [truncate] is not [none]"

Because the model is configured with a span option the validation check fails. The work around is to set span: -1 where the value -1 unsets span.

This change wipes out any preexisting span option when truncate is set to first or second. That in itself is a small change the rest is testing and a refactoring of the Roberta and BERT tokenizers to share common code.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @davidkyle, I've created a changelog YAML for you.

[dimitris-athanasiou]

Looks good, just a comment about clarifying constructors for RobertaTokenization.

[dimitris-athanasiou]

Why is there a constructor that doesn't expect doLowerCase for RobertaTokenization? The only public constructor before this change forces lower case to false. This seems something worth clarifying as we're refactoring the area.

[davidkyle]

I reverted this change as it was only a convenience for a test. The value of doLowerCase should always be false.

[dimitris-athanasiou]

LGTM

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.5