Add refs_to_tokens command

[ivyleavedtoadflax]

Now that there is https://github.com/wellcometrust/deep_reference_parser, I'm moving the associated utilities out of datalabs into wellcomeml.nlp so we can access them easily on any platform. This is quite large, so let me know if you would prefer me to chop it up into smaller PRs, I think most of it is tests though.

This class allows you to convert manual reference annotations in prodigy into token annotations.

The other significant change here is that I had added a __main__.py inspired by SpaCy. This basically gives us an entry point into the package functions like so:

$ python -m wellcomeml

ℹ Available commands
refs_to_token_annotations

This plugs into the cli of each of these functions, so you can also do:

$ python -m wellcomeml refs_to_tokens --help
usage: wellcomeml refs_to_tokens [-h] input_file output_file

 Converts a file output by prodigy (using prodigy db-out) from
    references level annotations to individual level annotations. The rationale
    for this is that reference level annotations are much easier for humans to
    do, but not useful when training a token level model.

    This function is predominantly useful fot tagging reference spans, but may
    also have a function with other references annotations.
    

positional arguments:
  input_file   Path to jsonl file containing chunks of references in prodigy
               format.
  output_file  Path to jsonl file into which fully annotate files will be
               saved.

optional arguments:
  -h, --help   show this help message and exit

I quite like this pattern for getting CLI access to the code without creating explicit entry points in setup.py. I think some of these functions are borderline in terms of their fit in WellcomeML, see what you think, happy to put them in deep_reference_parser if not here.

[ivyleavedtoadflax]

Moved to wellcometrust/deep_reference_parser#5