spaCy Tokenization Details

[tuzhucheng]

Hi, in the README.md it says:

A tokenized version of the supporting passage, using spaCy. Each token is a tuple of the token string and token character offset. The maximum number of tokens is 800.

I'm wondering if you guys can provide the script used for tokenization (ideally) or describe more specific details about how the tokenization is done. For example:

• Is the model en_core_web_sm, en_core_web_md, or en_core_web_lg?
• What are the custom arguments passed to the Tokenizer? I assume token_match=lambda t: t in ('[PAR]', '[TLE]', '[DOC]') for example.
• I believe I found some problem with tokenization in the data provided. If you look at line 5 of HotpotQA.jsonl, the [SEP] token is split up, which should not be in my opinion ([PAR], [TLE], [DOC] are not split up):

>>> sed '5q;d' HotpotQA.jsonl | jq '.context_tokens[2:7]'
[
  [
    "Ethanol",
    12
  ],
  [
    "[",
    20
  ],
  [
    "SEP",
    21
  ],
  [
    "]",
    24
  ],
  [
    "Ethanol",
    26
  ]
]

[ajfisch]

Hi, thanks for reporting!

The spacy model is en_core_web_sm.

The code for tokenizing looks like this:

def get_tokenizer(whitespace=False):
    """Load spacy tokenizer with special tokenization cases."""
    en = spacy.load('en', disable=['entity', 'tagger', 'parser'])
    if whitespace:
        tok = Tokenizer(en.vocab,
                        prefix_search=lambda _: None,
                        suffix_search=lambda _: None,
                        infix_finditer=lambda _: iter([]))
    else:
        tok = en.tokenizer
        for sep in SEPARATORS:
            tok.add_special_case(sep, [{spacy.attrs.ORTH: sep}])
    return tok

Indeed, there is a bug as [SEP] is not in SEPARATORS as you have seen. This is related to it being a special case for HotpotQA --- see #21 .

As it is late (and this has little effect on final performance), we won't fix it for now, but may issue a fixed version after the submission deadline for future work. [SEP] isn't used in the other datasets, so don't worry about doing custom hot-fixes for that.