Clarify the use of [SEP] tokens in HotpotQA

[chrisc36]

In HotPotQa, there are "[SEP]" tokens after "[TLE]" tokens, I assume(?) there were added by the organizers. For example, qid eaf089c068d2433f88f8f6b70d9946a9 from hotpotqa dev starts with the text "[PAR] [TLE] York [SEP] York () is a...."

However, the use of [SEP] tokens is not documented, and as far as I can tell, in other datasets [TLE] tokens are followed by [PAR] tokens not [SEP] tokens.

Was this intentional? If so, can you clarify how [SEP] tokens are used?

[ajfisch]

Hi Chris,

Sorry for the confusion and that this was not documented. This is a special case for HotpotQA. Here paragraphs are associated with a title (rather than full documents). The convention we chose was [PAR] [TLE] <The Title> [SEP] <The content.> [PAR] [TLE] <The Next Title> [SEP] <The next content.>.

For full documents, we chose [DOC] [TLE] <The Document Title> [PAR] <The first paragraph.> [PAR] <The second paragraph.>.