web_questions test.jsonl is not Xet in HGF

[cmacdonald]

Context terrierteam/pyterrier_rag#47

the test.jsonl file in
https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets/tree/main/web_questions
seems to be unable to parse from Pandas. I think this is because its not in Xet format?

This works:

pd.read_json("hf://datasets/RUC-NLPIR/FlashRAG_datasets/web_questions/train.jsonl", lines=True)

This doesnt:

pd.read_json("hf://datasets/RUC-NLPIR/FlashRAG_datasets/web_questions/test.jsonl", lines=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/miniconda3/lib/python3.12/site-packages/pandas/io/json/_json.py", line 791, in read_json
    json_reader = JsonReader(
                  ^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/pandas/io/json/_json.py", line 905, in __init__
    self.data = self._preprocess_data(data)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/lib/python3.12/site-packages/pandas/io/json/_json.py", line 917, in _preprocess_data
    data = data.read()
           ^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 12: invalid continuation byte

Workaround:

pd.read_json("https://huggingface.co/datasets/RUC-NLPIR/FlashRAG_datasets/resolve/main/web_questions/test.jsonl", lines=True)

[ignorejjj]

I downloaded this file to my local computer and it runs correctly using pd.read_json. Is it possible that there are some bugs in loading files from remote? Maybe you can try first downloading this by huggingface-cli.

[cmacdonald]

See this screenshot from

We rely on being able to download files using the HF:// url.

The problem appears that only test.jsonl isnt on Xet storage. Can you try reuploading it using huggingface_hub code?:

from huggingface_hub import HfApi
api = HfApi()
api.upload_file(
    path_or_fileobj="local/path/to/bigfile.csv",
    path_in_repo="data/bigfile.csv",
    repo_id="username/dataset_name",
    repo_type="dataset"
)

[cmacdonald]

pinging @ignorejjj

We also saw the same problem with test.jsonl of NQ