Explicitly setting encoding in tokenization_utils_base.py

[Muqi1029]

Add encoding explicitly

What does this PR do?

Problem:

When loading tokenizers, specifically those that utilize a chat_template.jinja file (e.g., THUDM/GLM-4-9B-0414), a UnicodeDecodeError can occur in certain environments. This happens because the open() call defaults to the system's preferred encoding, which can unexpectedly revert to ASCII (ANSI_X3.4-1968) within subprocesses, particularly in torch.distributed settings or isolated container environments.

The root cause is that these subprocesses may not correctly inherit LANG, LC_ALL, or LC_CTYPE environment variables from the parent shell, leading locale.getpreferredencoding() to return ASCII instead of UTF-8. This results in the following traceback when encountering non-ASCII bytes (e.g., 0xe5) in the chat_template.jinja file:

chat_template_handle <_io.TextIOWrapper name='/root/.cache/huggingface/hub/models--THUDM--GLM-4-9B-0414/snapshots/645b8482494e31b6b752272bf7f7f273ef0f3caf/chat_template.jinja' mode='r' encoding='ANSI_X3.4-1968'>
chat_template_handle.read()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/root/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/lib/python3.12/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 43: ordinal not in range(128)

Solution:

This PR explicitly sets the encoding="utf-8" parameter when opening the file. It then works as expected.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[Rocketknight1]

Yes, thank you for catching this!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.