Invalid json (with latin1 chars) get parsed instead of throwing an exception

[dominikzogg]

I've tried to enhance error handling of my deserializer library (https://github.com/chubbyphp/chubbyphp-deserialization/blob/master/src/Decoder/JsonTypeDecoder.php) and wished to support native json_decode and your library (what the library user prefers).

While testing i've seen, that the parser seems to allow latin1 chars within the json.
What i know, json needs to be utf-8 (or asci).

[dominikzogg]

chubbyphp-legacy/chubbyphp-deserialization#18

[Seldaek]

Interesting.. Not sure how we could fix this though, if you have a suggestion feel free.

[staabm]

so the question is: should the linter check the whole file for non-utf8 characters, right?

[Seldaek]

I guess it should reject any invalid utf8 char yes.

[dominikzogg]

@Seldaek a check having at least two chars > 127 instead of one would probably fix most, but it would be still possible having 'ää' as latin1 which seems to be utf-8 then

[LLyaudet]

Hello :),

I was looking for a JSON linter with an Ubuntu package and found yours.
I checked the issues here and since I already know it is easy to check valid UTF8 since I did it for EDI (Electronic Data Interchange), I added the required function in one of my projects.
See check_file_is_valid_UTF8 and check_string_is_valid_UTF8 in https://github.com/LLyaudet/DevOrSysAdminScripts/blob/main/unicode.libr.php

Let me know if you need a PR or if you can integrate such a check yourself.

Best regards,
Laurent Lyaudet

[Seldaek]

Feel free to send a PR then we can evaluate this.

[stof]

I don't think jsonlint should have public methods related to checking whether a string is UTF-8, which is not related to linting JSON.

checking whether a string is UTF-8 can already be done in PHP, either using mb_check_encoding or using preg_match. See php/php-src#20984 where I requested a core feature to have such use case covered by a dedicated function in core that would always be available (unlike mbstring)

[dominikzogg]

@stof (Answer by Gemini)

According to the official JSON standards (RFC 8259), the only valid encoding for JSON text exchanged in an "open ecosystem" is UTF-8.

While older versions of the specification (like RFC 4627) allowed for other Unicode encodings like UTF-16 or UTF-32, the current standard is strict about UTF-8 for maximum interoperability.

[stof]

@dominikzogg the fact that jsonlint needs to ensure the input is HTML does not mean it should expose a public API allowing to reuse only that part of the validation (which is then covered by the BC guarantees of the package and cannot be removed easily).
And as hinted in my previous comment, there is already efficient way to check for UTF-8 validity in PHP, being much faster than your function processing the whole string byte by byte in userland.

[LLyaudet]

Hello @stof :)

Of course you opened an issue on this topic 3 weeks ago,
and just ended on the current issue where nobody answered for more than 7 years
a few hours after my answer, probably looking for your lost pet ;) XD.
#TheTrumanShow XD

I don't think jsonlint should have public methods related to checking whether a string is UTF-8,
which is not related to linting JSON.

WRONG answer, you're out ;) XD.
If some RFCs, as @dominikzogg noted, state that JSON SHOULD be UTF-8,
then the CORRECT answer is to have an optional check for UTF-8 encoding.
If someone uses JSON with some other encoding, he can use enca if he wishes,
but that's not mainstream.
And I don't see the point of forcing users of jsonlint to have to use enca
or something else to have full validation for the mainstream use cases.

checking whether a string is UTF-8 can already be done in PHP, either using mb_check_encoding

As you now, you need to install php-mbstring.

or using preg_match.

Sledgehammer on a fly.

@Seldaek What do you prefer? Custom function? mb_check_encoding ? Something else?

Best regards,
Laurent Lyaudet

[LLyaudet]

@Seldaek What do you prefer? Custom function? mb_check_encoding ? Something else?

I have my answer: I will use a custom function.

mb_check_encoding only returns a boolean.
It is not suitable for a linter that should say on which line/which offset is the current problem.