jsonc support

[farzadmf]

Hi, I just installed your tool and tried doing hson <file.json>, where the file also contains comments (jsonc file basically), and I get an error:

Error: failed to parse input file: ./nvim/misc-files/vimspector/go.json

Caused by:
    InternalError(TapeError) at character 0 ('/')

[kantord]

oh that is an interesting problem! honestly I haven't worked much with JSONC files, so I'm not entirely sure what the expected behavior/perfect solution would be here. But basically I think this is the problem we're facing:

• this project uses simd_json for parsing JSON files, which is an extremely fast JSON parser, but does not support json comments. And it seems like basically none of the ultra-fast JSON parsers do
• using a very fast JSON parser allows hson to work on gigabyte-scale files in trivial times and to work extremely fast on small JSON files (i. e. easy to build it into a high performance API)
• at a first glance it doesn't seem like there are any mature JSONC libraries for Rust; nor to strip them
• plus it does not seem like, at least in your example, that there is a way to distinguish JSON vs JSONC just based on the filename - this is a problem for the multi-file mode where the parser is selected based on the file extensions

The ideal solution would be if sth like simd_json supported comments, but I imagine it would be hard to achieve the same performance with more complex syntax. Or else if we could at last use a dependency to strip the comments without a major performance cost, or with a heuristic that only adds the performance cost to JSONC files, not to all JSON files.

This is a conundrum, with no clear answers so I'm really looking for some input here.

In any case, in the simplest case of using it to summarize a single JSON file, you should probably rely on a simple shell one liner using whatever your preferred way it to strip comments from jsonc. Something like this should work today with no issues:

cat my-file.json | my-preferred-jsonc-comment-stripper | hson

You could of course also alias it in your bashrc/zshrc or add a little helper function around it for convenience. That would be pretty much a standard solution.

But this still leaves the problem of multifile mode, because even if you are using a separate JSONC comment stripper, that doesn't help you in the case of multi-file mode. I guess that this would require some magic, such as being able to use a custom JSON filter, or custom file-extension-based filters in general. (To establish rules like "filter .xyz files through foo before summarizing them)

Also, even if this was perfectly implemented, the comments would be stripped from the summaries. Keeping comments in summaries would actually probably require a restructuring of hson-s entire internal representation of tree structures because there is no place to put the comments basically 🤔 I mean, comments can probably appear in weird places where other values couldn't 🤔

[farzadmf]

@kantord Sorry that I disappeared on you after opening the issue and didn't reply.

First of all, thank you for taking the time to provide an answer with such detail. Your suggestion of stripping the comments could be useful, but as you yourself mentioned:

• Not easily feasible (probably impossible?) for multi-file inputs
• Defeats the purpose of having those comments in the first place
• Would be nice to show the comments as well when using hson (as I guess happens "normally" with other files)

I'm honestly surprised about why JSONC has such a limited support everywhere (Rust included).

... there is a way to distinguish JSON vs JSONC just based on the filename

I don't think this is always the case; I personally have .json files with comments (although it's very likely that I'm being stubborn and I do need to change them to be .jsonc)

As for performance implications, I'm honestly like a first-grader in Rust, let alone being able to comment on performance issues etc.

So, I guess I need to accept the fact that I shouldn't put comments in my JSON files 😆 (still shocked honestly about why the industry considers as something niche/weird/unacceptable!)

[kantord]

In case there is a properly Rust library for stripping the comments, then there is no reason to not support jsonc when the file type is clearly marked in the filename or indicated in a CLI flag. Literally all we need for that is a good comment stripping library.

Btw, there is also a competing standard for JSON with comments called JSON5: https://json5.org/

I think the problem with support - especially in Rust - is that people are focusing on using JSON for APIs and non-manually edited file formats where comments have no use, but would complicate the grammar and thereby hurt performance.

I forgot to mention though, that headson already has full support for YAML (without parsing comments). YAML is another superset of JSON that supports comments, although the comment syntax is different. This is perfectly valid YAML:

{
  "hello": 42,
  # Very important comment
  "world": ["foo", 42, "bar"],
}

of course, YAML is much more than just JSON with comments, it supports very different syntax as well.

[farzadmf]

Oh wow, had no idea we can write YAML like that 🤯! I do like YAML, but, probably unfortunately in this case, JSON is much more ubiquitous than YAML (I think!)

[kantord]

yup that's for sure. but if I had to take a guess, YAML is the most popular Json superset, so probably the most widely supported way to put comments in Json, even if the true purpose of YAML is very different than that