Autodoc: Javascript port of zig lexer.

[Myvar]

No description provided.

[andrewrk]

Why is it written in javascript instead of compiling our existing tokenizer to wasm?

This introduces quite a bit of redundant code to maintain, so I think there should be a clear justification.

[kristoff-it]

Why is it written in javascript instead of compiling our existing tokenizer to wasm?

This introduces quite a bit of redundant code to maintain, so I think there should be a clear justification.

We discussed yesterday this in a call and decided to avoid bumping up the minimum requirements for using autodoc until we're confident we want to opt into WASM with more than just a tokenizer. Using a WASM tokenizer also brings in the usual string-related issues where the bytes have to exist both in wasm as shared memory and as a separate copy accessible to the JS engine / browser, increasing memory consumption & slowing down the rendering process.

[Myvar]

@kristoff-it

1) the function generate_html_for_src is implemented in a very naive way, it's just a demo for using the lexer we should build a better version for actual Autodoc

2. testing this is complicated, it does not have to be complicated and we can certainly remove/fix the complexity but that is out of scope for this pr maybe in the future, for now just hit me up on Discord and il show you how testing it works