Non-ECMAScript JS APIs requested in Hermes

[tmikov]

This is intended to be a discussion and collection of non-ECMAScript APIs that have been requested by the community for implementation in Hermes. While the position of the Hermes team has been that we can't afford to keep adding such "extended" APIs to the core of Hermes, we acknowledge that this is a real problem, since such requests have been made more than once and have remained more or less unaddressed.

The usual solution of "find an NPM" for the missing functionality is not satisfactory, because:

• There are no guarantees of quality of every NPMs
• There is no consistency. Every single app doesn't need a different implementation of URL.
• This problem does not exist in browser or NodeJS where all of these APIs are provided out of the box by the integrating app.

A good solution would at the very least have to overcome the following challenges:

• Modularity. An app shouldn't carry megabytes of unused dead weight. (This is one of the reasons why these APIs can't be part of "core Hermes").
• Quality, maintenance and support. If these APIs are provided by a centralized location, it has to provide a minimal guarantee of quality and commitment to fix future problems.
• Convenience. It goes without saying, but it should be easy to use and provide close to the "out of the box" experience of NodeJS.

Before we can start figuring out what a practical solution might be, we also need to understand which APIs need to be present.

Please comment with ideas or APIs that you would want to see out of the box.

[tmikov]

crypto.getRandomValues()

See #915, #1003 for motivation.

[tmikov]

URL

See #132 and #987 .

[joshuayoes]

This would remove the need for packages like react-native-url-polyfill, which has 400,000+ downloads.

I see a lot of projects add query-string as well, which even just URLSearchParams would be enough to remove as a dependency.

[farwayer]

First class support will be great. Especially in context of using deep linking with Universal Links/App Links. URL is not ECMAScript but this is near the basic functionality that many applications need.

[jthoward64]

Of note: Expo is now shipping URL and URLSearchParams with SDK 50

[tmikov]

TextEncoder

See #948 .

[cvarjao]

should I add TextDecoder as a different item, or would the encode/decoder bundled as 1?

[tmikov]

This is not at the stage of a formal list yet; for now we are just gathering information. Now that you have mentioned it, I think it is sufficient.

[tmikov]

TextEncoder has been implemented as of 3863a36.

So, I am adding a separate item for TextDecoder.

[tmikov]

structuredClone

See #684 .

[CanadaHonk]

process.argv/etc?

[tmikov]

This is an interesting case, because it seems very platform specific. I am not sure to what degree it makes sense in an Android GUI app for example, and AFAIK iOS allows almost no process control.

Still, if we manage to solve the problem of modularity it would be possible to add it.

[CanadaHonk]

Math extensions like Math.clamp(), Math.signbit(), Math.scale(), Math.radians()?

[tmikov]

This is a bit tricky. I believe these would be relatively cheap to add, but are they on track for standardization, are they present in major platforms like browsers and NodeJS, or are they completely non-standard?

If it is the latter, I think it would be better to add them to an extension namespace that is clearly Hermes-specific or at least clearly non-standard.

[CanadaHonk]

There are a few proposals but they are tricky; most are Stage 1/0 and/or inactive:

• Math.clamp Proposal: Math.clamp (stage 0 - last commit april 2023)
• Math Extensions Proposal: Math.scale, Math.radians, Math.degrees, Math.RAD_PER_DEG, Math.DEG_PER_RAD (stage 1 - last commit september 2020)
• Math.signbit Proposal: Math.signbit (stage 1 - last commit february 2020)

No major platforms implement these at all yet (Math or custom global) afaik, people just implement them for themselves.

[tmikov]

I have seen this Tweet: https://twitter.com/C4nadaHonk/status/1681791910271242240 :-)

Seems like at least the names are agreed upon.

[ljharb]

I wouldn’t be so sure; nothings set til stage 3.

[ljharb]

signbit and the extensions proposal were both made inactive last week.

[ahmed-adly-khalil]

crypto.getRandomValues() and TextEncoder would be highly appreciated as they open the use case for websockets and other scenarios

[shamilovtim]

crypto

webcrypto (crypto.subtle)

stream

[joshuayoes]

Currently using react-native-webview-crypto for the crypto.subtle API

[CanadaHonk]

WinterCG would likely be useful for what to possibly add, especially Common Minimum API.

[jthoward64]

Didn't know either of these existed, but both seem like a great way to set a baseline target. I'd be very interested in seeing Hermes make some progress towards meeting them

[shamilovtim]

Sadly Hermes is nowhere close to that. I view this as flaw of the react-native runtime which should have formally defined meeting these APIs as a priority and marshaled the community into fulfilling them. Instead react-native core was thinned out which redirected the community into making ad-hoc mobile packages rather than meeting web specs. This problem confuses external stakeholders and generates some negativity towards the framework because they see JS and other common runtime APIs like fetch or flexbox and think React Native is a web compliant framework when it's more of a tiny subset of Node.js APIs. I still haven't seen a cohesive plan for this problem and it festers, undermining the framework.

[tmikov]

Hermes is just a JS engine. There never has been a requirement or expectation that Hermes should implement anything beyond the EcmaScript spec. No other JS engine does it either.

With that said, we realize that the distinction is only philosophical, users need these extended APIs and don't really care who is responsible for providing them. That's why we created this discussion and have started implementing some of the APIs, specifcially atob() and btoa().

As I mentioned in my opening post, part of the problem is that we can't add all possible APIs to Hermes by default - it would become too large. We are coming up with a plan for an "SDK" for Static Hermes, where APIs are automatically linked only if used.

Also see #1170.

[shamilovtim]

@tmikov agree completely. I don't think the Hermes engine should provide these APIs. I'm aware that it's just a JS engine and indeed, switching to JSC or V8 wouldn't make these APIs suddenly appear.

[shirakaba]

Node-API

...
(EDIT: comment moved to top level discussion here: #1074 by @tmikov. Please respond there.)

[tmikov]

Note: this is an important topic, but it doesn't technically belong in this discussion since it not about a Non-ECMAScript JS API , so to get proper visibility I moved it to a top level discussion here: #1074.

(I am responding there)

[avp]

atob, btoa

See #1178

Implemented in 13fafde and d2177c3.

[shamilovtim]

atob/btoa using JSI / JNI: https://github.com/craftzdog/react-native-quick-base64

[tmikov]

Implemented in 13fafde and d2177c3.

[ahmed-adly-khalil]

@tmikov thanks so much, now we can use RN in 1 less hacky way :)

[ljharb]

It'd also be great to implement, sooner than later, https://tc39.es/proposal-arraybuffer-base64/spec/ which just got stage 3 last week :-)

[evelant]

ReadableStream (and TextEncoder/TextDecoder)

I need ReadableStream https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream and it seems there's no easy way to polyfill it with reasonable performance right now. It's needed in combination with TextEncoder and TextDecoder for react-native to be able to support streaming responses in fetch.

@tmikov are any of these APIs likely to get implemented in hermes at all? or anytime soon? They are supported in literally every other JS engine, browser or not. Hermes and react-native get continuously harder to use as more time goes by with missing APIs that are available everywhere else. Whether or not those APIs are part of the ECMAScript spec doen't make much practical difference when they're available and effectively standard everywhere.

[evelant]

Thanks for the thoughtful reply! Totally makes sense and I'm sorry for mixing up runtime features with the engine.

From the perspective of people developing a react-native app (like myself) it's easy to mix up engine and runtime because the development experience boils down to "we can't use features that have been in node, deno, bun, and all major browsers for years". That can be frustrating, especially because react-native is internally very complex so even figuring out how to start adding the missing features is a big challenge.

Thanks for all your hard work on hermes, it's a cool project (looking forward to seeing where static hermes goes!).

[jthoward64]

If you need extra performance with a JS API the current solution (AFAIK) is mainly writing native code, so maybe look into something like Expo Modules or new arch turbo modules. There's also the older native modules

[tmikov]

Ideally all of these can be implemented in typed JS using Static Hermes. We will soon start implementing the JS library in JS, figuring out what is missing, making easier, etc.

[tmikov]

@evelant Hermes now supports TextEncoder.

[guest271314]

Hermes now supports TextEncoder.

So you arbitrarily pick and choose which non-ECMA-262 API's are implemented?

[tmikov]

TextDecoder

[jthoward64]

I believe only TextEncoder is supported, I don't see anything about TextDecoder in JSLib. Even so, you will only see any of these in a new react-native release until such a time as what tmikov mentioned above exists

[pfista]

+1 on TextDecoder support

[Badbird5907]

+1
need to pollyfill atm to support tRPC's http batch link.

[aretrace]

First it was TextEncoder, now it's TextDecoder, soon it'll be ReadableStream 😱
Just Say Yes 👵

[tmikov]

We accept PRs 🤣

[theoludwig]

crypto.randomUUID()

https://developer.mozilla.org/en-US/docs/Web/API/Crypto/randomUUID

[jthoward64]

Prior work: https://github.com/expo/expo/tree/main/packages/expo-crypto

[guest271314]

I think you want a complete implementation of Web Crytography API, not Node.js-specific crypto module, which cannot be polyfilled.

[gre]

CompressionStream

e.g.to be able to have built-in gzip support

https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API

[guest271314]

Read standard input into an ArrayBuffer, and write to standard output as an ArrayBuffer, TypedArray, or Blob (since WHATWG Streams are not ECMA-262 specified, yet have been excluded from consideration above).

[maceip]

socket()

posix socket access

Pros:

• enables p2p; certain mpc protocols need direct socket access
• would reduce websockify proxy dependency

Cons

• bricks origin

W3g https://wicg.github.io/direct-sockets/

[guest271314]

W3C Direct Sockets is an implementation of TCP and UDP client and TCP server in the browser, not in a JavaScript runtime. See https://github.com/guest271314/direct-sockets-http-ws-server.

[guest271314]

Pardon, WICG Direct Sockets.

[maceip]

I think I understand the differences but wanted to paste it here anyways. Just for reference

[lucaswitch]

Worker

https://developer.mozilla.org/pt-BR/docs/Web/API/Worker

Currently we're locked on js thread to do cpu heavy work, sure we have an npm package that can handle that but it doesn't come close of having such api bundled.
The ones that we got on react-native differs by:

• They all try to share variables between two separate hermes engines.
• Relies on babel transforming to be able to run.

Workers can work as an short lived separated runtime to execute some heavy work or even a persistent runtime to be run alongside with the react-native app.

Having that possibility will lead to react-native devs to split calls between different hermes runtimes leaving the main js thread lighter to render react components.

[guest271314]

I/O

Straight up capability to read STDIN and write to STDOUT without including C to do that. Something like what most modern JavaScript engines and runtimes have

globalThis.std.in.read()

or

process.stdin.read()

or

await Deno.stdin.readable.getReader().read()