MultiThreading Architecture : Brainstorming

[PhoenixIllusion]

I looked into the existing discussion about multithreading and the limitations that were run into (and reproduced) regarding Safari.

Approaching this as more of a custom JobPool solution with worker-api (not-pthread), I was able to confirm that at a raw SharedBufffer + Atomics level, Safari has no issues in general with Atomic store/load/add/sub/wait/notify. I was code to test on Safari a trivial test (non-JoltJS) case of launching a scheduler thread that then shared-mutex-signaled workers with shared-mem payloads and waited until it's atomic-controlled active-job-counter either changed or hit zero.

Current theoretical approach would be a JS-Worker-JobPool combined with collection of JobRunner classes.
The main Jolt.JS entry point I focused on was the Step() command and Job->Execute(). The physics Step logic appears to build up jobs and pass them onto the JobPool scheduler, which is then responsible for scheduling them on the threads and waiting until they all finish at the end.

The initial trivial logic was that a custom native C++ JobPool would be created and instantiate a collection of Job Runner Classes (say, numbered 0 to 4). It would do light checks on dependencies but otherwise just stuff the Job Runners full as long as they had unused job-slots.

Each Job Runner pointer is passed to the other worker threads on creation. The worker-thread's job is just "execute the Runner on-launch and wait. Sit on an internal pointer Job-slot. When the internal Job slot becomes non-null, execute that job, set the slot to null, atomic-subtract the ThreadPool job-runing count by 1, and sit on the Job-slot until non-null". Each worker thread doesn't even really need JS schema's besides the 1x emscripten command of "Job Runner class :: run( $ptrThis )"

The complicated in my planning logic was for any Job Runner with an assigned Job that has callbacks. The callbacks will work for any functions that exist in CPP, since those technically exist in the shared-memory buffer. The problem is JS-Interface callbacks. Those only exist on the original JS layer, in the JSContext they were created.

I see 4 possibly options for this:

1. Collect all collisions and fire the OnCollision callbacks after the Step finishes. This is useful, but breaks any Velocity modification or Collision Validation logic
2. The most common collision logic gets put into a configurable C++ objects. This can be combined with # 1 to cover more use-cases. This means that any common collision behavior can only work with Jolt-specific data that is available or copied-into the WASM layer every frame.
3. Replicate all custom JSContext callbacks into every Runner thread on-start. This works for some fancier objects, but these replicated Callbacks can only reference data that is either in the WASM layer, or is backed by a SharedArrayBuffer so that any important values influencing decisions time are reflected accurately across all threads.
4. Enforce that any Jobs that would have JS-Callbacks are run only on the AppCode JSContext (say, magic-number assign Job Runner Zero). The JobPool needs to flag specific job-names and only assign those to the AppCode Context. The negative part of this is that it seems any collision-related threads would now all be forced to run single-threaded style and those are one of the more common parallel jobs that I have seen.

My diagram has the Step() command run outside of the AppContext, because the execution flow would run:

1. On App JSContext, Signal JobPool thread to call Physics.Step(), App JSContext goes to # 2
   a. On JobPool Thread - Step() starts
   b. Thread Pool Runs
   c. Schedule and block/wait on threads until all Jobs are done
2. On App JSContext, Start Job Runner Zero()
3. Run any jobs provided until ThreadPool indicates last Step() Job is finished

[PhoenixIllusion]

I only moved the main AppCode into a worker because I was going to Atomic.wait on SharedBuffer values which requires WorkerThread execution, but technically the Job Runner Zero could be made into a more JS-compatible mode.
That could be done if I converted the Atomic locking behavior into setTimeout(0) spin-locking-ish on the atomic signal value that a job is available? That could pull the app-execution back onto the UI-thread but I'm not sure how much it delays the signaling. I'd seen some emscripten code say "do a true spin-lock with Atomics.load" but that sounds horrible.
https://github.com/emscripten-core/emscripten/blob/main/system/lib/wasm_worker/library_wasm_worker.c#L130

[PhoenixIllusion]

Technically it could have the JS layer use the Atomics.waitAsync instead of setTimeout, now that I see it requires a min of 4ms if used too much

[jrouwe]

Hello,

Thanks for looking into this! I have to say that I'm pretty noobish when it comes to JavaScript, I have never worked with web workers, so excuse me if I make stupid suggestions.

The initial trivial logic was that a custom native C++ JobPool would be created and instantiate a collection of Job Runner Classes (say, numbered 0 to 4). It would do light checks on dependencies but otherwise just stuff the Job Runners full as long as they had unused job-slots.

Would it be possible to copy paste the JobSystemThreadPool code and remove:

https://github.com/jrouwe/JoltPhysics/blob/515933138c6b16d661452fb907a8a9bbb71cb848/Jolt/Core/JobSystemThreadPool.cpp#L64-L68

(and a bit of code while destroying threads)

and then call JobSystemThreadPool::ThreadMain from each JS Worker? That would mean that the existing queue and semaphore logic would continue to work and no spin-locks are needed.

Collect all collisions and fire the OnCollision callbacks after the Step finishes. This is useful, but breaks any Velocity modification or Collision Validation logic

This would severely limit the things you can do with the library, so would not have my preference.

The most common collision logic gets put into a configurable C++ objects. This can be combined with # 1 to cover more use-cases. This means that any common collision behavior can only work with Jolt-specific data that is available or copied-into the WASM layer every frame.

This would also limit options a lot (and I wouldn't exactly know what 'common collision behavior' would be).

Replicate all custom JSContext callbacks into every Runner thread on-start. This works for some fancier objects, but these replicated Callbacks can only reference data that is either in the WASM layer, or is backed by a SharedArrayBuffer so that any important values influencing decisions time are reflected accurately across all threads.

I think this would have my preference. Most of the time you can get all the information you need from the body/bodies that you're colliding with and the data from these bodies comes from the WASM layer. If I quickly scan the JS worker documentation then it sounds like you can pass in parameters with each worker, so if you need some global context (like which body ID is a conveyor belt) then could you pass that in as a param?

Enforce that any Jobs that would have JS-Callbacks are run only on the AppCode JSContext (say, magic-number assign Job Runner Zero). The JobPool needs to flag specific job-names and only assign those to the AppCode Context. The negative part of this is that it seems any collision-related threads would now all be forced to run single-threaded style and those are one of the more common parallel jobs that I have seen.

Since you don't know ahead of time if a job is going to do a collision callback, this would mean you need to run all collision jobs on the main thread (including the broadphase job as that can also trigger collision callbacks), so essentially you would lose all parallelism for about 50% of the work in PhysicsSystem::Update.

[jrouwe]

B.t.w. I also haven't exposed the 'user data' (a uint64) that comes with each body yet. This could also be used to pass state from main thread to job threads.

[PhoenixIllusion]

The recommended modifications for JobSystemThreadedPool is probably a good starting point. It will require porting most of the Thread-specific methods and changing some class dependencies in ports (likely rewrite to use the same interfaces), since the referenced emscripten_worker_thread atomic/lock code is the actual closest-to-JS implementations of semaphore. pThread/StdLib versions of semaphore will explicitly not work with worker threads, and pThread is not encouraged for WASM performance, since it polyfills communication between all threads with tons of JS-layer postMessage/onMessage/postMessage back and forth for notifications, joins, creation, destruction, cancels, etc. Web Workers are more 'create it once, never dies, it is up to you to manage forever'
https://github.com/emscripten-core/emscripten/blob/main/src/library_pthread.js#L1255

Conceptually it is easier if the threads don't ever really plan on being destroyed. We can have a cleanup phase, yes, but for the general worker threads the simpler logic is that they are dedicated to process Jobs essentially forever. They can have cleanup logic trigger after the JobPool is done with an iteration, but they are not general-purpose threads so will dedicate their existence to just JobSystemThreadPool::ThreadMain.

• Collisions *
  To simplify the logic, I'm still going to conceptually split the logic of "Collision Detection" + "Collision Immediate Response",

For just "Detection", these likely want to be heard by your main App engine which shouldn't be replicated to every thread. You likely would still either want to keep some list of collision hits and process those back on the main thread after the physics step finishes, or each collision could be setup to trigger a cross-thread postMessage broadcast so that the App JSContext can hear them when they occur. I think the postMessage fire-and-forget logic shouldn't have too much overhead if they aim them back to the App JSContext and don't wait for a reply.

• Callbacks - Function Replication *
  The logic to replicate callbacks across threads is also a very costly task, so likely wants to be done infrequently or once on-start. The easiest way to do it is to pre-plan all methods ahead and compile it into their own JS module, and at boot time each thread import the callbacks module. By default, random runtime creation of callbacks is risky since the JS-callbacks may not have been setup on the worker threads by the time you actually run Step() and expect them to be 'installed'.

For more dynamic (and ugly) syncing the methods across threads, one alternative technique I've seen is to grab or dynamically compile the runtime JS source, stringifying it, and sending it over to the workers. There it is encoded into Data URIs and JS import-ed.

Replicating callback functions will require that we do a full Jolt.JS module import on the other classes to access the function-table logic that triggers the callbacks, but we likely just need to import the pure wasm-version in the Workers instead of compat-version, since the AppContext can pass the wasm-compat module over to the Workers after compilation.

In the workers, we should be free to just call wrapPointer on the raw pointer values in each thread and the the callbacks will auto-setup in the Jolt module, allowing us to be free to assign the actually callbacks on the returned wrapPointer object.

Currently the pointer-object hash-cache is tied to every class individually. A more unstable (but flexible) logic to handle the replicated callbacks is to find out what the compile-time name of the cache (it is PL in the existing binary) and rewrite that cache with a JS Proxy that doesn't care about the Pointer values ahead of time, so we can give a callback that will trigger on every single callback request for that JSClass , and pass it the Pointer ID along with the callback params.

• User Data and Shared JS-Level Data *
  The UserData can be used to pull serialized data stored in an non-wasm shared buffer.
  Safest is to always tie to shared-array data, so I would go with offsets.

Possibly:

• protobuf offsets (discouraged from modifying in-place, not ideal)
• Custom classes with internal ArrayBuffer data-views and getter/setters at offsets
• JS interfaces combined with Proxies that read/write offsets into the buffer (similar to above)
• For Typescript users, the above elements could probably be made into @decorators

[PhoenixIllusion]

Tried to do the trivial implementation on https://github.com/PhoenixIllusion/JoltPhysics.js/blob/feature/web-worker-threadpool/

Currently I've run into some issues with thread_local logic in the system is messing things up. I think I'd need to get more familiar with the inner workings to get through debugging all of the multi-threaded access. Kind of frustrating, and I am not use to CPP in wasm for the multi-threading.

I learned that the WASM mulit-threading requires some stack setup since each thread will have it's own thread, and that it is not easy to grab the workers after they spawn from inside WASM.

I gotten as far as having a bunch of thread-access asserts trigger, but during debugging the Profiler was also throwing issues from thread_local references. Mainly lots of out-of-bounds exceptions for now.

[PhoenixIllusion]

I think I am getting to the end of the function? Just getting bad refs on the destructors:

[PhoenixIllusion]

... the app is being compiled with a TOTAL stack size of 1mb, which I guess is shared among all threads...

[PhoenixIllusion]

I had been running into an issue with while (head != mTail), but found out it was because the execution thread was keeping one of the mHeads slots open but didn't use it, so it would always be set to 0 and fail the tail-head > max-len checks.

Briefly seemed Safari was working, but now it's just replicating the same issue with pThread of items falling through the ground. I'm probably just going to leave the branch as-is for now.

[jrouwe]

Ok too bad. I might take a look at it later but for now I seem to have caught the flu, so I'll probably be afk for a while

[PhoenixIllusion]

The inconsistency with iOS/Safari may be happening for a while. It's hard to dig into since Safari does not seem to have good debugger support on Web, and only updates it infrequently. I can rarely even get a javascript page to show source when the multithreading is running. Likely will need to wait for them to progress more on WebKit's support for multi-threading to see changes. I have Chrome, Firefox, and Mac OS Sonoma 14 - Safari Technology Preview channel.

The discussion here at least does give some starting points on planning, useful even if reverting to pure pThread build, since we will still need to handle callbacks across threads regardless.