Merge opaque closure modules with the rest of the workqueue#50724
Merge opaque closure modules with the rest of the workqueue#50724pchintalapudi merged 2 commits intomasterfrom
Conversation
pchintalapudi
added
compiler:codegen
vtjnash
left a comment
vtjnash
left a comment
There was a problem hiding this comment.
SGTM. Though I hope in the future we can make this workqueue a multi-threaded-supporting channel (once the JIT stops making bad assumptions about needing to optimize in the same order as linking), which is why it wasn't threaded through like this before.
|
Maybe codegen_params can hang onto a reference to (workqueue, compiled_functions, external_functions) and have that be provided by JIT/AOT? That way we should be able to deduplicate both across the JIT and across whatever code is being AOT compiled. |
|
I think each of those are largely an implementation artifact of not being properly isolated from the enclosing scope, rather than an actual proper feature. In a future implementation, those should all be removable since each compilation unit would be self-contained + a list of other units that are required. It would then be up to the system, post-linking, to ensure all of those dependencies are satisfied. Right now all of that happens before compilation, which limits threading scalability of compilation. |
pchintalapudi
added
the
backport 1.10
|
Marking for backport to fix the issues noted by Keno earlier. |
|
@pchintalapudi, the open PRs from you with backport label tend to have a lot of conflicts that I cannot resolve myself. You could take on the effort of cherry picking these on top of #50708? You could just push them there when its done. |
This sticks the compiled opaque closure module into the `compiled_functions` list of modules that we have compiled for the particular `jl_codegen_params_t`. We probably should manage that vector in codegen_params, since it lets us see if a particular codeinst has already been compiled but not yet emitted. (cherry picked from commit 441fcb1)
Backported PRs: - [x] #50637 <!-- Remove SparseArrays legacy code --> - [x] #50665 <!-- print `@time` msg into print buffer --> - [x] #50523 <!-- Avoid generic call in most cases for getproperty --> - [x] #50635 <!-- `versioninfo()`: include build info and unofficial warning --> - [x] #50670 <!-- Make reinterpret specialize fully. --> - [x] #50666 <!-- include `--pkgimage=no` caches for stdlibs --> - [x] #50765 - [x] #50764 - [x] #50768 - [x] #50767 - [x] #50618 <!-- inference: continue const-prop' when concrete-eval returns non-inlineable --> - [x] #50689 <!-- Attach `tanpi` docstring to method --> - [x] #50671 <!-- Fix rdiv of complex lhs by real factorizations --> - [x] #50598 <!-- only limit types in stack traces in the REPL --> - [x] #50766 <!-- Don't partition alwaysinline functions --> - [x] #50771 <!-- re-allow non-string values in ENV `get!` --> - [x] #50682 <!-- Add fallback if we have make a weird GC decision. --> - [x] #50781 <!-- fix `bit_map!` with aliasing --> - [x] #50172 <!-- print feature flags used for matching pkgimage --> - [x] #50844 <!-- Bump OpenBLAS binaries to use the new GEMM multithreading threshold --> - [x] #50826 <!-- Update dependency builds --> - [x] #50845 <!-- fix #50438, use default pool for at-threads --> - [x] #50568 <!-- `Array(::AbstractRange)` should return an `Array` --> - [x] #50655 <!-- fix hashing regression. --> - [x] #50779 <!-- Minor refactor to image generation --> - [x] #50791 <!-- Make symbols internal in jl_create_native, and only externalize them when partitioning --> - [x] #50724 <!-- Merge opaque closure modules with the rest of the workqueue --> - [x] #50738 <!-- Add alignment to constant globals --> - [x] #50871 <!-- macOS: Don't inspect dead threadtls during exception handling. --> Need manual backport: Contains multiple commits, manual intervention needed: Non-merged PRs with backport label: - [ ] #50850 <!-- Remove weird Rational dispatch and add pi functions to list --> - [ ] #50823 <!-- Make ranges more robust with unsigned indexes. --> - [ ] #50809 <!-- Limit type-printing in MethodError --> - [ ] #50663 <!-- Fix Expr(:loopinfo) codegen --> - [ ] #50594 <!-- Disallow non-index Integer types in isassigned --> - [ ] #50385 <!-- Precompile pidlocks: add to NEWS and docs --> - [ ] #49805 <!-- Limit TimeType subtraction to AbstractDateTime -->
KristofferC
removed
the
backport 1.10
3 participants
This sticks the compiled opaque closure module into the
compiled_functionslist of modules that we have compiled for the particularjl_codegen_params_t. We probably should manage that vector in codegen_params, since it lets us see if a particular codeinst has already been compiled but not yet emitted.