Closed
Conversation
… manager cleanup, queue drain, precomputed tar index map Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>
forsyth2
reviewed
Feb 25, 2026
Comment on lines
-552
to
-554
| multiprocess_worker.print_monitor.wait_turn( | ||
| multiprocess_worker, files_row.tar | ||
| ) |
| hpss = config.hpss | ||
| else: | ||
| raise TypeError("Invalid args.hpss={}".format(args.hpss)) | ||
| raise TypeError("Invalid config.hpss={}".format(config.hpss)) |
| except queue.Empty: | ||
| pass | ||
|
|
||
| manager.shutdown() |
| # Drain any remaining failures after all processes have exited. | ||
| try: | ||
| while True: | ||
| failures.append(failure_queue.get_nowait()) |
Collaborator
|
Cherry-picked commit to #402 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Python 3.14 support with minimal implementation changes. The core issue was that Python 3.14's multiprocessing changes caused deadlocks and ordering failures in the parallel extraction path. The fix restores correct parallel behavior without sacrificing the original heap-based load balancing.
DevOps
<3.15insetup.py,conda/dev.ymlfail-fast: falseParallel extraction fixes (
zstash/extract.py,zstash/parallel.py)Removed serializing
wait_turn()call from extraction start — it was blocking workers from downloading/extracting until previous workers finished printing, collapsing parallelism to serial. Output ordering is still enforced insideprint_all_contents(), which callswait_turn()internally before flushing output.HPSS auth fallback —
extractFiles()now falls back toconfig.hpsswhenargs.hpss is None. Previously it raised unconditionally, breaking--workerswithout--hpsswhen the HPSS path came from the database.Manager lifecycle —
manager.shutdown()is called after all worker processes exit, preventing a leaked manager server process.Reliable failure queue drain — replaced
queue.empty()polling (unreliable across processes per Python docs) withget_nowait()/queue.Emptyloops.Precomputed tar→index map in
PrintMonitor—_tar_to_index: Dict[str, int]replaces linearlist.index()lookups; unknown tar names now raiseRuntimeErrorinstead of silently returning.tar.extract()filter for Python 3.12+ — addedfilter="tar"to suppress the deprecation warning introduced in 3.12.Select one: This pull request is...
Big Change
1. Does this do what we want it to do?
Required:
If applicable:
2. Are the implementation details accurate & efficient?
Required:
If applicable:
zstash/conda, not just animportstatement.3. Is this well documented?
Required:
4. Is this code clean?
Required:
If applicable:
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.