feat(zip): Support deflate compression (bring your own) by kriskowal · Pull Request #2997 · endojs/endo

kriskowal · 2025-10-28T00:00:40Z

Preface

Endo has its own implementation of ZIP because a survey of the options available on npm, I found that it would be easier to create a library that was pure JavaScript, would work in the XS engine, would operate directly on Uint8Arrays, and would have received sufficient scrutiny to be sure it did not inadvertently invite pathological decompression. Many legacy libraries entrained vestigial code for modeling ZIP files as byte strings (which would have invited pathological performance on XS), entrained multiple compression systems regardless of whether there were needed, and had unnecessary layers of abstraction. It is a safe bet that they were generally designed to tolerate the widest variety of valid ZIP files rather than strictly those that were well-formed. Many were quite old and missed opportunities to use modern idioms. In short, our need was for a more compact, stricter ZIP library.

It proved to be straightforward to implement the subset of ZIP we needed, allowing folks to use off-the-shelf ZIP utilities to open the archives, but obligating them to use our bundler to create them. This also gave us an opportunity to scrutinize the implementation and to take the strict road instead of the permissive road when presented the option, in order to obviate hazards another implementation might have missed like overlapping content regions and explosive decompression. We omitted compression entirely.

In the intervening years, the web and Node.js beginning with version 20 provided native support for asynchronous raw deflate compression and decompression. So, adding support for DEFLATE has become a low-hanging fruit for those environments. And, reflecting on our own qualms with other libraries, gave us an opportunity to add compression without forcing all dependent projects to entrain the gamut of compression concerns.

One of our central library design principles is to avoid barrel modules. Or, generally, a dependent package should not need to entrain any modules it is not using. So, if you are reading a zip file, you should not be obligated to retain the code for writing a zip file, and vice versa. if you do not need compression, you should not entrain a compression library.

We are generally also fans of the principle of least authority, which leads us to generally inject dependencies.

So, we use dependency injection to both select and enable compression for ZIP archives. We also leave open the option of adding support for other compression systems with additional dependency injection.

We continue to omit entirely the concern of being able to read or write ZIP files that do not fit in a processes’s memory, for which we would want to rely on an asynchronous storage with seek, read, and write capabilities. This we leave as an exercise for the event of a future need.

Description

This change adds support for DEFLATE compression and decompression to the @endo/zip package. This feature is enabled by injecting the deflate or inflate capability to the ZipReader or ZipWriter constructor. The web and Node.js 18 provide the necessary facilities without any additional dependencies.

So, we reserve the option to package @endo/deflate in the future, using browser and node conditions to use the native support in those environments, and falling through to a JavaScript implementation by default.)

Security Considerations

None, to my knowledge.

Scaling Considerations

None, to my knowledge.

Documentation Considerations

This change includes updates for README and NEWS.

Testing Considerations

This change minimally tests the new asynchronous DEFLATE support.

Compatibility Considerations

This change deprecates the read and write functions because they were previously synchronous and when providing both synchronous and asynchronous functions, we prefer to qualify the synchronous variant. To that end, we’ve introduced get, getNow, set, and setNow. The deprecated function remains in place.

Although introducing asynchrony is necessary to embrace the platform’s asynchronous DEFLATE, we manage to preserve backward compatibility by ensuring that the ZipReader and ZipWriter continue to lack compression by default. Injecting DEFLATE features obligates the user to change usage. This is a minimal hindrance in practice since the primary user is CompartmentMapper which uses adapters to treat the ZIP archive as an async file system, since it must assume that some storage systems are async-only.

Upgrade Considerations

None.

Copilot

Pull request overview

This pull request adds DEFLATE compression and decompression support to the @endo/zip package through dependency injection. The implementation maintains backward compatibility by keeping files uncompressed by default and requires users to opt-in to compression by providing deflate/deflateNow or inflate/inflateNow functions.

Changes:

Introduces async compression/decompression methods (set/get) alongside sync methods (setNow/getNow)
Deprecates read() and write() methods in favor of the new explicit sync/async API
Updates type definitions to support compression callbacks and richer file metadata

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
packages/zip/src/writer.js	Adds compression support with dependency injection, introduces `set`/`setNow` methods, deprecates `write`
packages/zip/src/reader.js	Adds decompression support with dependency injection, introduces `get`/`getNow` methods, deprecates `read`
packages/zip/src/types.js	Updates type definitions to include compression/decompression callbacks and compression-related file metadata
packages/zip/src/format-writer.js	Simplifies file encoding pipeline to work with pre-compressed content
packages/zip/src/format-reader.js	Removes CRC-32 checking from format reader (moved to higher-level reader), streamlines file record processing
packages/zip/src/compression.js	Adds DEFLATE constant (8) alongside existing STORE constant
packages/zip/test/zip.test.js	Adds tests for compression/decompression round-trip and reading deflated archives
packages/zip/test/_fixture.zip	Adds fixture file for testing native DEFLATE-compressed ZIP archives
packages/zip/reader.js	Exports `readZip` helper function
packages/zip/writer.js	Exports `writeZip` helper function
packages/zip/README.md	Comprehensive documentation of new compression features and updated API
packages/zip/NEWS.md	Documents DEFLATE support in upcoming release
packages/zip/package.json	Updates package description
packages/compartment-mapper/test/stability.test.js	Adds defensive optional chaining for file access
packages/compartment-mapper/test/retained.test.js	Adds defensive optional chaining for file access
packages/compartment-mapper/test/integrity.test.js	Updates tests to maintain consistency between compressed/uncompressed lengths, fixes error messages

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-30T05:43:02Z

packages/zip/src/writer.js

+      type: /** @type {const} */ ('file'),
      date,
+      mode,
+      crc32: crc32(content),


The CRC-32 checksum is being calculated on the compressed content instead of the uncompressed content. According to the ZIP specification, the CRC-32 should be computed on the uncompressed data. This will cause integrity check failures when reading ZIP files created with compression enabled.

Copilot · 2026-01-30T05:43:02Z

packages/zip/src/writer.js

+      type: /** @type {const} */ ('file'),
+      compressionMethod,
+      date,
+      crc32: crc32(content),


The CRC-32 checksum is being calculated on the compressed content instead of the uncompressed content. According to the ZIP specification, the CRC-32 should be computed on the uncompressed data. This will cause integrity check failures when reading ZIP files created with compression enabled.

Copilot · 2026-01-30T05:43:02Z

packages/zip/README.md

+### Helper functions
+
+The package also exports constructor-free adapters.
+These make the archive more like a file system by adding gratuitious


Typo in the word "gratuitious" - should be "gratuitous".

Suggested change

These make the archive more like a file system by adding gratuitious

These make the archive more like a file system by adding gratuitous

Copilot · 2026-01-30T05:43:02Z

packages/zip/README.md

+const zipBytes = writer.snapshot();
+```
+
+#### Options for `write()`


The documentation references a write() method but the example uses set(). For consistency with the rest of the documentation, this should reference set() since write() is deprecated.

Copilot · 2026-01-30T05:43:03Z

packages/zip/test/zip.test.js

+const deflate = compressedBytes => compress(compressedBytes, 'deflate-raw');
+const inflate = uncompressedBytes =>
+  decompress(uncompressedBytes, 'deflate-raw');


The variable names are backwards: the function compress is assigned to a variable called deflate, and the parameter uncompressedBytes is passed where compressedBytes is expected. Similarly, decompress is assigned to inflate with compressedBytes passed as uncompressedBytes. The parameter names in the helper functions should be swapped for clarity.

Suggested change

const deflate = compressedBytes => compress(compressedBytes, 'deflate-raw');

const inflate = uncompressedBytes =>

decompress(uncompressedBytes, 'deflate-raw');

const deflate = uncompressedBytes => compress(uncompressedBytes, 'deflate-raw');

const inflate = compressedBytes =>

decompress(compressedBytes, 'deflate-raw');

Copilot · 2026-01-30T05:43:03Z

packages/zip/README.md

+The implementation operates on whole ZIP archives in memory and operates
+exclusively on `Uint8Array` file contents.
+
+The library does entrain a specific DEFLATE compressor or decompressor, but it


The README statement "The library does entrain a specific DEFLATE compressor" should be "The library does not entrain a specific DEFLATE compressor" based on the context. The word "does" should be "does not" to match the intended meaning.

Suggested change

The library does entrain a specific DEFLATE compressor or decompressor, but it

The library does not entrain a specific DEFLATE compressor or decompressor, but it

Copilot · 2026-01-30T05:43:03Z

packages/zip/src/reader.js

+    const { inflateNow, inflate = inflateNow } = reader;
+    if (!inflate) {
+      throw new Error(
+        `Cannot decompress ${name} in ZIP file ${reader.name}: no synchronous inflate implementation configured`,


The error message says "no synchronous inflate implementation configured" but this function accepts either inflate (async) or inflateNow (sync) via the fallback pattern on line 124. The error message should say "no inflate implementation configured" to accurately reflect that either implementation would work.

Suggested change

`Cannot decompress ${name} in ZIP file ${reader.name}: no synchronous inflate implementation configured`,

`Cannot decompress ${name} in ZIP file ${reader.name}: no inflate implementation configured`,

Copilot · 2026-01-30T05:43:03Z

packages/zip/README.md

+
+```javascript
+const writer = new ZipWriter({ deflateNow });
+writer.write('data.txt', textEncoder.encode('Data...'));


The documentation uses the deprecated write() method. Since this PR deprecates write() in favor of setNow(), the example should be updated to use writer.setNow('data.txt', textEncoder.encode('Data...')) to align with the new API.

Suggested change

writer.write('data.txt', textEncoder.encode('Data...'));

writer.setNow('data.txt', textEncoder.encode('Data...'));

Copilot · 2026-01-30T05:43:04Z

packages/zip/src/reader.js

   * @param {string} name
   * @returns {Uint8Array}
+   * @deprecated Use {@link getNow} for a direct replacement. Use {@link get}
+   * if you need async decompression}.


The JSDoc comment has a typo: "if you need async decompression}" has an extra closing brace. Should be "if you need async decompression."

Suggested change

* if you need async decompression}.

* if you need async decompression.

Copilot · 2026-01-30T05:43:04Z

packages/zip/README.md

+const reader = new ZipReader(zipBytes);
+
+// Read a file (synchronous for uncompressed files)
+const fileBytes = reader.read('hello.txt');


The documentation still uses the deprecated read() method. Since this PR deprecates read() in favor of getNow() and get(), the documentation should be updated to use the new methods. The example should use reader.getNow('hello.txt') instead of reader.read('hello.txt') to align with the new API.

Suggested change

const fileBytes = reader.read('hello.txt');

const fileBytes = reader.getNow('hello.txt');

kriskowal · 2026-01-31T12:45:00Z

I poked at this a little bit and have decided to hold it back until a more significant effort that would add this feature, but also rename the package zip (2.0.0) so it can leave some cruft behind. It makes sense to do that and make the new zip package a hardened module or at least wrap it as a hardened module with @endo/zip. And, for that, we should wait for #3008 to land.

changeset-bot · 2026-02-21T01:28:45Z

🦋 Changeset detected

Latest commit: 4871cd0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages

Name	Type
@endo/zip	Minor
@endo/bundle-source	Patch
@endo/check-bundle	Patch
@endo/compartment-mapper	Patch
@endo/cli	Patch
@endo/daemon	Patch
@endo/import-bundle	Patch
@endo/test262-runner	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

kriskowal force-pushed the kriskowal-zip-compression branch 3 times, most recently from 0ec8fec to 5c472d3 Compare October 28, 2025 21:43

kriskowal requested review from danfinlay and erights January 4, 2026 07:28

kriskowal requested a review from Copilot January 30, 2026 05:37

Copilot started reviewing on behalf of kriskowal January 30, 2026 05:38 View session

kriskowal force-pushed the kriskowal-zip-compression branch from 5c472d3 to edeaefb Compare January 30, 2026 05:38

Copilot AI reviewed Jan 30, 2026

View reviewed changes

kriskowal marked this pull request as draft January 31, 2026 12:45

feat(zip): Support deflate compression (bring your own)

4871cd0

kriskowal force-pushed the kriskowal-zip-compression branch from edeaefb to 4871cd0 Compare February 21, 2026 01:28

	These make the archive more like a file system by adding gratuitious
	These make the archive more like a file system by adding gratuitous

	The library does entrain a specific DEFLATE compressor or decompressor, but it
	The library does not entrain a specific DEFLATE compressor or decompressor, but it

	`Cannot decompress ${name} in ZIP file ${reader.name}: no synchronous inflate implementation configured`,
	`Cannot decompress ${name} in ZIP file ${reader.name}: no inflate implementation configured`,

	writer.write('data.txt', textEncoder.encode('Data...'));
	writer.setNow('data.txt', textEncoder.encode('Data...'));

	* if you need async decompression}.
	* if you need async decompression.

	const fileBytes = reader.read('hello.txt');
	const fileBytes = reader.getNow('hello.txt');

Conversation

kriskowal commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Preface

Description

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

Compatibility Considerations

Upgrade Considerations

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

kriskowal commented Jan 31, 2026

Uh oh!

changeset-bot bot commented Feb 21, 2026

🦋 Changeset detected

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kriskowal commented Oct 28, 2025 •

edited

Loading