-
|
Is it possible to add sqlite-zstd extension to |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 2 replies
-
|
Maybe, but it really depends on what you mean. Any extension you can compile and statically link into your own Wasm, is probably fair game. I can't give specific instructions for sqlite-zstd, but you can check how If, however, want you really want is readonly compressed databases, my suggestion is that you compress your entire database with Zstandard Seekable Format, and use zstd-seekable-format-go with the reader VFS. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! I'll give zstd-seekable-format-go a shot. |
Beta Was this translation helpful? Give feedback.
-
|
Please report back! |
Beta Was this translation helpful? Give feedback.
-
Just for my understanding, this solution would only be suitable for reading an already compressed database, but wouldn't support to read and write to the database from Go as usual, with compression happening transparently in the background? Did you give it a try nevertheless, @franchb? Would love to have a compression VFS implementation! Apart from that, thanks a lot for your efforts, @ncruces! |
Beta Was this translation helpful? Give feedback.
-
|
Yes, The architecture for the official ZIPVFS depends on non public APIs. As detailed on that page, the SQLite pager depends on writing fixed size power-of-two pages to power-of-two offsets. I was able to circumvent the issue for my encryption extensions, by using encryption schemes that do not change the size of pages/sectors. Changing the size of pages is kinda the point of compression. More importantly, a page can be really small when first written to (bunch of zeros for an half-filled page) then updated to be incompressible. We can't just make something in the middle of a file grow. The way the ZIPVFS works is kinda like there's a database “in memory” that saves compressed pages as BLOBs to a second database on disk, just more optimized. But we have no special APIs (other than SQL) to make this fast. I wonder, at that point why not use some KV store. Anyway, this is more work than I can take in. But hopefully making this a discussion gives others ideas to work on. |
Beta Was this translation helpful? Give feedback.
Maybe, but it really depends on what you mean.
Any extension you can compile and statically link into your own Wasm, is probably fair game.
I can't give specific instructions for sqlite-zstd, but you can check how
sqlite3.wasmis built, and how various extensions are linked and initialized. Then it's just a matter of using your own Wasm.If, however, want you really want is readonly compressed databases, my suggestion is that you compress your entire database with Zstandard Seekable Format, and use zstd-seekable-format-go with the reader VFS.