fix(chokidar, dictionaries-entry): fix memory leak in intlayer watch#400
Conversation
getUnmergedDictionaries() and getDictionaries() used clearModuleCache() + require() to reload dictionary JSON on every file change. V8 retains internal module bookkeeping after cache eviction, leaking ~1MB/call. Replace require() with readFileSync + JSON.parse for dictionary JSON, and vm.runInThisContext for CJS entry point execution. Improvement from ~972KB/call to ~1KB/call
aymericzip
left a comment
There was a problem hiding this comment.
merging to test
Thanks @modanub, good catch
|
restoring dictionary-entry file for now because of build optimization error on vite apps bun run build
$ vite build
Failed to transform with Babel plugin: Error: Calling `require` for "../dictionary/app.json" in an environment that doesn't expose the `require` function. See https://rolldown.rs/in-depth/bundling-cjs#require-external-modules for more details.
at file:///Users/aymericpineau/Documents/intlayer_/packages/@intlayer/dictionaries-entry/dist/esm/_virtual/_rolldown/runtime.mjs:4:8
at /Users/aymericpineau/Documents/intlayer_/examples/vite-preact-app/.intlayer/main/dictionaries.cjs:2:31
at getDictionaries (file:///Users/aymericpineau/Documents/intlayer_/packages/@intlayer/dictionaries-entry/dist/esm/index.mjs:20:128)
at intlayerOptimize (file:///Users/aymericpineau/Documents/intlayer_/packages/vite-intlayer/dist/esm/intlayerOptimizePlugin.mjs:27:24)
at intlayerPlugin (file:///Users/aymericpineau/Documents/intlayer_/packages/vite-intlayer/dist/esm/intlayerPlugin.mjs:65:15)
at file:///Users/aymericpineau/Documents/intlayer_/examples/vite-preact-app/node_modules/.vite-temp/vite.config.ts.timestamp-1774155494704-17e03ab0b6724.mjs:10:2
at ModuleJob.run (node:internal/modules/esm/module_job:263:25)
at async ModuleLoader.import (node:internal/modules/esm/loader:540:24)
at async loadConfigFromBundledFile (file:///Users/aymericpineau/Documents/intlayer_/node_modules/.bun/vite@8.0.1+5044978246f43f13/node_modules/vite/dist/node/chunks/node.js:34544:12)
at async bundleAndLoadConfigFile (file:///Users/aymericpineau/Documents/intlayer_/node_modules/.bun/vite@8.0.1+5044978246f43f13/node_modules/vite/dist/node/chunks/node.js:34405:17) |
|
Hey, sorry about the Vite build breakage — the |
|
nice, merged FYI, the cache clean logic was implemented for intlayer-editor & live sync feature, once a change was made on the json, we had to restart the editor to get the updated jsons, cleaning cache was fixing that On vite / next / expo, that entry file is aliased for web compatibility I will test it tomorrow |
|
hey @modanub Works well, thanks |
Summary
intlayer watchleaks ~1MB per file change event, eventually hitting OOM.Root cause:
getUnmergedDictionaries()andgetDictionaries()usedclearModuleCache()+require()to reload dictionary JSON on every change. V8 retains internal module bookkeeping even afterrequire.cacheeviction, so each clear+reload cycle leaked the full dictionary payload (~850KB for 23 JSON files).Fix:
JSON.parse(fs.readFileSync(...))instead ofrequire()for JSON importsdictionaries-entryandunmerged-dictionaries-entryexecute the entry point viareadFileSync+vm.runInThisContextinstead ofclearModuleCache+require()This bypasses
require.cacheentirely.Improvement from ~972KB/call to ~1KB/call. Memory stays stable at ~200-300MB during extended watch sessions.
Test plan
intlayer buildgenerates dictionaries correctlyintlayer watchprocesses file changes normallybun run devworks end-to-endgetUnmergedDictionaries()— ~1KB/call (was ~972KB)