enable support for javascript-based search filters, closes #257

* filter function typed in TypeScript
* Monaco Editor used as a code editor

Author: vladimiry

README.md

@@ -31,7 +31,8 @@ Some Linux package types are available for installing from the repositories (`Pa
 - :octocat: **Open Source**.
 - :gear: **Reproducible builds**. See details in [#183](https://github.com/vladimiry/ElectronMail/issues/183).
 - :gear: **Cross platform**. The app works on Linux/OSX/Windows platforms. Binary installation packages located [here](https://github.com/vladimiry/ElectronMail/releases).
-- :mag_right: **Full-text search**. Including email body content scanning capability. Enabled with [v2.2.0](https://github.com/vladimiry/ElectronMail/releases/tag/v2.2.0) release. See the respective [issue](https://github.com/vladimiry/ElectronMail/issues/92) for details.
+- :mag_right: **Full-text search**. Including email **body content** scanning capability. Enabled with [v2.2.0](https://github.com/vladimiry/ElectronMail/releases/tag/v2.2.0) release. See the respective [issue](https://github.com/vladimiry/ElectronMail/issues/92) for details.
+- :mag_right: **JavaScript-based/unlimited messages filtering**. Enabled since [v4.11.0](https://github.com/vladimiry/ElectronMail/releases/tag/v4.11.0) release. See the respective [#257](https://github.com/vladimiry/ElectronMail/issues/257) for details. Requires [local store](https://github.com/vladimiry/ElectronMail/wiki/FAQ) feature to be enabled.
 - :package: **Offline access to the email messages** (attachments content not stored locally, but emails body content). The [local store](https://github.com/vladimiry/ElectronMail/wiki/FAQ) feature enables storing your messages in the encrypted `database.bin` file (see [FAQ](https://github.com/vladimiry/ElectronMail/wiki/FAQ) for file purpose details). So the app allows you to view your messages offline, running full-text search against them, exporting them to EML files. etc. Enabled since [v2.0.0](https://github.com/vladimiry/ElectronMail/releases/tag/v2.0.0) release. 
 - :mailbox: **Multi accounts** support including supporting individual [API entry points](https://github.com/vladimiry/ElectronMail/issues/29). For example, you can force the specific email account added in the app connect to the email provider via the [Tor](https://www.torproject.org/) only by selecting the `https://protonirockerxow.onion/` API entry point in the dropdown list and configuring a proxy as described in [this](https://github.com/vladimiry/ElectronMail/issues/113#issuecomment-529130116) message.
 - :unlock: **Automatic login into the app** with a remembered the system keychain remembered master ([keep me signed in](images/keep-me-signed-in.png) feature). Integration with as a system keychain is done with the [keytar](https://github.com/atom/node-keytar) module. By the way, on Linux [KeePassXC](https://github.com/keepassxreboot/keepassxc) implements the [Secret Service](https://specifications.freedesktop.org/secret-service/latest/) interface and so it can be acting as a system keychain (for details, see the "automatic login into the app"-related point in the [FAQ](https://github.com/vladimiry/ElectronMail/wiki/FAQ)). 
@@ -47,7 +48,7 @@ Some Linux package types are available for installing from the repositories (`Pa
 - :bell: **Native notifications** for individual accounts clicking on which focuses the app window and selects respective account in the accounts list.
 - :calendar: **Calendar notifications / alarms** regardless of the open page (mail/calendar/settings/account/drive). The opt-in feature has been enabled since [v4.9.0](https://github.com/vladimiry/ElectronMail/releases/tag/v4.9.0). See [#229](https://github.com/vladimiry/ElectronMail/issues/229) for details.
 - :sunglasses: **Making all email "read"** in a single mouse click. Enabled since [v3.8.0](https://github.com/vladimiry/ElectronMail/releases/tag/v3.8.0). Requires [local store](https://github.com/vladimiry/ElectronMail/wiki/FAQ) feature to be enabled.
-- :sunglasses: **Routing images through proxy**. The opt-in feature has been enabled since [v4.9.0](https://github.com/vladimiry/ElectronMail/releases/tag/v4.9.0). See [#312](https://github.com/vladimiry/ElectronMail/issues/312) for details. Requires [local store](https://github.com/vladimiry/ElectronMail/wiki/FAQ) feature to be enabled.
+- :sunglasses: **Routing images through proxy**. The opt-in feature has been enabled since [v4.9.0](https://github.com/vladimiry/ElectronMail/releases/tag/v4.9.0). See [#312](https://github.com/vladimiry/ElectronMail/issues/312) for details.
 - :sunglasses: **Batch mails removing** bypassing the trash. Enabled since [v4.9.0](https://github.com/vladimiry/ElectronMail/releases/tag/v4.9.0). Requires [local store](https://github.com/vladimiry/ElectronMail/wiki/FAQ) feature to be enabled.
 - :sunglasses: **Batch mails moving between folders**. Enabled since [v4.5.0](https://github.com/vladimiry/ElectronMail/releases/tag/v4.5.0). Requires [local store](https://github.com/vladimiry/ElectronMail/wiki/FAQ) feature to be enabled.
 

package.json

@@ -1,7 +1,7 @@
 {
   "name": "electron-mail",
   "description": "Unofficial ProtonMail Desktop App",
-  "version": "4.10.3",
+  "version": "4.11.0",
   "author": "Vladimir Yakovlev <[email protected]>",
   "license": "MIT",
   "homepage": "https://github.com/vladimiry/ElectronMail",
@@ -67,9 +67,9 @@
     "assets:dev": "npm-run-all assets:copy:dev assets:webclient:dev",
     "assets:copy": "cpx \"./src/assets/dist/**/*\" ./app/assets",
     "assets:copy:dev": "cpx \"./src/assets/dist/**/*\" ./app-dev/assets",
-    "assets:webclient": "yarn assets:webclient:base ./app/webclient",
-    "assets:webclient:dev": "yarn assets:webclient:base ./app-dev/webclient",
-    "assets:webclient:base": "yarn ts-node:shortcut ./scripts/prepare-webclient/protonmail.ts",
+    "assets:webclient": "yarn assets:webclient:base ./app",
+    "assets:webclient:dev": "yarn assets:webclient:base ./app-dev",
+    "assets:webclient:base": "yarn ts-node:shortcut ./scripts/prepare-webclient/index.ts",
     "electron-builder:install-app-deps": "electron-builder install-app-deps --arch=x64",
     "electron-builder:dir": "electron-builder --x64 --dir",
     "electron-builder:dist": "npm exec --package=ts-node -- ts-node --files --require tsconfig-paths/register ./scripts/electron-builder/run-with-default-evn-vars.ts --x64 --publish never",
@@ -101,7 +101,7 @@
     "scripts/download-tray-icon-font": "yarn ts-node:shortcut ./scripts/download-tray-icon-font.ts",
     "scripts/transfer": "yarn ts-node:shortcut ./scripts/transfer/index.ts",
     "ts-node:shortcut": "cross-env TS_NODE_FILES=true ts-node --require tsconfig-paths/register",
-    "webpack:shortcut": "cross-env TS_NODE_FILES=true npm exec --package=webpack-cli --node-options=\"--require tsconfig-paths/register\" -- webpack"
+    "webpack:shortcut": "cross-env TS_NODE_PROJECT=./webpack-configs/tsconfig.json TS_NODE_FILES=true npm exec --package=webpack-cli --node-options=\"--require tsconfig-paths/register\" -- webpack"
   },
   "ava": {
     "extensions": [
@@ -145,6 +145,7 @@
     "proxy-agent": "4.0.1",
     "pure-uuid": "1.6.2",
     "pureimage": "0.2.7",
+    "quickjs-emscripten": "0.11.0",
     "reflect-metadata": "0.1.13",
     "remeda": "0.0.27",
     "rxjs": "6.6.6",
@@ -212,6 +213,7 @@
     "cpx2": "3.0.0",
     "cross-env": "7.0.3",
     "css-loader": "5.1.2",
+    "dts-generator": "3.0.0",
     "electron": "12.0.1",
     "electron-builder": "22.10.5",
     "escape-string-regexp": "4.0.0",
@@ -227,6 +229,7 @@
     "immer": "8.0.1",
     "import-sort-cli": "6.0.0",
     "import-sort-parser-typescript": "6.0.0",
+    "imports-loader": "2.0.0",
     "jasmine": "3.6.4",
     "jsdom": "16.5.1",
     "karma": "6.2.0",
@@ -235,6 +238,7 @@
     "karma-webpack": "5.0.0",
     "lint-staged": "10.5.4",
     "mini-css-extract-plugin": "1.3.9",
+    "monaco-editor": "0.23.0",
     "ndx": "1.0.2",
     "ndx-query": "1.0.1",
     "ngx-bootstrap": "6.2.0",

scripts/lib.ts

@@ -223,3 +223,16 @@ export const catchTopLeventAsync = (asyncFn: () => Promise<unknown>): void => {
         process.exit(1);
     });
 };
+
+export const applyPatch = async ({patchFile, cwd}: { patchFile: string; cwd: string }): Promise<void> => {
+    await execShell([
+        "git",
+        [
+            "apply",
+            "--ignore-whitespace",
+            "--reject",
+            patchFile,
+        ],
+        {cwd},
+    ]);
+};

scripts/prepare-webclient/index.ts

@@ -0,0 +1,22 @@
+import path from "path";
+import pathIsInside from "path-is-inside";
+
+import {CWD_ABSOLUTE_DIR} from "scripts/const";
+import {buildProtonClients} from "scripts/prepare-webclient/protonmail";
+import {catchTopLeventAsync} from "scripts/lib";
+import {resolveProtonMetadata} from "scripts/prepare-webclient/monaco-editor-dts";
+
+const [, , appDestDir] = process.argv;
+
+if (!appDestDir) {
+    throw new Error("Empty base destination directory argument");
+}
+
+if (!pathIsInside(path.resolve(CWD_ABSOLUTE_DIR, appDestDir), CWD_ABSOLUTE_DIR)) {
+    throw new Error(`Invalid base destination directory argument value: ${appDestDir}`);
+}
+
+catchTopLeventAsync(async () => {
+    await buildProtonClients({destDir: path.join(appDestDir, "./webclient")});
+    await resolveProtonMetadata({destDir: path.join(appDestDir)});
+});

scripts/prepare-webclient/monaco-editor-dts.ts

@@ -0,0 +1,81 @@
+import fs from "fs";
+import fsExtra from "fs-extra";
+import path from "path";
+
+import {CONSOLE_LOG} from "scripts/lib";
+import {IpcMainServiceScan} from "src/shared/api/main";
+import {PROTON_MONACO_EDITOR_DTS_ASSETS_LOCATION} from "src/shared/constants";
+import {PROTON_SHARED_MESSAGE_INTERFACE} from "src/shared/proton-apps-constants";
+
+// TODO "require/var-requires"-based import
+export const dtsGenerator: { // eslint-disable-line @typescript-eslint/no-unsafe-assignment
+    default: (options: { baseDir: string, files: string[], out: string }) => Promise<string>
+} = require("dts-generator"); // eslint-disable-line @typescript-eslint/no-var-requires
+
+export const resolveProtonMetadata = async (
+    {destDir}: { destDir: string },
+): Promise<IpcMainServiceScan["ApiImplReturns"]["staticInit"]["monacoEditorExtraLibArgs"]> => {
+    const options = {
+        system: {
+            base: "./node_modules/typescript/lib",
+            in: "./node_modules/typescript/lib/lib.esnext.d.ts",
+            out: path.join(destDir, PROTON_MONACO_EDITOR_DTS_ASSETS_LOCATION.system),
+        },
+        protonMessage: {
+            base: "./output/git/proton-mail/node_modules/proton-shared",
+            in: path.join("./output/git/proton-mail/node_modules/proton-shared", PROTON_SHARED_MESSAGE_INTERFACE.projectRelativeFile),
+            out: path.join(destDir, PROTON_MONACO_EDITOR_DTS_ASSETS_LOCATION.protonMessage),
+        },
+    } as const;
+
+    // TODO replace "dts-generator" dependency with something capable to combine "./node_modules/typescript/lib/lib.esnext.d.ts"
+    for (const key of [/* "system", */ "protonMessage"] as const) {
+        const sourceFile = options[key].in;
+        if (!fsExtra.pathExistsSync(sourceFile)) {
+            throw new Error(`The source "${sourceFile}" file doesn't exits.`);
+        }
+        await dtsGenerator.default({baseDir: options[key].base, files: [sourceFile], out: options[key].out});
+        CONSOLE_LOG(`Merged "${options[key].in}" to "${options[key].out}"`);
+    }
+
+    // TODO drop custom "./node_modules/typescript/lib/lib.esnext.d.ts" combining when "dts-generator" gets replaced
+    {
+        const referenceTagRe = /\/\/\/[\s\t]+<reference[\s\t]+lib=["']+(.*)["']+[\s\t]+\/>/;
+        const mergedFiles: string[] = [];
+        const extractContent = (file: string): string => {
+            const lines = fs.readFileSync(file).toString().split("\n");
+            const resultLines: string[] = [`// === file: ${file}`];
+            for (const line of lines) {
+                const match = referenceTagRe.exec(line);
+                const libName = match && match[1];
+                if (libName) {
+                    resultLines.push(
+                        extractContent(path.join(path.dirname(file), `lib.${libName}.d.ts`)),
+                    );
+                } else {
+                    resultLines.push(line);
+                }
+            }
+            mergedFiles.push(file);
+            return resultLines.join("\n");
+        };
+        fsExtra.ensureDirSync(path.dirname(options.system.out));
+        fs.writeFileSync(options.system.out, extractContent(options.system.in));
+        CONSOLE_LOG(`Merged to "${options.system.out}" files:`, mergedFiles);
+    }
+
+    return {
+        system: [
+            fs.readFileSync(options.system.out).toString(),
+            `in-memory:${options.system.in}`,
+        ],
+        protonMessage: [
+            (
+                fs.readFileSync(options.protonMessage.out).toString()
+                +
+                `declare const mail: Omit<import("lib/interfaces/mail/Message").Message, "Body"> & {Body: string};`
+            ),
+            `in-memory:${PROTON_SHARED_MESSAGE_INTERFACE.url}`,
+        ],
+    };
+};

scripts/prepare-webclient/lib.ts renamed to scripts/prepare-webclient/protonmail-lib.ts

@@ -1,10 +1,9 @@
 import fs from "fs";
 import fsExtra from "fs-extra";
 import path from "path";
-import pathIsInside from "path-is-inside";
 
 import {CONSOLE_LOG, execShell, resolveGitCommitInfo, resolveGitOutputBackupDir} from "scripts/lib";
-import {CWD_ABSOLUTE_DIR, GIT_CLONE_ABSOLUTE_DIR} from "scripts/const";
+import {GIT_CLONE_ABSOLUTE_DIR} from "scripts/const";
 import {PROVIDER_REPO_MAP} from "src/shared/proton-apps-constants";
 import {RUNTIME_ENV_CI_PROTON_CLIENTS_ONLY, WEB_CLIENTS_BLANK_HTML_FILE_NAME} from "src/shared/constants";
 
@@ -26,16 +25,6 @@ const reposOnlyFilter: DeepReadonly<{ value: Array<keyof typeof PROVIDER_REPO_MA
     return {value: result, envVariableName};
 })();
 
-const [, , BASE_DEST_DIR] = process.argv;
-
-if (!BASE_DEST_DIR) {
-    throw new Error("Empty base destination directory argument");
-}
-
-if (!pathIsInside(path.resolve(CWD_ABSOLUTE_DIR, BASE_DEST_DIR), CWD_ABSOLUTE_DIR)) {
-    throw new Error(`Invalid base destination directory argument value: ${BASE_DEST_DIR}`);
-}
-
 export interface FolderAsDomainEntry<T extends any = any> { // eslint-disable-line @typescript-eslint/no-explicit-any
     folderNameAsDomain: string;
     options: T;
@@ -75,18 +64,20 @@ export async function executeBuildFlow<T extends FolderAsDomainEntry[], O = Unpa
         repoType,
         folderAsDomainEntries,
         repoRelativeDistDir = PROVIDER_REPO_MAP[repoType].repoRelativeDistDir,
+        destDir,
         destSubFolder,
         flow,
     }: {
-        repoType: keyof typeof PROVIDER_REPO_MAP;
-        folderAsDomainEntries: T;
-        repoRelativeDistDir?: string;
-        destSubFolder: string;
+        repoType: keyof typeof PROVIDER_REPO_MAP
+        folderAsDomainEntries: T
+        repoRelativeDistDir?: string
+        destDir: string
+        destSubFolder: string
         flow: {
-            postClone?: Flow<O>;
-            install?: Flow<O>;
-            build: Flow<O>;
-        };
+            postClone?: Flow<O>
+            install?: Flow<O>
+            build: Flow<O>
+        }
     },
 ): Promise<void> {
     if (
@@ -101,7 +92,7 @@ export async function executeBuildFlow<T extends FolderAsDomainEntry[], O = Unpa
     const repoDir = path.join(GIT_CLONE_ABSOLUTE_DIR, repoType);
 
     for (const folderAsDomainEntry of folderAsDomainEntries) {
-        const targetDistDir = path.resolve(BASE_DEST_DIR as string, folderAsDomainEntry.folderNameAsDomain, destSubFolder);
+        const targetDistDir = path.resolve(destDir, folderAsDomainEntry.folderNameAsDomain, destSubFolder);
 
         CONSOLE_LOG(
             `Prepare web client build [${repoType}]:`,
@@ -157,19 +148,29 @@ export async function executeBuildFlow<T extends FolderAsDomainEntry[], O = Unpa
 
         const repoDistBackupDir = resolveGitOutputBackupDir({repoType, suffix: `dist-${folderAsDomainEntry.folderNameAsDomain}`});
 
-        if (fsExtra.pathExistsSync(repoDistBackupDir)) { // taking dist from the backup
-            const src = repoDistBackupDir;
-            const dest = repoDistDir;
-            CONSOLE_LOG(`Copying backup ${src} to ${dest}`);
-            await fsExtra.copy(src, dest);
-        } else { // executing the build
+        const installModules = async (): Promise<void> => {
             if (fsExtra.pathExistsSync(path.resolve(repoDir, "node_modules"))) {
                 CONSOLE_LOG("Skip dependencies installing");
             } else if (flow.install) {
                 await flow.install(flowOptions);
             } else {
                 await execShell(["npm", ["ci"], {cwd: repoDir}]);
             }
+        };
+
+        if (fsExtra.pathExistsSync(repoDistBackupDir)) { // taking dist from the backup
+            const src = repoDistBackupDir;
+            const dest = repoDistDir;
+
+            CONSOLE_LOG(`Copying backup ${src} to ${dest}`);
+            await fsExtra.copy(src, dest);
+
+            if (repoType === "proton-mail") {
+                // installing modules since required by "./scripts/prepare-webclient/monaco-editor-dts.ts"
+                await installModules();
+            }
+        } else { // executing the build
+            await installModules();
 
             if (shouldFailOnBuild) {
                 throw new Error(`Halting since "${shouldFailOnBuildEnvVarName}" env var has been enabled`);

scripts/prepare-webclient/protonmail.ts

@@ -2,10 +2,10 @@ import fsExtra from "fs-extra";
 import path from "path";
 
 import {BINARY_NAME} from "src/shared/constants";
-import {catchTopLeventAsync, execShell} from "scripts/lib";
 import {CWD_ABSOLUTE_DIR} from "scripts/const";
-import {executeBuildFlow, FolderAsDomainEntry, printAndWriteFile} from "./lib";
+import {FolderAsDomainEntry, executeBuildFlow, printAndWriteFile} from "./protonmail-lib";
 import {PROVIDER_REPO_MAP, PROVIDER_REPO_NAMES} from "src/shared/proton-apps-constants";
+import {applyPatch, execShell} from "scripts/lib";
 
 const folderAsDomainEntries: Array<FolderAsDomainEntry<{
     configApiParam:
@@ -178,24 +178,12 @@ function resolveWebpackConfigPatchingCode(
     return result;
 }
 
-const applyPatch = async ({patchFile, cwd}: { patchFile: string; cwd: string }): Promise<void> => {
-    await execShell([
-        "git",
-        [
-            "apply",
-            "--ignore-whitespace",
-            "--reject",
-            patchFile,
-        ],
-        {cwd},
-    ]);
-};
-
-catchTopLeventAsync(async () => {
+export const buildProtonClients = async ({destDir}: { destDir: string }): Promise<void> => {
     for (const repoType of PROVIDER_REPO_NAMES) {
         await executeBuildFlow({
             repoType,
             folderAsDomainEntries,
+            destDir,
             destSubFolder: PROVIDER_REPO_MAP[repoType].baseDirName,
             flow: {
                 async install({repoDir}) {
@@ -277,4 +265,4 @@ catchTopLeventAsync(async () => {
             },
         });
     }
-});
+};