feat: rebuild only necessary canisters

[vporton]

Description

DFX doesn't compile canisters for which all dependencies are elder than the .wasm file. This results in big compilation speedups.

The PR also adds tracing and comments intended for future improvement of compilation speed.

This PR is a WIP. There is potential for further speedup and there is missing logging. More testing is necessary.

How Has This Been Tested?

I run it before the .wasm file has been created and after it with a significant speedup. Also dfx build -vv correctly does not output any moc compilation logs, when invoking dfx build for the second time.

Later I did bats make_like.bash, add_dependency.bash, broken_canister_dep.bash, and dotenv.bash automated testing.

Checklist:

• The title of this PR complies with Conventional Commits.
• I have edited the CHANGELOG accordingly.
• I have made corresponding changes to the documentation.

[vporton]

dfx deploy of my current DFX version running in https://github.com/vporton/zondirectory2 repo causes an infinite loop (but dfx build --all doesn't).

[vporton]

The last commit (79faa772dd3b8a7683ed9670a1447ecd0b2893ba) seems working!

Also note that unlike the original (master) code, my code does recompile dependent canisters (canister:* dependencies) when a dependency changed.

The code needs refactoring and more serious testing.

[vporton]

Testing shows a bug: When specifying a dependency to build: dfx build dependency, the dependent canister may recompile, too.

[vporton]

The latest version commit 030f9af902515edd70d60acde5d2cb547c200b40 works as expected on simple (manual) tests (with more tests than the past time).

You may try to use it for your home projects, but don't try it for mission-critical tasks or if you fear to be lost in wrong recompilations, because errors are still possible.

[vporton]

Fixed crash on dfx build --all (creates a new bug, about compiling non-Motoko canister: dependencies).

[vporton]

This MR has been "battle-tested" a little. It worked as expected in developing https://github.com/vporton/zondirectory2 and in testing of https://github.com/vporton/joining-proxy-rust without any visible errors.

[vporton]

There has been found yet a bug:

Compiling vporton/zondirectory2@b145a4c it was output:

...

The command '"moc-wrapper" "src/backend/personhood.mo" "-o" "/home/porton/Projects/zondirectory/.dfx/local/canisters/personhood/personhood.wasm" "-c" "--debug" "--idl" "--stable-types" "--public-metadata" "candid:service" "--public-metadata" "candid:args" "--actor-idl" "/home/porton/Projects/zondirectory/.dfx/local/canisters/idl/" "--actor-alias" "CanDBIndex" "bkyz2-fmaaa-aaaaa-qaaaq-cai" "--actor-alias" "ic_eth" "avqkn-guaaa-aaaaa-qaaea-cai" "--package" "base" ".mops/base@0.11.1/src" "--package" "candb" ".mops/_github/candb#no-blob/src" "--package" "nacdb" ".mops/nacdb@0.13.0/src" "--package" "stable-rbtree" ".mops/stable-rbtree@1.0.0/src" "--package" "stable-buffer" ".mops/_github/stable-buffer#v0.2.0/src" "--package" "prng" ".mops/prng@0.0.3/src" "--package" "stableheapbtreemap" ".mops/stableheapbtreemap@1.3.0/src" "--package" "sha2" ".mops/sha2@0.0.4/src" "--package" "stablebuffer" ".mops/_github/stablebuffer#v0.2.0/src" "--package" "StableBuffer" ".mops/_github/StableBuffer#v0.2.0/src" "--package" "icrc1" ".mops/icrc1@0.0.2/src" "--package" "array" ".mops/_github/array#main/src" "--package" "StableTrieMap" ".mops/_github/StableTrieMap#main/src" "--package" "itertools" ".mops/itertools@0.1.2/src" "--package" "base-0.7.3" ".mops/_github/base-0.7.3#aafcdee0c8328087aeed506e64aa2ff4ed329b47/src" "--package" "xtendedNumbers" ".mops/_github/xtendedNumbers#v1.1.0/src" "--package" "motoko-lib" ".mops/_github/motoko-lib#0.7/src" "--package" "map7" ".mops/_github/map7#v7.0.0/src" "--package" "sha" ".mops/_github/sha#master/src" "--package" "matchers" ".mops/_github/matchers#v1.3.0/src" "--package" "map" ".mops/map@9.0.1/src" "--package" "btree" ".mops/_github/btree#v0.3.3/src" "--package" "CanDBMulti" ".mops/_github/CanDBMulti#v0.10.0@86d00711216349ba4879370d6313d44a26d16f61/src" "--package" "xtended-numbers" ".mops/xtended-numbers@0.2.1/src" "--package" "json.mo" ".mops/json.mo@0.1.2/src" "--package" "parser-combinators" ".mops/_github/parser-combinators#v0.1.2/src" "--package" "nacdb-reorder" ".mops/nacdb-reorder@3.0.0/src" "--package" "passport-client" ".mops/passport-client@0.4.6/src" "--package" "sequence" ".mops/_github/sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "--package" "stable-hash-map" ".mops/_github/stable-hash-map#v0.2.1@eb7edf4127233f2b1f3732ede7e2c55827ac6886/src" "--package" "encoding" ".mops/_github/encoding#v0.3.2@8e0fe1d8f5c2d284e77d719703c42e0e271839b1/src" "--package" "candy" ".mops/_github/candy#v0.3.0@907a4e7363aac6c6a4e114ebc73e3d3f21e138af/src" "--package" "stablebuffer_1_3_0" ".mops/_github/stablebuffer_1_3_0#v1.3.0@acdde6bb5b939227997cebdbb8919d2e6da8691c/src" "--package" "map9" ".mops/_github/map9#v9.0.1@10b68f6ea8df5e72dfa4c07a50c8bb60a916c233/src" "--package" "candb-multi" ".mops/candb-multi@0.11.0/src" "--package" "test" ".mops/test@1.2.0/src" "--package" "date.mo" ".mops/_github/date.mo#master@cfe51ad7d8a576d53e701136d4234233b94970eb/src" "--package" "testing" ".mops/_github/testing#main/src" "--package" "fmt" ".mops/_github/fmt#v0.1.0/src" "--package" "motoko-sequence" ".mops/_github/motoko-sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "-fshared-code" "--actor-idl" ".dfx/local/lsp"' failed with exit status 'exit status: 1'. Stdout:

Stderr: src/backend/personhood.mo:5.1-5.40: import error [M0009], file ".dfx/local/lsp/bkyz2-fmaaa-aaaaa-qaaaq-cai.did" does not exist

src/backend/personhood.mo:5.1-5.40 is about CanDBIndex (bkyz2-fmaaa-aaaaa-qaaaq-cai).

[vporton]

I've removed "that's weird" from my comment, because it has been written by me in error.

[vporton]

Note that the above error was caused by dfx start --background --clean && dfx deploy frontend.

[vporton]

There has been found yet a bug:

I run dfx deploy frontend 15 times and wasn't able to reprise the above bug. Probably, I just installed a wrong version of dfx and there is no bug in this version. I will keep using it in practice.

[vporton]

I've fixed a serious bug.

[vporton]

Found bugs, while attempting deploying a real app to ic network:

1. ic_eth (Rust) canister was decided to be compiled after a Motoko canister dependent on it.
2. Trying to deploy a real app to ic network:

Caused by: The command '"moc-wrapper" "/home/porton/Projects/zondirectory/src/backend/personhood.mo" "-o" "/home/porton/Projects/zondirectory/.dfx/ic/canisters/personhood/personhood.wasm" "-c" "--debug" "--idl" "--stable-types" "--public-metadata" "candid:service" "--public-metadata" "candid:args" "--actor-idl" "/home/porton/Projects/zondirectory/.dfx/ic/canisters/idl/" "--actor-alias" "CanDBIndex" "n674v-zqaaa-aaaap-qbpza-cai" "--actor-alias" "ic_eth" "jrzee-liaaa-aaaap-qhnoa-cai" "--package" "base" ".mops/base@0.11.1/src" "--package" "candb" ".mops/_github/candb#no-blob/src" "--package" "nacdb" ".mops/nacdb@0.13.0/src" "--package" "stable-rbtree" ".mops/stable-rbtree@1.0.0/src" "--package" "stable-buffer" ".mops/_github/stable-buffer#v0.2.0/src" "--package" "prng" ".mops/prng@0.0.3/src" "--package" "stableheapbtreemap" ".mops/stableheapbtreemap@1.3.0/src" "--package" "sha2" ".mops/sha2@0.0.4/src" "--package" "stablebuffer" ".mops/_github/stablebuffer#v0.2.0/src" "--package" "StableBuffer" ".mops/_github/StableBuffer#v0.2.0/src" "--package" "icrc1" ".mops/icrc1@0.0.2/src" "--package" "array" ".mops/_github/array#main/src" "--package" "StableTrieMap" ".mops/_github/StableTrieMap#main/src" "--package" "itertools" ".mops/itertools@0.1.2/src" "--package" "base-0.7.3" ".mops/_github/base-0.7.3#aafcdee0c8328087aeed506e64aa2ff4ed329b47/src" "--package" "xtendedNumbers" ".mops/_github/xtendedNumbers#v1.1.0/src" "--package" "motoko-lib" ".mops/_github/motoko-lib#0.7/src" "--package" "map7" ".mops/_github/map7#v7.0.0/src" "--package" "sha" ".mops/_github/sha#master/src" "--package" "matchers" ".mops/_github/matchers#v1.3.0/src" "--package" "map" ".mops/map@9.0.1/src" "--package" "btree" ".mops/_github/btree#v0.3.3/src" "--package" "xtended-numbers" ".mops/xtended-numbers@0.2.1/src" "--package" "json.mo" ".mops/_github/json.mo#master@49612cf3865dc24bca7bb0e8b6fcf275d11a8cb2/src" "--package" "parser-combinators" ".mops/_github/parser-combinators#main/src" "--package" "nacdb-reorder" ".mops/nacdb-reorder@3.1.1/src" "--package" "cycles-simple" ".mops/cycles-simple@0.11.0/src" "--package" "stablehashmap" ".mops/stablehashmap@1.0.0/src" "--package" "candb-multi" ".mops/candb-multi@0.11.8/src" "--package" "stable-hash-map" ".mops/_github/stable-hash-map#v0.2.1@eb7edf4127233f2b1f3732ede7e2c55827ac6886/src" "--package" "encoding" ".mops/_github/encoding#v0.3.2@8e0fe1d8f5c2d284e77d719703c42e0e271839b1/src" "--package" "candy" ".mops/_github/candy#v0.3.0@907a4e7363aac6c6a4e114ebc73e3d3f21e138af/src" "--package" "stablebuffer_1_3_0" ".mops/_github/stablebuffer_1_3_0#v1.3.0@acdde6bb5b939227997cebdbb8919d2e6da8691c/src" "--package" "map9" ".mops/_github/map9#v9.0.1@10b68f6ea8df5e72dfa4c07a50c8bb60a916c233/src" "--package" "join-proxy-motoko" ".mops/_github/join-proxy-motoko#master@3582456b29cb81e0e304f7df2a3935da670725f6/src" "--package" "parser-combinators.mo" ".mops/_github/parser-combinators.mo#master@f87c9417ab145017fb1f83e19ec2fc45588dbd41/src" "--package" "passport-client" ".mops/passport-client@0.4.9/src" "--package" "sequence" ".mops/_github/sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "--package" "test" ".mops/test@1.2.0/src" "--package" "date.mo" ".mops/_github/date.mo#master@cfe51ad7d8a576d53e701136d4234233b94970eb/src" "--package" "testing" ".mops/_github/testing#main/src" "--package" "fmt" ".mops/_github/fmt#v0.1.0/src" "--package" "motoko-sequence" ".mops/_github/motoko-sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src"' failed with exit status 'exit status: 1'.
Stdout:

Stderr:
/home/porton/Projects/zondirectory/src/backend/personhood.mo:5.1-5.40: import error [M0009], file "/home/porton/Projects/zondirectory/.dfx/ic/canisters/idl/n674v-zqaaa-aaaap-qbpza-cai.did" does not exist
/home/porton/Projects/zondirectory/src/backend/personhood.mo:6.1-6.32: import error [M0009], file "/home/porton/Projects/zondirectory/.dfx/ic/canisters/idl/jrzee-liaaa-aaaap-qhnoa-cai.did" does not exist
/home/porton/Projects/zondirectory/src/backend/personhood.mo:9.1-9.56: import error [M0009], file "/home/porton/Projects/zondirectory/src/libs/configs/passport.config/lib.mo" does not exist

[raymondk]

closed due for age and inactivity