[new release] dune (18 packages) (3.22.0~alpha1)

[shonfeder]

Fast, portable, and opinionated build system

• Project page: https://github.com/ocaml/dune
• Documentation: https://dune.readthedocs.io/

CHANGES:

Fixed

• Dyn.to_string now uses a smarter way to convert floats. This ensures that
  floats are printed with enough precision to round-trip and are valid OCaml
  lexemes. (feat(dyn): use smarter printing of floats dune#12982, fixes Request to make Dyn.float more precise dune#12980, @Alizter)

• Fix dune install --prefix failing with relative paths outside the workspace
  like ../foo (fix: support relative paths in dune install --prefix dune#12993, fixes dune install doesn't process relative paths correctly dune#12241, @benodiwal)

• Place the default trace file inside the build directory at the
  workspace root, rather than relative to the current directory.
  (Place default trace file in the build directory dune#13735, @vouillon)

• Fixed interpreting relative paths in %{bin:..} and %{bin-available:..}.
  These are now interpreted correctly, relative to the dune file they're in.
  (fix: interpret relative paths in %{bin:..} and %{bin-available:..} dune#13712, fixes Relative paths and %{bin:..} and %{bin-available:..} forms dune#9564, @anmonteiro)

• Delete sandboxes with broken permissions (fix: try harder to delete sandboxes with modified permissions dune#13511, @rgrinberg)

• Fix compiling Menhir parsers that refer to sibling modules within a
  subdirectory of (include_subdirs qualified). (fix(menhir): compile inferred interface in its corresponding module group dune#13118, fixes Sibling modules are unbound in qualified Menhir parsers dune#11119,
  @anmonteiro)

• Fixed the dependency specification of C stubs, which could result in C
  stubs not getting rebuilt when needed (which could in turn lead to
  segmentation faults and other hard-to-track bugs).
  (C stubs: depend on Caml headers dune#13652, fixes [BUG] C stubs dependencies are underspecified dune#13651, @nojb)

• Fix the Dune cache on Windows by correctly handling renames onto read-only
  files. Before this change, the Dune cache would be filled but the stored
  artifacts would not generally be usable by Dune. (fix: allow rename to overwrite read-only destination on Windows when deduplicating Dune cache dune#13713, @Nevor)

• Fix rpc not transferring promotion warnings to the client
  (Fix rpc not transferring promotion warnings to the client dune#12604, fixes dune promote doesn't fail when not available in the promotion database dune#12578, @ElectreAAS)

• Fix issue where dune exec -w was unable to kill running programs on
  rebuild. ( fix: avoid killing process group when not applicable  dune#12360, fixes dune exec --watch broken for long-running processes dune#12323, @Alizter)

• Fix package extraction on systems with tar implementations that don't
  auto-detect compression (e.g., OpenBSD). Dune now passes explicit
  decompression flags (-z for gzip, -j for bzip2) when needed, and provides
  clear error messages for unsupported formats like XZ and LZMA. (fix(pkg): pass explicit -z/-j flags for non-auto-detecting tar dune#13688,
  fixes tar on OpenBSD dune#10123, @Alizter)

• Resolve context and workspace binaries introduced by the respective (env (binaries ..)) stanzas. (fix: account for workspace binaries when generating rules for ./.bin dune#12952, fixes "No rule found" when using "binaries" defined in "env" stanza in dune-workspace dune#6220, @anmonteiro)

• Fix diff promotions originating from sandboxed rules. Previously, they
  would be completely ignored as the sandbox with the promoted file would be
  destroyed if the promotion fired (fix: register promotions from sandboxed actions dune#13520, @rgrinberg)

• Fix failure to digest installed directory targets, allowing them to be used
  as dependencies to other rules. (test: demonstrate failure to digest installed dir target dependency dune#13045, @anmonteiro)

• Fix handling of (select ..) field when used with (include_subdirs ..).
  (select <path> from ..) modules now parse path as a relative path
  starting from the module group root (fix: (select ..) field + (include_subdirs ..) dune#13175, fixes Alternative dependencies do not work with include_subdirs unqualified dune#4383, Can include_subdirs qualified and select work together? dune#12450,
  @anmonteiro)

• Fix dune trying to kill processes that were already reaped due to race
  conditions (Scheduler: fix pid races dune#13245, @rgrinberg)

• Add O_CLOEXEC to all files used for stdin/stdout/stderr (fix: add missing O_CLOEXEC to redirect files dune#13385, @rgrinberg)

• Fix $ dune promote dir/foo when dir does not exist (fix: dune promote should promote into non existent dirs dune#13493, @rgrinberg)

• Fix (select ..) field evaluation when a transitive library has optional
  dependencies (fixes All MirageOS unikernels fail to build: dune build fails to succeed (with select in a monorepo) dune#13299, fix(select-field): eval closure when resolving select dune#13389, @anmonteiro)

• Fix sandboxed builds of library stanzas that set
  (stdlib (modules_before_stdlib ..)) (fix(stdlib): depend on modules_before_stdlib dune#13624, @anmonteiro)

• Dune cache: use of hard links under Windows. (Dune cache: use hard links on Windows dune#13714, @Nevor)

• Fixed non-build caches not following $DUNE_CACHE_ROOT and instead only
  relying on $XDG_CACHE_HOME.
  This means the normal build cache moves:
  $DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db (no changes if that variable was
  unset). Affected users can prevent a full cache invalidation by moving
  previous contents:
  cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db.
  (use DUNE_CACHE_HOME instead of XDG_CACHE_HOME dune#11612, fixes Toolchains cache does not follow DUNE_CACHE_ROOT dune#11584, @ElectreAAS)

• $ dune promotion list writes output to stdout rather than stderr (fix: dune promotion list should write its output to stdout dune#13462)

• Improve handling of empty files in the diff action. These are now correctly
  distinguished from empty files. (fix: diff action missing files dune#13696, @rgrinberg)

• Pass /dev/null to --diff-command instead of non-existent files (fix: diff action missing files dune#13696,
  @rgrinberg)

• Fix failure when multiple rocq.extraction stanzas existing in a directory
  (Fix #8042, multiple rocq.extraction in directory. dune#13531, fixes Having several Coq extraction stanzas leads to multiple rule error from Dune 3.7 dune#8042, @rlepigre-skylabs-ai)

• Print $ dune promotion show output to stdout rather than stderr (fix: dune promotion show should write to stdout dune#13481,
  @rgrinberg)

• Fix deadlock in the memo library in the presence of dependency cycles
  (fix(memo): fill ivar cells for dependency cycle nodes to avoid deadlock dune#13625, @anmonteiro)

• Fix promotions that modify a directory into a file (fix: promoting a file into a directory dune#13516, fixes cram tests: spurious error when transforming a single-file test into a directory test dune#4067,
  @rgrinberg)

• Fix installation of implementations of virtual libraries. This failed when
  the implementation had no private modules, but the virtual library did
  (Uncaught exception when a public library has a virtual module dune#10635, @rgrinberg)

• Respect the (dir ..) field on packages when setting up cram tests (fix: respect the dir stanza for assigning cram tests to packages dune#13581,
  @rgrinberg)

Added

• Add support for generating .cms files using oxcaml and adding .cms or
  .cmt files as compilation dependencies (Add support for OxCaml's .cms files dune#13397, @spiessimon)

• Add trace events for custom actions (feature: trace some custom actions dune#13265, @rgrinberg)

• Allow enabling extensions with (using ..) in dune-workspace files
  (Support for using oxcaml 0.1 in workspace files dune#13395, @spiessimon)

• Add sandbox extraction trace event (feature: sandbox extraction event dune#13544, @rgrinberg)

• Add the initial cwd to the first config event (Dune_trace: add pid and initial_cwd to initial event dune#13026, @rgrinberg)

• Dune dune produces trace events in DUNE_ACTION_TRACE_DIR if this variable
  is set. (feature: dune produces action traces dune#13302, @rgrinberg)

• Add file watching events to the trace file (dune_trace: add file watching events dune#13038, @rgrinberg)

• Introduce the $ dune trace cat subcommand to view the trace file. (Dune trace subcommand dune#13055,
  @rgrinberg)

• Add diagnostic events to the trace. (Add diagnostic events to the trace dune#13041, @rgrinberg)

• Add DUNE_JOBS environment variable for controlling concurrency of Dune from
  environment. The INSIDE_DUNE variable also now no longer controls
  concurrency (don't control concurrency from INSIDE_DUNE  dune#12800, @Alizter)

• Support for Rocq expected output tests (Add expected output tests for Rocq. dune#13632, @rlepigre-skylabs-ai)

• Add rusage information to completed processes in the trace (@rgrinberg,
  dune_trace: add resource usage to process events dune#13241)

• Add process start events to the trace (feature: process start events dune#13261, rgrinberg)

• Generate odoc documentation in markdown using the @doc-markdown alias
  (Markdown for odoc without system dune#12581, @davesnx)

• Add timing information for every command executed by cram (Cram: Add command level timing information dune#13092,
  @rgrinberg)

• Add the workspace root to the config trace event (Add dune root to config event dune#12922, @rgrinberg)

• Introduce the dune-action-trace library. This public library is to be used
  by custom actions to emit trace events while executed as part of a dune
  build. The trace events emitted through this library will be incorporated
  into dune's own trace (feature: introduce a public action tracing library dune#13348, @rgrinberg)

• Add dune-find-dominating to dune.el, a command to find the
  dominating dune file. (Add dune-find-dominating command to dune.el dune#12696, @arvidj)

• Add a --no-recursive flag to $ dune describe workspace (feature: add --no-recursive flag to $ dune describe workspace dune#13590, @rgrinberg)

• Trace events for files written directly by dune (feature: trace file writing events dune#13618, @rgrinberg)

• Allow expansion of special forms like (:include ..) and %{read-lines:..}
  in the modules specification for the ocamllex, ocamlyacc and menhir
  stanzas. (feat(ocamllex): support dynamic (modules ..) evaluation  dune#13105, feat(ocamlyacc): support expanding (modules ..) dune#13135, feat(menhir): allow expanding (modules ..) field dune#13157, @anmonteiro)

• Add a trace event for snapshotting the asndbox (feature: sandbox snapshot event dune#13541, @rgrinberg)

• Add signal send and receive events to the trace (dune_trace: trace all signals dune#13193, @rgrinberg)

• Emit final trace event before exiting. (Dune trace: emit final event dune#13018, @rgrinberg)

• dune runtest can now run individual test executables from (tests) stanzas
  and inline tests from (library (inline_tests)) stanzas by providing their
  source files as arguments. (feat(runtest): dune runtest for (tests) and (inline_tests) dune#13064, fixes It should be possible to run tests from a specific module with runtest dune#870, @Alizter)

• Add a shell field to the cram stanza. This field allows customizing the
  shell to be bash rather than sh (cram: allow customizing the shell dune#13083, @haochenx)

Changed

• Start sandboxing the execution of tests defined with the test and tests
  stanzas (feature: start sandboxing test execution starting from 3.22 dune#13510, fix: sandbox all test stanza variants starting from 3.22 dune#13617, @rgrinberg)

• Disabled cram tests can now be run explicitly with dune runtest disabled.t.
  The enabled_if field now only controls whether a test is included in
  the @runtest alias. (feat(cram): explicitly requested disabled tests now execute dune#13081, @Alizter)

• Process categories in trace events are moved to their own field in args
  (dune trace: move process categories to field in args dune#13024, @rgrinberg)

• Sandbox running ocamllex and ocamlyacc actions. (ocamllex / ocamlyacc: sandbox rules dune#13098, @anmonteiro)

• Sandboxing mdx test actions is now the default starting from 0.5 (feature(mdx): start sandboxing in version 0.5 dune#13504,
  @rgrinberg)

• Start sandboxing Melange rules by default in the (library ..) and
  (melange.emit ..) stanzas (feat(melange): sandbox rules by default dune#13619, @anmonteiro)

• Introduce a promotion trace event and remove the corresponding verbose log
  message. (Add a promotion trace event dune#12949, feature: trace events for promotion registration dune#13444, @rgrinberg)

• Change dune's trace format to emit canonical s-expressions. This improves
  performance and is better aligned with dune's usage of the format
  elsewhere. $ dune trace cat can also emit the trace in --chrome-trace
  for perfetto, or --sexp for regular s-expressions for interactive usage.
  (dune_trace: csexp dune#13059, @rgrinberg)

• Move all logging statements to the trace file. All log statements now contain
  structured payloads (Structured Logging dune#13015, fixes Rationalize Logging/Tracing dune#12904, @rgrinberg)

• Add a target resolution event to replace the equivalent log message (Dune_trace: target resolution event dune#12955,
  @rgrinberg)

[shonfeder]

I have identified the following additional failures when compared against the previous results, using a WIP script and after eliminating obviously irrelevant results:

Breaking API change to dune-rpc:

• chrome-trace:
  • (revdeps/compilers/4.14): ocaml-lsp-server.1.21.0-4.14 (failed: The functor application is ill-typed.)
  • (revdeps/compilers/5.4): ocaml-lsp-server.1.25.0 (failed: This application of the functor Drpc.Client.Make is ill-typed.)
• dune-rpc:
  • (revdeps/compilers/4.14): ocaml-lsp-server.1.21.0-4.14 (failed: The functor application is ill-typed.)
  • (revdeps/compilers/5.4): ocaml-lsp-server.1.25.0 (failed: This application of the functor Drpc.Client.Make is ill-typed.)
• ocamlc-loc:
  • (revdeps/compilers/4.14): ocaml-lsp-server.1.21.0-4.14 (failed: The functor application is ill-typed.)
  • (revdeps/compilers/5.4): ocaml-lsp-server.1.25.0 (failed: This application of the functor Drpc.Client.Make is ill-typed.)

Addressed in #29544

Unrelated build failures

• dune:
  • (revdeps/compilers/4.14): zlib.0.6 (failed: implicit declaration of function 'strnlen'; did you mean 'strlen'? [-Wimplicit-function-declaration])
  • (revdeps/compilers/5.4): zlib.0.6 (failed: implicit declaration of function 'strnlen'; did you mean 'strlen'? [-Wimplicit-function-declaration])

See madroach/ocaml-zlib#8

• dune-configurator:
  • (revdeps/compilers/4.14): xen-gnt-unix.4.0.2 (failed: passing argument 1 of 'failwith_xc' from incompatible pointer type [-Wincompatible-pointer-types])
  • (revdeps/compilers/5.4): xen-evtchn-unix.2.1.0 (failed: assignment to 'xc_interface *' {aka 'struct xc_interface_core *'} from incompatible pointer type 'xc_evtchn *' {aka 'struct xenevtchn_handle *'} [-Wincompatible-pointer-types])

GCC >= 14 issue, as seen in many places, including #28095

Unrelated test failures

• dune-build-info:
  • (revdeps/compilers/5.4): server-reason-react.0.4.1 (failed: The compilation of server-reason-react.0.4.1 failed at "dune build -p server-reason-react -j 255 @install @runtest".)

A test failure due to a trivial discrepency in expect tests (one missing line), likely due to a brittle test logic, and no signs of being due to breakage from dune, AFAICT.

Regression identified

• xdg:
  • (revdeps/compilers/4.14): opam-test.0.3.0 (failed: mccs.1.1+19 failed to build)
  • (revdeps/compilers/5.4): opam-test.0.3.0 (failed: mccs.1.1+19 failed to build)

This is an error with c stubs

#=== ERROR while compiling mccs.1.1+19 ========================================#
# context              2.5.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | file:///home/opam/opam-repository
# path                 ~/.opam/4.14/.opam-switch/build/mccs.1.1+19
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune build -p mccs -j 255
# exit-code            1
# env-file             ~/.opam/log/mccs-7-f93079.env
# output-file          ~/.opam/log/mccs-7-f93079.out
### output ###
# File "src/dune", line 5, characters 63-73:
# 5 |    new_criteria notuptodate_criteria removed_criteria mccscudf mccs_stubs)
#                                                                    ^^^^^^^^^^
# (cd _build/default/src && /usr/bin/gcc -O2 -fno-strict-aliasing -fwrapv -pthread -fPIC -Wall -Wextra -Wno-unused-parameter -x c++ -std=gnu++11 -DUSEGLPK -g -I /home/opam/.opam/4.14/lib/ocaml -I /home/opam/.opam/4.14/lib/cudf -I /home/opam/.opam/4.14/lib/extlib -I glpk -o mccs_stubs.o -c mccs_stubs.cpp)
# In file included from mccs_stubs.cpp:21:
# glpk_solver.h:15:10: fatal error: config.h: No such file or directory
#    15 | #include "config.h"
#       |          ^~~~~~~~~~
# compilation terminated.
# File "src/dune", line 4, characters 30-41:
# 4 |    cudf_reductions cudf_tools glpk_solver lp_solver lexagregate_combiner
#                                   ^^^^^^^^^^^
# (cd _build/default/src && /usr/bin/gcc -O2 -fno-strict-aliasing -fwrapv -pthread -fPIC -Wall -Wextra -Wno-unused-parameter -x c++ -std=gnu++11 -DUSEGLPK -g -I /home/opam/.opam/4.14/lib/ocaml -I /home/opam/.opam/4.14/lib/cudf -I /home/opam/.opam/4.14/lib/extlib -I glpk -o glpk_solver.o -c glpk_solver.cpp)
# In file included from glpk_solver.cpp:16:
# glpk_solver.h:15:10: fatal error: config.h: No such file or directory
#    15 | #include "config.h"
#       |          ^~~~~~~~~~
# compilation terminated.

The build failure for mccs does not occur with dune 21, and it appears to be a regression.

• Reported in regression in dune lang 1.0 semantics of cxx_ fields due to changes in select semantics dune#13802
• Fixed by fix(dir_contents): discover (select ..) targets for non-OCaml files dune#13804

[shonfeder]

Closing now that the identified regression has been fixed and with the needed upper bounds added to ocaml-lsp-server. The next (and last? 🤞 ) RC is incoming.