[testing] add remote build tests

[Meulengracht]

Now that the Phase 1 system test harness exists and validates the local runtime workflow, extend the system test suite to cover the remote build path.

Today the distributed build path remains unverified in CI. We want release confidence that the remote builder components can work together to build a real example recipe and produce output that is usable by the rest of the system.

Goal

Add a Phase 2 system test workflow that validates remote building of examples/recipes/hello-world using:

• daemons/waiterd
• daemons/cookd
• tools/bake

Background

Phase 1 established:

• a reusable tests/system harness
• daemon/process orchestration
• logging and cleanup
• a runtime end-to-end test using hello-world

Phase 2 should build on that foundation rather than introducing a separate testing approach.

In scope

• Add reusable harness extensions as needed for remote daemon orchestration
• Add a remote build workflow test for hello-world
• Integrate the remote build workflow into CI appropriately
• Document how to run and debug the remote build test locally

Out of scope

• dummy/fake store implementation
• store tool testing
• multi-worker scaling scenarios
• fault injection / resilience testing
• large matrix coverage

Required test

remote-build-hello.sh

Expected workflow:

1. create isolated temp environment
2. start waiterd
3. start one cookd
4. invoke tools/bake in remote build mode against the remote build setup
5. build examples/recipes/hello-world
6. assert build success
7. assert that the expected remote build artifact is produced

Preferred extension if practical

If feasible without making the test too brittle, add a stronger end-to-end flow:

remote-build-and-run-hello.sh

Expected workflow:

1. perform remote build of hello-world
2. start required runtime daemons
3. install the produced package
4. run the installed application
5. assert:
   • exit code is 0
   • stdout contains expected hello-world output

This is optional if it makes the initial Phase 2 too large; in that case it can be a Phase 2b issue.

Test design requirements

• Test through public daemon/CLI boundaries
• Use isolated temp roots
• Preserve daemon and command logs
• Use readiness checks rather than fixed sleeps
• Reuse the Phase 1 harness and conventions

Repo-specific questions to resolve

The implementation should determine and document:

1. how waiterd is configured for tests
2. how cookd connects/registers for tests
3. how tools/bake is pointed at the remote build setup
4. how to reliably detect that the build actually used the remote path
5. where the resulting artifact is emitted and how to discover it reliably

CI integration

• Extend the existing system-test execution path
• If needed, remote build coverage can initially run in a separate CI job
• If runtime or stability is a concern, it can start as non-blocking, but the goal is eventual normal CI coverage

Acceptance criteria

• A remote build system test exists
• It uses the existing system-test harness
• It starts the required remote-build daemons
• It builds examples/recipes/hello-world through the remote path
• It verifies successful output artifact creation
• CI runs the remote build test
• Logs are preserved on failure
• Documentation explains how to run and debug it locally