Table of Contents

1. Overview
2. Status
3. Quick Start
   1. Installation
   2. Basic Usage
4. Test Types
   1. Simple Tests
   2. Matrix Tests
      1. Matrix Options
   3. Coordinated Tests
      1. Coordinated Options
5. Test Body Bindings
   1. All Test Types
   2. Matrix Tests
   3. Coordinated Tests
6. Harness (Setup/Teardown)
7. CLI Reference
8. Filtering
9. Verbosity Levels
10. Memory Tracking
11. Architecture
    1. Subprocess Isolation
    2. Wire Protocol
    3. Directory Structure
12. BEJ Symlink Configuration
13. Examples
14. Design Principles
    1. No Global State
    2. Macro Hygiene
    3. Matrix Value Immutability
15. Self-Testing
16. Roadmap & Contributing
    1. Areas Seeking Help
       1. Core Framework
       2. Tooling
    2. How to Contribute
17. Acknowledgments
18. License

Overview

janet-assay is an ambitious testing framework for Janet that supports sophisticated testing patterns beyond simple unit tests:

• Matrix Testing: Generate test combinations from parameter matrices
• Coordinated Tests: Server/client patterns with emit/await signaling
• Subprocess Isolation: Each suite runs in its own subprocess
• Parallel Execution: Run tests across fibers, threads, or subprocesses
• Flexible Output: Configurable verbosity with display groups
• Production Ready: BEJ symlink config, JSON output, filtering

Status

Experimental - The framework is functional and used in production by jsec (a Janet security/cryptography project), but internals may change.

What is stable:

• Test definition API (def-suite, def-test, :type, :matrix, :coordinated)
• Test body semantics (harness, skip/fail cases, timeout, etc.)
• Basic runner invocation patterns

What may change:

• Runner CLI arguments and output formatting
• Internal wire protocol and worker architecture
• JSON output schema
• Some advanced filtering syntax details

We recommend pinning to a specific commit if stability is critical.

Quick Start

Installation

janet-assay is designed to be included as a sub-project within your Janet project.

Basic Usage

# suites/unit/suite-example.janet
(import assay :prefix "")

(def-suite :name "Example Tests"

  (def-test "simple test"
    (assert (= 4 (+ 2 2))))

  (def-test "matrix test"
    :type :matrix
    :matrix {:a [1 2] :b [3 4]}
    (assert (< a b))))

# test/runner.janet
(import ../assay :prefix "")

(def-runner
  :name "My Test Runner"
  :suites-dir "../suites"
  :env-prefix "MYTEST")

Run tests:

janet test/runner.janet
janet test/runner.janet -v      # verbosity level 1
janet test/runner.janet -vvv    # verbosity level 3
janet test/runner.janet --help  # show all options

Test Types

Simple Tests

Basic tests with assertions:

(def-test "addition"
  (assert (= 4 (+ 2 2))))

(def-test "expected to fail"
  :expected-fail "this test intentionally errors"
  (error "expected error"))

(def-test "skipped test"
  :skip-reason "not implemented"
  (assert false))

Matrix Tests

Generate combinations from all matrix values:

(def-test "connection test"
  :type :matrix
  :matrix {:protocol [:tcp :unix]
           :ssl [:on :off]}
  # Generates 4 combos: tcp/on, tcp/off, unix/on, unix/off
  # Variables 'protocol' and 'ssl' are bound in test body
  (test-connection protocol ssl))

Matrix Options

Option	Description
:skip-cases	Combos to skip: [{:key val} "reason"]
:fail-cases	Combos expected to fail
:ensured-cases	Combos that always run (even when sampling)
:dedup-groups	Groups where value swaps are duplicates

(def-test "with options"
  :type :matrix
  :matrix {:x [1 2 3] :y [1 2 3]}
  :skip-cases [{:x 1 :y 1} "same value"]
  :fail-cases [{:x 3 :y 1} "known issue"]
  :ensured-cases [{:x 1 :y 2}]
  :dedup-groups [[:x :y]]  # x=1,y=2 same as x=2,y=1
  body...)

Coordinated Tests

Multiple participants with emit/await signaling:

(def-test "server-client"
  :type :coordinated

  (def-test "server"
    (emit :ready "listening")
    (serve-forever))

  (def-test "client"
    (await :server :ready)
    (connect-and-test)))

Coordinated Options

Option	Description
:count	Number of participant instances (or array for matrix)
:spawn-type	:fiber, :thread, or :subprocess (or array for matrix)

(def-test "load test"
  :type :coordinated

  (def-test "server"
    :spawn-type :subprocess
    (emit :ready)
    (serve))

  (def-test "client"
    :count 100
    :spawn-type :thread
    (await :server :ready)
    (run-client)))

Test Body Bindings

Inside test bodies, these local functions are automatically bound by macros (not exported from assay module):

All Test Types

Binding	Description
scratch-dir	(scratch-dir) - returns temp directory path
should-stop?	(should-stop?) - true when graceful shutdown
time-remaining	(time-remaining) - seconds until hard kill
log-message	(log-message level msg) - send to runner
report-data	(report-data table) - send metrics

Matrix Tests

Matrix dimension keys become local bindings:

:matrix {:protocol [:tcp :tls] :size [1024 4096]}
# 'protocol' and 'size' are local vars in test body

Coordinated Tests

Binding	Description
emit	(emit event &opt value) - signal event
await	(await participant event &opt timeout) - wait

Harness (Setup/Teardown)

(def-test "with resources"
  :harness [:server {:setup (fn [cfg ctx] (start-server cfg))
                     :close (fn [srv] (:close srv))}
            :client {:setup (fn [cfg ctx] (connect (ctx :server)))
                     :close (fn [c] (:close c))}]
  # 'server' and 'client' are bound from setup return values
  (test-with server client))

CLI Reference

Option	Description
-v	Increase verbosity (stack: -vvv)
–verbosity N	Set verbosity 0-6
-f, –filter EXPR	Filter expression (see Filtering)
–parallel MODE:N	fiber:N, thread:N, or subprocess:N
–timeout N	Test timeout in seconds
–matrix-sample N	Sample N combos per matrix
–ensured-only	Run only ensured combos
–dry-run	Show what would run
–list MODE	List categories/suites/tests/all
–json FILE	JSON output
–memory N	Memory tracking verbosity 0-3
–show-forms	Show failing assertion forms
–help	Show all options

Filtering

Unified filter syntax: category/suite/test[matrix]<coordinated>

# Category/suite filtering
janet runner.janet -f 'unit/*'           # all unit tests
janet runner.janet -f 'unit/TLS*'        # TLS suites
janet runner.janet -f '!unit/Slow*'      # skip slow tests

# Matrix filtering
janet runner.janet -f '[size=1024]'      # specific value
janet runner.janet -f '[size=1..10]'     # range
janet runner.janet -f '[size=1,2,4..8]'  # mixed

# Coordinated params
janet runner.janet -f '<server=2,client=4>'

Verbosity Levels

Level	Display
0	Final summary only (default)
1	+ Suite pass/fail status
2	+ Category headers, timing, matrix counts
3	+ Suite-level details
4	+ Skip/expected-fail reasons
5	+ Individual combo results
6	+ Stack traces, failing forms

Memory Tracking

janet runner.janet --memory 2 -v2

Memory tracking uses a native C module for cross-platform support (Linux, BSD, macOS). Returns RSS, virtual memory, and platform-specific extras.

Note: Memory tracking is experimental. Values are approximate, especially in parallel execution modes.

Architecture

Subprocess Isolation

Suites run in separate subprocesses providing:

• Crash isolation
• Clean state between suites
• Automatic resource cleanup
• True parallelism

Wire Protocol

The wire module provides unified IPC across transport types:

• Channels (fibers)
• Thread channels (threads)
• Pipes (subprocesses)
• TCP/Unix sockets

The wire protocol uses length-prefixed JDN for reliable message framing. This design was inspired by the spork project's approach to IPC.

Directory Structure

project/
├── test/
│   └── runner.janet
├── suites/
│   ├── unit/
│   │   └── suite-*.janet
│   └── integration/
│       └── suite-*.janet
└── assay/

BEJ Symlink Configuration

Create pre-configured runner aliases using Base64-encoded JDN in symlinks:

# Create config
janet runner.janet --configure-symlink "fast:verbosity=3,filter=unit/*"

# Use it
./fast

Examples

See examples/ directory:

• basic-suite.janet - Simple tests, expected failures, skips
• matrix-suite.janet - Matrix testing with all options
• coordinated-suite.janet - Server/client coordination
• custom-runner.janet - Runner configuration reference

Design Principles

No Global State

All state is passed explicitly via tables, closures, and config objects. Critical for subprocess and thread safety.

Macro Hygiene

User-visible names (protocol, server, emit) are bound directly. Internal variables use gensym.

Matrix Value Immutability

Matrix values must be JDN-encodable. The framework verifies values are not mutated during test execution. Use harness for complex objects.

Self-Testing

cd janet-assay
janet test/runner.janet -v

Roadmap & Contributing

We welcome contributions! janet-assay is evolving rapidly, and there are several high-impact features we are looking to implement.

Areas Seeking Help

Core Framework

• Hang Detection: Mechanism to detect true hangs (blocked on sync I/O) vs long-running tests.
• Dynamic Matrix Logic: Support for function-based matrix decisions (:skip-fn, :accept-fn) to allow more complex filtering than static values.
• Flaky Test Detection: Automated logic to re-run parallel failures in serial mode to distinguish concurrency bugs from logic errors.

Tooling

• Regression Analysis: Standalone tool to compare two JSON output files to detect performance regressions or status changes between runs.
• Test Bisection: Automated bisection to locate the specific test causing a suite hang or crash.

How to Contribute

1. Check existing issues.
2. Open an issue to discuss the approach for major features.
3. Ensure tests pass: janet test/runner.janet
4. Submit a PR or patch for fossil.

Acknowledgments

• JDN utilities based on janet-jdn by Andrew Chambers (MIT License)
• Wire protocol design inspired by spork's IPC patterns
• Used in production by jsec, a project for bringing TLS to Janet
• The Janet Community - Your code has helped me in numerous ways, if you feel you've been left out of acknowledgements and should be acknowledged let me know and I'll rectify it accordingly.

License

ISC License. See LICENSE file.

The JDN module (assay/jdn.janet) is based on code by Andrew Chambers and is licensed under the MIT License. See the file header for details.