First-Class End-to-End Testing — Test Recorder, Runtime Emulator, Program Presets, Ice Test Syntax, and Selector API

[hecrj]

This PR implements the foundations of a first-class end-to-end testing toolkit compatible with any iced application.

The Test Recorder

The most important new tool of this update is the tester (or test recorder).

The tester can be enabled through the new tester feature flag. When enabled, your iced application will be decorated with an interface that allows you to both record and play tests as well as export and import them as files.

iced-tester.mp4

The long term goal of this tool is to be a fully-featured testing suite for QA engineers to create tests that emulate user interactions as well as define assertions and expectations about a user interface.

The Emulator

Once end-to-end tests have been recorded and saved, we need a way to run them—ideally in a completely headless environment.

The new Emulator type implements a headless runtime where test instructions can be executed.

Unlike the existing Simulator type, an Emulator does execute tasks and subscriptions and, therefore, side effects will be materialized. It's as close to running the real thing as possible!

An Emulator can be configured with different modes, which change the waiting strategy before proceeding to the next instruction.

You should rarely have to use this type directly. Instead, users are expected to leverage the new iced_test::run function to run an entire directory of tests for their applications:

fn main() -> iced::Result {
    application().run()
}

fn application() -> Application<impl Program<Message = Message>> {
    iced::application(State::new, State::update, State::view)
        // ...
}

#[cfg(test)]
mod tests {
    use super::application;

    #[test]
    fn it_passes_the_e2e_tests() -> Result<(), Error> {
        iced_test::run(
            application(),
            format!("{}/tests", env!("CARGO_MANIFEST_DIR")),
        )
    }
}

Program Presets

In order to be useful, end-to-end tests must be reproducible.

This can be a challenge, since executing them often produces side effects that change external state. When done naively, these side effects can affect next executions of the test; losing reproducibility in the process.

To mitigate this, an iced application can now define a list of presets. A Preset simply describes a certain way to create the initial state of an application.

Tests can then reference a particular preset, which can be leveraged to ensure an application is exactly in a reproducible state before the test is executed.

For instance, the todos example defines a couple of presets:

fn presets() -> impl IntoIterator<Item = Preset<Todos, Message>> {
    [
        Preset::new("Empty", || {
            (Todos::Loaded(State::default()), Task::none())
        }),
        Preset::new("Carl Sagan", || {
            (
                Todos::Loaded(State {
                    input_value: "Make an apple pie".to_owned(),
                    filter: Filter::All,
                    tasks: vec![Task {
                        id: Uuid::new_v4(),
                        description: "Create the universe".to_owned(),
                        completed: false,
                        state: TaskState::Idle,
                    }],
                    dirty: false,
                    saving: false,
                }),
                Task::none(),
            )
        }),
    ]
}

Ice Test Syntax

End-to-end tests that have been exported are saved as text files using a new custom syntax.

These test files are expected to have the .ice extension and, as a result, we call these tests "ice tests" or simply "ice". I think the name works well not only because the library is named iced, but also because end-to-end tests are meant to "freeze" the behavior of an application completely in time.

The details of the syntax are very likely to change in the near future and, therefore, relying to much on this new feature is not encouraged yet! The changes here simply set up the first foundations for all of the testing infrastructure.

In any case, if you are curious, here is what the current carl_sagan.ice test looks like for the todos example, which is already running as part of our CI pipeline:

viewport: 500x800
mode: Immediate
preset: Empty
-----
click "What needs to be done?"
type "Create the universe"
type enter
type "Make an apple pie"
type enter
expect "2 tasks left"
click "Create the universe"
expect "1 task left"
click "Make an apple pie"
expect "0 tasks left"

As you can see, there is a bit of metadata at the beginning—stating the environment of the test—followed by the list of test instructions.

The syntax is meant to be easy to read and write. In the long run, I'd like for QA engineers to be able to write tests from scratch without necessarily using the recorder.

The Selector API

In order to implement these tools, I have formalized the old Selector API in iced_test and moved it into its own subcrate: iced_selector.

This new Selector API can be leveraged to easily traverse the widget tree and extract any data from it:

/// A type that traverses the widget tree to "select" data and produce some output.
pub trait Selector {
    /// The output type of the [`Selector`].
    ///
    /// For most selectors, this will normally be a [`Target`]. However, some
    /// selectors may want to return a more limited type to encode the selection
    /// guarantees in the type system.
    ///
    /// For instance, the implementations of [`String`] and [`str`] of [`Selector`]
    /// return a [`target::Text`] instead of a generic [`Target`], since they are
    /// guaranteed to only select text.
    type Output;

    /// Performs a selection of the given [`Candidate`], if applicable.
    ///
    /// This method traverses the widget tree in depth-first order.
    fn select(&mut self, candidate: Candidate<'_>) -> Option<Self::Output>;

    /// Returns a short description of the [`Selector`] for debugging purposes.
    fn description(&self) -> String;
}

This trait is implemented for String, &str, widget::Id, and any FnMut(Candidate<_>) -> Option<T>.

When the selector feature flag is enabled, the API can be leveraged through the widget::selector module:

fn find_first_task_bounds() -> Task<Option<Rectangle>> {
    selector::delineate("Create the universe")
}

And that should be all! I don't expect these features to be quite useful right away. I am just planting the seeds for now.

[Decodetalkers]

I have some thing about the settings... sometime I want to use the special setting for some special platform, but in this pr, the settings is fixed. So can it become a generic type?

Ok, I think I can ignore it. no mind