I propose changing the folder structure, such that each test file lives in its own folder. A test folder contains a Typst file (e.g. "test.typ") plus the reference image.

CeTZ currently uses its own test script, which structures the tests as given above, but we would like to use typst-test.

CeTZ's tests are recognized by recursively scanning the tests/ directory for test.typ files and, if found, use the folder of those files as test-cases. This allows nesting/grouping tests by topic by just putting them in common parent directories.

By adding typst as a dependency, test output generation and comparison could be done entirely in memory. This seems like a sensible thing to do, here are some of the pros and cons:

Pros

• less I/O by doing in-memory reference to image comparison
  • increased performance
• typst-test can be run without an external binary for typst, reducing error paths and improving usability
• multiple tests can be parsed from a single test file with most of the config living in this file
  • avoids config files for configuring per test behavior
• allows for separation of typst-test into a library crate and binary crate
  • opens the door for others to use the same underlying implementation for test running

Cons

• fixes typst-test to a typst version for its default invocation
• the default behavior of typst-test may diverge from the user's installed typst compiler on a version mismatch
• massively increased dependency tree and binary size
• increased complexity of the implementation
  • many implementations will have to be copied verbatim from typst-cli to ensure the same world behavior

When generating a diff image, the following things could be improved:

• generate diff images on a page size mismatch (#8)
• highlight changes, bounding boxes of small change sets can be created very easily using a border finding algorithm (see for example Suzuki and Abe ¹)
• highlight which kind of change it is
  • red: deletion from reference to output
  • green: addition from reference to output
• overlay these boxes on the output or reference image to make it easier to visually compare

Currently whenever a new test is created it is populated with "Hello World!". I then have to copy and paste in the package's import statement and other setup which is common to all tests I want to run.

I would like a way to change the default test on a per project basis.

The current implementation is a little brittle, this has a lot of downsides:

• error reporting is sometimes done using ?, other times it's swallowed and simply printed to stdout
• some actually likely error cases are not properly handled and result in a simple error message with no hint
• as it currently is, reporting a nonzero exit code on test failure would need an ugly hack
  • this is currently the primary blocker for adoption of typst-test in cetz and in code bases which test using CI in general

The core should be reworked into a solid foundation on which new features can be built. At this stage, the project is still fairly small, and a rewrite is feasible.

I want to achieve the following goals with the rewrite:

• Abstract user facing reporting sufficiently to allow in place progress per test for multiple tests
• Properly handle common error cases better
• Make code easier to follow and reason about
• Make code, which depends on FS operations, easier to test automatically
• Abstract the testing in a way that makes separation into a core library and cli frontend easier down the line

Built from commit version: ce06d7e
Rust: 1.75.0
Platform: Windows 11 Home 22H2
Repository: https://github.com/fenjalien/metro/

Running typst-test run works as expected apart from the diff images are not being generated in the diff folders. I've linked the repository I'm having the issue with.

By extracting the core implementation of test scaffolding and test running, other projects such as typst-lsp and typst-preview could benefit from it without needing to interface with the bare bones binary interface.

Typst-preview already supports a special presentation mode for polylux, it's not too far-fetched to assume they could be interested in a special test-watching mode. Likewise, typst-lsp could allow discovering and running tests easily.

This is blocked by #12.

With a folder structure like test/higher/lower/test.typ and no test.typ in higher/. Using the add command removes everything in higher/.

Allow users to run custom actions between test stages.

On compelling use case for this is being able to test typstfmt (see #3). This avoids pulling it in as a dependency or even tying it to a command line option of typst-test at all.

With sufficiently flexible custom actions, typstfmt could generate the references from the unaltered test script and the output from the altered version, failing if the formatting introduce a visual regression.

Looks really nice!

Maybe it could be use to test formatters as well!

Given a formatter and a test file

• Tests that the compilation gives the same result before and after formatting
• Else pinpoints where it broke the code file
  Apply diffs with dichotomy and test the output, it's a bit of a headache to implement because the diff types I used were hard to understand I think

It could happily live next to typstfmt when I get up to it but I think it might be better suited in typst-test