WIP multi-target JS test framework. Run tests in any JS runtime.
Theory is an in-development multi target test framework built to ease development of isomorphic JavaScript, and testing of difficult-to-reach environments.
The initial inspiration behind this library was to develop something similar to ava for the purposes of testing VSCode extensions. As initial prototyping of the API surface progressed, it became clear that this could be made more generic such that tests might be run in arbitrary environments and runtimes. It is hoped that this expansion of scope might see more environments tested against with the hope being that more code is specification compliant (or at least handles the various deviations from the spec) and portable.
Tests are exported, avoiding the need for test registry systems and enforcing test identity uniqueness at a language level.
Runtimes are supported via agent packages which keep integration logic out of the core.
Use an efficient agent for test discovery (not test files, just identities). Avoids duplicating effort.
Certain test agents are likely to take a long time to spin up, shouldn't hold up execution of tests on faster agents.
Support specifying a configuration.
Core libraries should never throw, instead expressing errors via the result. try..catch
carries a significant performance cost (so minimise their usage) and as they aren't expressed as part of a functions signiture are a source of runtime errors.
Support randomising execution order and complete isolation to flush out flaky tests which are affected by other tests.
Watch mode is a common feature for JS test frameworks, it should be a part of v1.
Reusing runtimes for tests is faster, but can be tricky. Memory usage may accumulate over time (old parsed source persisting) and many JS APIs are not pure. It should still be an option however, if not in v1.
Opt-in analytics. Handy for;
Useful data includes;
Differentiate between failure types. e.g.;
Mistakes happen, particularly where async/await and callbacks are concerned. Should make it as easy as possible (and at the very least possible) to trace rouge promises and callbacks that are continuing after the test has terminated. This is doable on NodeJS via async hooks, other runtimes are much more difficult.
In the JS ecosystem coverage is typically achieved by running the test runner through a coverage runner. e.g.
# Use NodeJS V8 coverage, with instanbul reporters
c8 ava
# Instanbul
nyc ava
Jest approaches this problem differently. Instead coverage is only captured during execution of individual test functions, which prevents test preparation contributing to coverage that otherwise might mask a gap in test coverage.
The runtime agnostic approach to this project means a different approach is needed to get coverage data from more runtimes anyway, so we might as well borrow coverage trimming in the process.
Chaos engineering, implemented via a plugin. Good way to prove framework extensibility. Good way to help people catch common pitfalls more easily.
Test file as execution alias. Running the file via a supported runtime should test it as though runner was triggered with a single file. Tricky given there is no test wrapper.
node path/to/test-file.test.js
First-class multi-config support. Should be possible to run everything with 1 command and no performance hit. Would necessitate a new process for each config to guard against side effects.
Memory monitor. Useful for identifying memory leaks and unintended memory mismanagement. Handy for tracking down issues like that caches that never are unloaded.
Mutation testing. A technique where a program modifies code to create "mutations" that help to;
Generation Memory Persist generations of tested code for a time such that it may be referenced.
pnpm i
pnpm run build -r
pnpm run test -r