Tasty is a modern testing framework for Haskell.
It lets you combine your unit tests, golden tests, QuickCheck/SmallCheck properties, and any other types of tests into a single test suite.
Features:
To find out what's new, read the change log.
Here's how your test.hs
might look like:
import Test.Tasty
import Test.Tasty.SmallCheck as SC
import Test.Tasty.QuickCheck as QC
import Test.Tasty.HUnit
import Data.List
import Data.Ord
main = defaultMain tests
tests :: TestTree
tests = testGroup "Tests" [properties, unitTests]
properties :: TestTree
properties = testGroup "Properties" [scProps, qcProps]
scProps = testGroup "(checked by SmallCheck)"
[ SC.testProperty "sort == sort . reverse" $
\list -> sort (list :: [Int]) == sort (reverse list)
, SC.testProperty "Fermat's little theorem" $
\x -> ((x :: Integer)^7 - x) `mod` 7 == 0
-- the following property does not hold
, SC.testProperty "Fermat's last theorem" $
\x y z n ->
(n :: Integer) >= 3 SC.==> x^n + y^n /= (z^n :: Integer)
]
qcProps = testGroup "(checked by QuickCheck)"
[ QC.testProperty "sort == sort . reverse" $
\list -> sort (list :: [Int]) == sort (reverse list)
, QC.testProperty "Fermat's little theorem" $
\x -> ((x :: Integer)^7 - x) `mod` 7 == 0
-- the following property does not hold
, QC.testProperty "Fermat's last theorem" $
\x y z n ->
(n :: Integer) >= 3 QC.==> x^n + y^n /= (z^n :: Integer)
]
unitTests = testGroup "Unit tests"
[ testCase "List comparison (different length)" $
[1, 2, 3] `compare` [1,2] @?= GT
-- the following test does not hold
, testCase "List comparison (same length)" $
[1, 2, 3] `compare` [1,2,2] @?= LT
]
And here is the output of the above program:
(Note that whether QuickCheck finds a counterexample to the third property is determined by chance.)
tasty is the core package. It contains basic definitions and APIs and a console runner.
In order to create a test suite, you also need to install one or more «providers» (see below).
The following providers exist:
It's easy to create custom providers using the API from Test.Tasty.Providers
.
Ingredients represent different actions that you can perform on your test suite. One obvious ingredient that you want to include is one that runs tests and reports the progress and results.
Another standard ingredient is one that simply prints the names of all tests.
It is possible to write custom ingredients using the API from Test.Tasty.Runners
.
Some ingredients that can enhance your test suite are:
tasty
by itself forces you to explicitly write out the TestTree
yourself.
The packages listed below allow you to write tests at the top-level, and will
automatically collect them into a single TestTree
.
tasty
to a benchmark frameworkcriterion
and gauge
.Options allow one to customize the run-time behavior of the test suite, such as:
There are two main ways to set options:
When using the standard console runner, the options can be passed on the
command line or via environment variables. To see the available options, run
your test suite with the --help
flag. The output will look something like this
(depending on which ingredients and providers the test suite uses):
% ./test --help
Mmm... tasty test suite
Usage: test [-p|--pattern PATTERN] [-t|--timeout DURATION] [--no-progress]
[-l|--list-tests] [-j|--num-threads NUMBER] [-q|--quiet]
[--hide-successes] [--color never|always|auto] [--ansi-tricks ARG]
[--smallcheck-depth NUMBER] [--smallcheck-max-count NUMBER]
[--quickcheck-tests NUMBER] [--quickcheck-replay SEED]
[--quickcheck-show-replay] [--quickcheck-max-size NUMBER]
[--quickcheck-max-ratio NUMBER] [--quickcheck-verbose]
[--quickcheck-shrinks NUMBER]
Available options:
-h,--help Show this help text
-p,--pattern PATTERN Select only tests which satisfy a pattern or awk
expression
-t,--timeout DURATION Timeout for individual tests (suffixes: ms,s,m,h;
default: s)
--no-progress Do not show progress
-l,--list-tests Do not run the tests; just print their names
-j,--num-threads NUMBER Number of threads to use for tests execution
(default: # of cores/capabilities)
-q,--quiet Do not produce any output; indicate success only by
the exit code
--hide-successes Do not print tests that passed successfully
--min-duration-to-report DURATION
The minimum amount of time a test can take before
tasty prints timing information (suffixes: ms,s,m,h;
default: s)
--color never|always|auto
When to use colored output (default: auto)
--ansi-tricks ARG Enable various ANSI terminal tricks. Can be set to
'true' or 'false'. (default: true)
--smallcheck-depth NUMBER
Depth to use for smallcheck tests
--smallcheck-max-count NUMBER
Maximum smallcheck test count
--quickcheck-tests NUMBER
Number of test cases for QuickCheck to generate.
Underscores accepted: e.g. 10_000_000
--quickcheck-replay SEED Random seed to use for replaying a previous test run
(use same --quickcheck-max-size)
--quickcheck-show-replay Show a replay token for replaying tests
--quickcheck-max-size NUMBER
Size of the biggest test cases quickcheck generates
--quickcheck-max-ratio NUMBER
Maximum number of discared tests per successful test
before giving up
--quickcheck-verbose Show the generated test cases
--quickcheck-shrinks NUMBER
Number of shrinks allowed before QuickCheck will fail
a test
Every option can be passed via environment. To obtain the environment variable
name from the option name, replace hyphens -
with underscores _
, capitalize
all letters, and prepend TASTY_
. For example, the environment equivalent of
--smallcheck-depth
is TASTY_SMALLCHECK_DEPTH
.
Note on boolean options: by convention, boolean ("on/off") options are specified
using a switch on the command line, for example --quickcheck-show-replay
instead of --quickcheck-show-replay=true
. However, when
passed via the environment, the option value needs to be True
or False
(case-insensitive), e.g. TASTY_QUICKCHECK_SHOW_REPLAY=true
.
If you're using a non-console runner, please refer to its documentation to find out how to configure options during the run time.
You can also specify options in the test suite itself, using
localOption
. It can be applied not only to the whole test tree, but also to
individual tests or subgroups, so that different tests can be run with
different options.
It is possible to combine run-time and compile-time options, too, by using
adjustOption
. For example, make the overall testing depth configurable
during the run time, but increase or decrease it slightly for individual
tests.
This method currently doesn't work for ingredient options, such as --quiet
or
--num-threads
. You can set them by setting the corresponding environment
variable before calling defaultMain
:
import Test.Tasty
import System.Environment
main = do
setEnv "TASTY_NUM_THREADS" "1"
defaultMain _
It is possible to restrict the set of executed tests using the -p/--pattern
option.
Tasty patterns are very powerful, but if you just want to quickly run tests containing foo
somewhere in their name or in the name of an enclosing test group, you can just
pass -p foo
. If you need more power, or if that didn't work as expected, read
on.
A pattern is an awk expression. When the expression is evaluated, the field $1
is set to the outermost test group name, $2
is set to the next test group
name, and so on up to $NF
, which is set to the test's own name. The field $0
is set to all other fields concatenated using .
as a separator.
As an example, consider a test inside two test groups:
testGroup "One" [ testGroup "Two" [ testCase "Three" _ ] ]
When a pattern is evaluated for the above test case, the available fields and variables are:
$0 = "One.Two.Three"
$1 = "One"
$2 = "Two"
$3 = "Three"
NF = 3
Here are some examples of awk expressions accepted as patterns:
$2 == "Two"
— select the subgroup Two
$2 == "Two" && $3 == "Three"
— select the test or subgroup named Three
in the subgroup named Two
$2 == "Two" || $2 == "Twenty-two"
— select two subgroups$0 !~ /skip/
or ! /skip/
— select tests whose full names (including group names) do not contain the word skip
$NF !~ /skip/
— select tests whose own names (but not group names) do not contain the word skip
$(NF-1) ~ /QuickCheck/
— select tests whose immediate parent group nameQuickCheck
As an extension to the awk expression language, if a pattern pat
contains only
letters, digits, and characters from the set ._ -
(period, underscore, space, hyphen),
it is treated like /pat/
(and therefore matched against $0
).
This is so that we can use -p foo
as a shortcut for -p /foo/
.
The only deviation from awk that you will likely notice is that Tasty
does not implement regular expression matching.
Instead, $1 ~ /foo/
means that the string foo
occurs somewhere in $1
,
case-sensitively. We want to avoid a heavy dependency of regex-tdfa
or
similar libraries; however, if there is demand, regular expression support could
be added under a cabal flag.
The following operators are supported (in the order of decreasing precedence):
The following built-in functions are supported:
substr(s, m[, n])
Return the at most n
-character substring of s
that begins at
position m
, numbering from 1. If n
is omitted, or if n
specifies
more characters than are left in the string, the length of the substring
will be limited by the length of the string s
.
tolower(s)
Convert the string s
to lower case.
toupper(s)
Convert the string s
to upper case.
match(s, pat)
Return the position, in characters, numbering from 1, in string s
where the
pattern pat
occurs, or zero if it does not occur at all.
pat
must be a literal, not an expression, e.g. /foo/
.
length([s])
Return the length, in characters, of its argument taken as a string, or of the whole record, $0
, if there is no argument.
In order to run tests in parallel, you have to do the following:
-threaded
+RTS -N -RTS
.To apply timeout to individual tests, use the --timeout
(or -t
) command-line
option, or set the option in your test suite using the mkTimeout
function.
Timeouts can be fractional, and can be optionally followed by a suffix ms
(milliseconds), s
(seconds), m
(minutes), or h
(hours). When there's no
suffix, seconds are assumed.
Example:
./test --timeout=0.5m
sets a 30 seconds timeout for each individual test.
The following options control behavior of the standard console interface:
It is possible to add custom options, too.
To do that,
IsOption
includingOptions
ingredientaskOption
.See the Custom options in Tasty article for some examples.
There may be several ways to organize your project. What follows is not Tasty's requirements but my recommendations.
Place your test suite sources in a dedicated subdirectory (called tests
here) instead of putting them among the main library sources.
The directory structure will be as follows:
my-project/
my-project.cabal
src/
...
tests/
test.hs
Mod1.hs
Mod2.hs
...
test.hs
is where your main
function is defined. The tests may be
contained in test.hs
or spread across multiple modules (Mod1.hs
, Mod2.hs
,
...) which are then imported by test.hs
.
Add the following section to the cabal file (my-project.cabal
):
test-suite test
default-language:
Haskell2010
type:
exitcode-stdio-1.0
hs-source-dirs:
tests
main-is:
test.hs
build-depends:
base >= 4 && < 5
, tasty >= 0.7 -- insert the current version here
, my-project -- depend on the library we're testing
, ...
All the above applies, except you can't depend on the library if there's no library. You have two options:
Hs-source-dirs
. Note that thisTasty executes tests in parallel to make them finish faster. If this parallelism is not desirable, you can declare dependencies between tests, so that one test will not start until certain other tests finish.
Dependencies are declared using the after
or sequentialTestGroup
combinator:
after AllFinish "pattern" my_tests
will execute the test tree my_tests
only after allafter AllSucceed "pattern" my_tests
will execute the test tree my_tests
only after allmy_tests
will be skipped.sequentialTestGroup groupName dependencyType [tree1, tree2, ..]
will execute all teststree1
first, after which it will execute all tests in tree2
, and so forth. Likeafter
, dependencyType
can either be set to AllFinish
or AllSucceed
.The relevant types are:
after
:: DependencyType -- ^ whether to run the tests even if some of the dependencies fail
-> String -- ^ the pattern
-> TestTree -- ^ the subtree that depends on other tests
-> TestTree -- ^ the subtree annotated with dependency information
sequentialTestGroup
:: TestName -- ^ name of the group
-> DependencyType -- ^ whether to run the tests even if some of the dependencies fail
-> [TestTree] -- ^ trees to execute sequentially
-> TestTree
data DependencyType = AllSucceed | AllFinish
The pattern follows the same AWK-like syntax and semantics as described in
Patterns. There is also a variant named after_
that accepts the
AST of the pattern instead of a textual representation.
Let's consider some typical examples. (A note about terminology: here
by "resource" I mean anything stateful and external to the test: it could be a file,
a database record, or even a value stored in an IORef
that's shared among
tests. The resource may or may not be managed by withResource
.)
Two tests, Test A and Test B, access the same shared resource and cannot be run concurrently. To achieve this, make Test A a dependency of Test B:
testGroup "Tests accessing the same resource"
[ testCase "Test A" $ ...
, after AllFinish "Test A" $
testCase "Test B" $ ...
]
Test A creates a resource and Test B uses that resource. Like above, we make
Test A a dependency of Test B, except now we don't want to run Test B if Test
A failed because the resource may not have been set up properly. So we use
AllSucceed
instead of AllFinish
testGroup "Tests creating and using a resource"
[ testCase "Test A" $ ...
, after AllSucceed "Test A" $
testCase "Test B" $ ...
]
Here are some caveats to keep in mind when using patterns to specify dependencies in Tasty:
If Test B depends on Test A, remember that either of them may be filtered out
using the --pattern
option. Collecting the dependency info happens after
filtering. Therefore, if Test A is filtered out, Test B will run
unconditionally, and if Test B is filtered out, it simply won't run.
Tasty does not currently check whether the pattern in a dependency matches anything at all, so make sure your patterns are correct and do not contain typos. Fortunately, misspecified dependencies usually lead to test failures and so can be detected that way.
Dependencies shouldn't form a cycle, otherwise Tasty with fail with the message "Test dependencies have cycles." A common cause of this is a test matching its own dependency pattern.
Using dependencies may introduce quadratic complexity. Specifically, resolving dependencies is O(number_of_tests × number_of_dependencies), since each pattern has to be matched against each test name. As a guideline, if you have up to 1000 tests, the overhead will be negligible, but if you have thousands of tests or more, then you probably shouldn't have more than a few dependencies.
Additionally, it is recommended that the dependencies follow the natural order of tests, i.e. that the later tests in the test tree depend on the earlier ones and not vice versa. If the execution order mandated by the dependencies is sufficiently different from the natural order of tests in the test tree, searching for the next test to execute may also have an overhead quadratic in the number of tests.
Use sequentialTestGroup
to mitigate these problems.
Q: When my tests write to stdout/stderr, the output is garbled. Why is that and what do I do?
A: It is not recommended that you print anything to the console when using the console test reporter (which is the default one). See #103 for the discussion.
Some ideas on how to work around this:
Q: Why doesn't the --hide-successes
option work properly? The test headings
show up and/or the output appears garbled.
A: This can happen sometimes when the terminal is narrower than the
output. A workaround is to disable ANSI tricks: pass --ansi-tricks=false
on the command line or set TASTY_ANSI_TRICKS=false
in the environment.
See issue #152.
Q: Patterns with slashes do not work on Windows. How can I fix it?
A: If you are running Git for Windows terminal, it has a habit of
converting slashes to backslashes. Set MSYS_NO_PATHCONV=1
when running the
Git for Windows terminal and MSYS2_ARG_CONV_EXCL=*
when running a MinGW
bash directly to prevent this behaviour, or follow other suggestions from
Known
Issues.
Blog posts and other publications related to tasty. If you wrote or just found something not mentioned here, send a pull request!
We only support the GHC/base versions from the last 5 years.
Roman Cheplyaka is the primary maintainer.
Oliver Charles is the backup maintainer. Please get in touch with him if the primary maintainer cannot be reached.