Turn your markdown docs into test suites running Scala CLI
APACHE-2.0 License
Turn each Scala CLI snippet from your markdown documentation into a test case and each markdown document into a test suite.
Create test-snippets.scala
(use at least Java 11!):
//> using scala 3.3.3
//> using jvm temurin:1.11.0.23
//> using dep "com.kubuszok::scala-cli-md-spec:0.1.1"
import com.kubuszok.scalaclimdspec.*
@main def run(args: String*): Unit = testSnippets(args.toArray) { cfg =>
new Runner.Default(cfg) // or provide your own :)
}
then run it with Scala CLI:
# run all tests
scala-cli run test-snippets.scala -- "$PWD/docs"
# run only tests from Section in my-markdown.md
scala-cli run test-snippets.scala -- --test-only="my-markdown.md#Section*" "$PWD/docs"
To see how one can customize the code to e.g. inject variables or use the newest library version in an arbitrary markdown documentation generator see Chimney's example.
If you are not providing any modification, you can run it straight from the Coursier:
# run all tests
coursier launch com.kubuszok:scala-cli-md-spec_3:0.1.1 -M com.kubuszok.scalaclimdspec.testSnippets -- "$PWD/docs"
# run only tests from Section in my-markdown.md
coursier launch com.kubuszok:scala-cli-md-spec_3:0.1.1 -M com.kubuszok.scalaclimdspec.testSnippets -- --test-only="my-markdown.md#Section*" "$PWD/docs"
each markdown is its own test suite
by default only Scala (and Java) snipets containing //> using
are considered tests
other snippets are considered pseudocode and are ignored
// will be tested
//> using scala 3.3.3
println("yolo")
// will NOT be tested
println("yolo")
by default snippets are tested for the lack of errors
by the lack of errors we mean that Scala CLI returns 0
// should pass
//> using scala 3.3.3
println("yolo")
// thou shall NOT pass!
//> using scala 3.3.3
throw Exception("yolo")
if there is // expected output:
followed by inline comments in the immediate next lines,
the snippet will be expected to succeed and its standard output will be expected to contain the content provided in these comments
// should pass
//> using scala 3.3.3
println("yolo")
// expected output:
// yolo
// thou shall NOT pass!
//> using scala 3.3.3
println("yolo")
// expected output:
// eee macarena!
if there is // expected error:
followed by inline comments in the immediate next lines,
the snippet will be expected to fail and its standard error will be expected to contain the content provided in these comments
// should pass
//> using scala 3.3.3
throw Exception("yolo")
// expected error:
// yolo
// should pass
//> using scala 3.3.3
summon[String]
// expected error:
// No given instance of type String was found
// thou shall NOT pass!
//> using scala 3.3.3
println("yolo")
// expected error:
// yolo
by default each snippet is a standalone Scala snippet, it will be tested in a separate directory, containing a single snippet.sc
file
multiple pieces of code can be combined into one multi-file snippet with:
// file: [filename] - part of [example name used for grouping]
syntax - e.g. // file: filename.scala - part of X example
would group all X example
snippets in the same directory,
and use filename.scala
as a filename for this particular piece of code
// file: model.scala - part of multi-file
case class Model(a: Int)
// file: example.sc - part of multi-file
println(Model(10))
// expected output:
// Model(10)
With multi-file //> using
is not required to consider the code as a Scala CLI test. Remember that to make it work, like with normal Scala CLI app,
there should be either exactly one .sc
file or only .scala
files with exactly one explicitly defined main
.
if at least one file in a multi-file snippet has a name ending with .test.scala
, then scala-cli test [dirname]
will be used unstead of scala-cli run [dirname]
(useful for e.g. defining macros in the compile scope and showing them in the test scope since Scala CLI is NOT multi modular and you cannot demonstrate macros in another way)
// file: macro.scala - part of macro example
//> using scala 3.3.3
object MyMacro:
inline def apply[A](a: A): Unit = ${ applyImpl[A]('a) }
import scala.quoted.*
def applyImpl[A: Type](a: Expr[A])(using Quotes): Expr[Unit] = '{ () }
// file: macro.test.scala - part of macro example
//> using test.dep org.scalameta::munit::1.0.0-RC1
class MacroSpec extends munit.FunSuite {
test("Macro(a) should do thing") {
assert(MyMacro("wololo") == ())
}
}
Java snippets should not only use java
in markdown, but also define // file: filename.java - part of ...
// file: MyEnum.java - part of java enum example
enum MyEnum {
ONE, TWO;
}
// file: snippet.sc - part of java enum example
println(MyEnum.values())
if --test-only
flag is used, only suites containing at least 1 matching snippet and, within them, only
the matching snippets will be run and displayed (but all markdowns still need to be read to find snippets
and match them against the pattern!)
# quotes around * are needed in shell
# test all snippets
scala-cli run test-snippets.scala -- --test-only '*' "$PWD/docs"
# test all snippets in my-markdown.md
scala-cli run test-snippets.scala -- --test-only 'my-markdown.md#*' "$PWD/docs"
# test all snippets in my-markdown.md, in section name starting with My Section
scala-cli run test-snippets.scala -- --test-only 'my-markdown.md#My section*' "$PWD/docs"
Some people would ask: if you need to make sure code in your documentation compiles, why not use something like mdoc?
Well, because mdoc doesn't work for my cases:
scalacOptions
, different Scala versions and different libraries availablescalacOptions
, libraries) code might not be exactlyMeanwhile, there is one perfectly suitable tool for the job - Scala CLI. With
its //> using
directives it is very easy to create a self-contained, perfectly reproducible snippet. As a matter
of the fact, it even supports running snippets in markdown files.
Its markdown support has a downside, however, because this mode considers all snippets to be defined in the same scope (same Scala version,
same libraries, same compiler options - different //> using
classes are appended and may override conflicting options).
This library:
/tmp
subdirectoriesallowing each snippet to be a self-contained, reproducible example.