This plug-in adds support to TAP test result files to Jenkins. It lets you specify an ant-like pattern for a directory that contains your TAP files and scans and creates views for your test results in Jenkins.
TAP Plug-in depends on tap4j - a TAP implementation for Java, and on the Jenkins JUnit Plug-in.
NOTE: You may get errors if the JUnit Plug-in is not active in your Jenkins instance (see JENKINS-27227 for more).
The plug-in looks for TAP files like the following one.
1..2
ok 1 - Yahoo!
not ok 2 - org.tap4j.Error...
When a TAP stream like the above is found, the plug-in delegates the parsing the tap4j. The results of tap4j parsing are then analysed, organized and displayed to the user as graphs and custom pages.
You can test your TAP streams with tap4j, using InstantTAP.
One of the easiest ways to learn something new is by examples. So here we will show some examples of TAP streams. For being human-friendly, it shouldn't confuse you. We will use comments to explain each line.
1..2 # a plan stating that we have two tests cases, from 1 to 2.
ok 1 - Yahoo! # the first test result was executed successfully and has a description of ' - Yahoo!'.
not ok 2 - org.tap4j.Error... # unfortunately, the second test result failed. The description here was used to display some nasty Exception.
The plug-in cannot handle prove's default output (since it includes more information than simply TAP, and causes tap4j parser to fail). The best way to handle the prove output is by using Perl module TAP::Harness::Archive. Supposing you have your tests under t/ directory, you can create another directory (say, output) and archive your TAP tests with prove by using a command line similar to the below:
prove t/ --archive output
The result files will be stored under output/t/. You can use a pattern in the plug-in configuration like t/*/.t.
The following is a TAP with attachments, using YAMLish. If you are familiar with YAML, this example should be very easy to read.
1..2
ok 1
not ok 2 - br.eti.kinoshita.selenium.TestListVeterinarians#testGoogle
---
extensions:
Files:
my_message.txt:
File-Title: my_message.txt
File-Description: Sample message
File-Size: 31
File-Name: message.txt
File-Content: TuNvIGNvbnRhdmFtIGNvbSBtaW5oYSBhc3T6Y2lhIQ==
File-Type: image/png
...
The Files entry has an array of files. You have to Base64 encode your content data.
Subtests let you group several TAP streams unto a single one. This way, you can organize your tests in similar fashion to JUnit or TestNG test suites. Indentation is important for TAP subtests.
Subtests and YAMLish are not officially in TAP 13 specification
1..3
ok 1 - First test
1..2
ok 1 - This is a subtest
ok 2 - So is this
1..2
ok 1 - This is a subtest
ok 2 - So is this
ok 2 - An example subtest
ok 3 - Third test
.hpi
(or .jpi
) file.tap
(report
use report.tap
).For commercial support, please get contact us via @tupilabs
JCertif - Make your tests speak TAP - Speaker: Bruno P. Kinoshita September, 2011 - Brazzaville, Congo
Moved to CHANGES.md.
The idea of the plug-in surged after tap4j was created. After learning about Smolder, it became evident that Jenkins could be used as a replacement for it. All that was needed was just adding TAP support to Jenkins and implementing a nice UI to display the test results. After some messages in jenkins-dev-list, Max and Nick commented about their need to show test results in a different manner than how Jenkins was doing at that moment. Soon after that Max, Nick, Bruno (tap4j) and Cesar (tap4j) started to work together, exchanging mail messages and discussing an initial design for this plug-in.
In July 2011 the first version of the plug-in was ready to be released. The graph code used here was adapted from TestNG Plugin (big thanks to the development team, great work). The diagnostic (YAMLish) was implemented in Jelly + Java + CSS. And the road map was incremented based on what Gábor Szabó posted about Smolder and testing reports in his blog (see resources for links).
You can use a heredoc to write a TAP file with Shell, and use it with the plug-in. This is useful for testing.
#!/bin/bash
for x in {1..100}; do
cat > $x.tap <<EOF
ok 1
not ok 2
ok 3 # SKIP
not ok 4
ok 5
ok 6
ok 7
EOF
done ;