Agnostic visual regression testing service with Sauce Labs and Travis CI.
MPL-2.0 License
Agnostic visual regression testing service with Sauce Labs and Travis CI.
The world of Sherlocked:
Sherlocked is hosted at sherlocked.dev.mozaws.net.
To use Sherlocked in your project, install the Node library, get a Sauce Labs API key, write a Sherlocked test script, and hook it into your Travis build.
npm install --save-dev sherlocked
You must have a Sauce Labs account and find your API key. Once you are logged in, you can find it at your SauceLabs account page like below.
Keep your API key for later.
A Sherlocked test script adds screenshots to the case with
.investigate(captureName, captureFn)
where captureFn
operates
on a WebdriverIO client. Then the investigation begins
with .begin(environmentList)
where environmentList
is a list of
configurations of what environments we want to take captures of, in the form of
Selenium's [desiredCapabilities]
(https://code.google.com/p/selenium/wiki/DesiredCapabilities)
require('sherlocked')
.investigate('Home Page', function(client) {
return client
.url('http://localhost:8000')
.waitFor('main');
})
.begin([
{
browserName: 'firefox',
version: '40',
platform: 'OS X 10.9',
},
{
browserName: 'chrome',
platform: 'OS X 10.9',
}
]);
First, you will want to encrypt your Sauce Labs API key and set it as an
environment variable in Travis CI. -add
will automatically add it to
your .travis.yml
.
gem install travis
travis encrypt SAUCE_ACCESS_KEY=my-sauce-key --add
travis encrypt SAUCE_USERNAME=my-sauce-username --add
Then, you'll want to enable your Travis' sauce_connect
addon to allow
Travis CI to tunnel through the firewall into Sauce Labs.
addons:
sauce_connect: true
Lastly, all you need to do is to call your Sherlocked test script from your Travis CI build.
node my-sherlocked-script.js
GET /builds/
Lists builds. Note this will not fetch related fields such as captures
.
POST /builds/
Create a build. Returns 201 on success.
Parameter | Description |
---|---|
travisId | The Travis build ID, retrieved from the Travis CI environment. |
travisPullRequest | The pull request triggering the build, retrieved from the Travis CI environment. |
travisRepoSlug | The slug of the repository (owner_name /repo_name ), retrieved from the TravisCI environment. |
GET /builds/:travisId
Get a build.
Example response:
{
"travisBranch": "newAwesomeDetectiveHat",
"travisCommit": "01189998819991197253",
"travisId": 221,
"travisPullRequest": 222,
"travisRepoSlug": "sherlocked/adlerjs",
"masterBuild": {
"travisBranch": "master",
...
},
"captures": [
{
"browserEnv": {
"id": 1,
"name": "firefox",
"platform": "OS X 10.9",
"version": "40",
},
"captures": {
"homepageOnMobile": {
"diff": {
"dimensionDifferenceHeight": 0,
"dimensionDifferenceWidth": 0,
"isSameDimensions": 1,
"mismatchPercentage": null,
"src": "https://sherlocked.paas/api/captures/diff/N239"
},
"name": "homepageOnMobile",
"sauceSessionId": "N239",
"src": "https://sherlocked.paas/api/captures/N239"
},
"homepageOnDesktop": {
"name": "homepageOnDesktop",
"sauceSessionId": "N221B",
"src": "https://sherlocked.paas/N221B.png"
}
},
"masterCaptures": {
"homepageOnMobile": {
"name": "homepageOnMobile",
"sauceSessionId": "N221",
"src": "https://sherlocked.paas/N221.png"
},
},
]
}
POST /builds/:buildId/captures/
Attach a capture to a build. Returns 201 on success.
Parameter | Description |
---|---|
browserName | Browser that the capture was created in (e.g., firefox ) |
browserPlatform | Browser platform that the capture was created in (e.g., OS X 10.9 ) |
browserVersion | Browser version that the capture was created in (e.g., 40 ) |
image | Base64 image |
name | Name of the capture (e.g., homePageOnDesktop ) |
sauceSessionId | Sauce Labs session ID, each session generates one capture. |
Sherlocked consist of several moving parts: the backend API, the npm module, and the web frontend.
The backend service consisting of the API and database. Powered by Express, with Bookshelf as the ORM. Unfortunately, we have to upload and host images since Sauce Labs only takes captures containing the desktop and browser chrome whereas we want captures of the whole pages themselves.
npm install
node_modules/.bin/migrate
node index.js
This will start a server on port 1077 by default. Set SHERLOCKED_PORT environment variable to configure it.
The npm module ran by consuming projects to hook Sauce Labs, Travis CI, and the Sherlocked API together. Follow the Setup section on how to use it.
npm install --save-dev sherlocked
To let Sauce Labs through a firewall, use Sauce Connect. You download the binary, and then run it in the background with your sauce username and sauce key:
bin/sc -u my-sauce-username -k my-sauce-key
The frontend client that consumes the Sherlocked API, displaying image comparisons and visual diffs. Powered by React, Gulp, and Browserify.
npm install
gulp
This will start a server on port 2118 by default.
It is possible to run Sherlocked manually on a project from the CLI. For
development purposes to test the npm module, you can clone Sherlocked and run
npm link
from within the Sherlocked lib/
folder.
First install and run Sauce Connect. This will allow us to tunnel into Sauce Labs.
bin/sc -u my-sauce-username -k my-sauce-key -i 100
SAUCE_KEY=my-sauce-key SAUCE_USERNAME=my-sauce-username \
TRAVIS_BUILD_ID=abc TRAVIS_JOB_NUMBER=100 node ./my-sherlocked-script.js
Sherlocked always has clients in need.
Are you using Sherlocked? Let us know!