Visually Validate Playwright Tests Without Flaky Selectors

Static selectors break with code changes and can't prove that a site "looks good". Is that button really missing or was the id changed? Is the site responsive on mobile? Is the correct image showing? These kinds of tests are impossible to validate with selectors alone and take a lot of time to test manually. GoodLooks.ai lets you visually validate your web pages with natural language prompts instead of selectors.

Check out our other products: TestDriver.ai and Dashcam.io.

Quickstart

git clone [email protected]:dashcamio/goodlooks.git
npm install
npx playwright test

Note that these examples use a demo key that gets rotated weekly; you'll want to create your own API KEY.

Examples

Element Visibility

Validate that a cookie banner shows up.

const { test, expect } = require("@playwright/test");

const goodlooks = require("goodlooks");
goodlooks.configure("zpka_c0d0539ada014283bc974f0fd55835ea_2b745cbf");

expect.extend(goodlooks);

test("framer", async ({ page }) => {
  await page.goto("https://framer.com/");
  await page.waitForTimeout(5000);
  await expect(page).goodlooks("A request to enable cookies shows up");
});

Mobile Responsiveness

Ensure a page is rendering mobile view properly.

const { test, expect, devices } = require("@playwright/test");

const goodlooks = require("goodlooks");
goodlooks.configure("zpka_c0d0539ada014283bc974f0fd55835ea_2b745cbf");

expect.extend(goodlooks);

test("is mobile responsive", async ({ page }) => {
  page.setViewportSize(devices["iPhone X"].viewport);
  await page.goto("https://cnn.com/");

  await expect(page).goodlooks("should be mobile responsive");

});

Application State

Validate that the video player is not playing.

const { test, expect } = require("@playwright/test");

const goodlooks = require("goodlooks");
goodlooks.configure("zpka_c0d0539ada014283bc974f0fd55835ea_2b745cbf");

expect.extend(goodlooks);

test("rickroll", async ({ page }) => {
  await page.goto("https://www.youtube.com/watch?v=dQw4w9WgXcQ");
  await expect(page).goodlooks("video is not playing");
});

Color

Ensure a page renders correct image contents via img or canvas.

const { test, expect, devices } = require("@playwright/test");

const goodlooks = require("goodlooks");
goodlooks.configure("zpka_c0d0539ada014283bc974f0fd55835ea_2b745cbf");

expect.extend(goodlooks);

test("ycombinator", async ({ page }) => {
  await page.goto("https://news.ycombinator.com");
  await expect(page).goodlooks(
    "there is an orange strip at the top of the page"
  );
});

Image Contents

Ensure a page renders correct image contents via img or canvas.

const { test, expect, devices } = require("@playwright/test");

const goodlooks = require("goodlooks");
goodlooks.configure("zpka_c0d0539ada014283bc974f0fd55835ea_2b745cbf");

expect.extend(goodlooks);

test("correct image appears", async ({ page }) => {
  await page.goto("https://eloquentjavascript.net/");
  await expect(page).goodlooks("there is bird on this page");
});

Opinionated Image Contents

Ensure a diverse representation of people appears on the page. Of course, this judgement is left up to AI.

const { test, expect } = require("@playwright/test");
const goodlooks = require("goodlooks");
goodlooks.configure("zpka_c0d0539ada014283bc974f0fd55835ea_2b745cbf");

test("diversity", async ({ page }) => {
  await page.goto("https://diversityequityinclusion.com/about/");
  await expect(page).goodlooks("diverse people show up");
});

Setup

Install via NPM.

npm install goodlooks

Use in Playwright tests!

const { test, expect, devices } = require("@playwright/test");
const goodlooks = require("goodlooks");
goodlooks.configure("zpka_c0d0539ada014283bc974f0fd55835ea_2b745cbf");
expect.extend(goodlooks);

Debugging

It seems like GoodLooks is wrong? I'm sure the element exists that I'm checking for.

Run your playwright tests in UI mode:

npx playwright test --ui

Check the log to see the time frame where the goodlooks check is called. Hover over that.
A red highlight will appear in the Playwright UI, showing you the GUI state when the screenshot was taken
You can also see the debug logs within the Console or Error tabs.

What part of the page is checked?

Only the visible part of the page is checked, not the full page. You must scroll to check other page parts.

What are the limits?

The AI is great at identifying what is in an image, but it's not great at identifying where those things are in relation to other things. For example, don't ask GoodLooks to count items to validate their position on the screen.