October 6, 2022

Robotic Notes

All technology News

Automating Visual UI Tests with Playwright and GitHub Actions · mmazzarolo.com

11 min read


Visual user interface testing (also known as regression testing) is a testing technique to confirm that your changes do not have unexpected effects on the user interface. Typically, such tests take a snapshot of the entire application under test or a specific element and then compare the image to a pre-approved baseline image. If the images are the same (within a specified pixel tolerance), the web application is determined to look the same to the user. If there are differences, then there has been some change in the DOM layout, fonts, colors, or other visual properties that need to be investigated.

This post will explore how to automate visual regression testing of a “modern” web application using Playwright and GitHub actions. The goal is to build a test setup that checks for UI regressions on every download request and allows for selective updating of base images when needed.

Basic knowledge of JavaScript (or TypeScript) and GitHub Actions is recommended.

We assume that you will integrate this setup into an existing web application. If you want to try it from scratch, I recommend creating a new web application using Vite.

You can find a complete example of the resulting application in this GitHub repo.

For reference, here’s what the small web app we’re testing looks like:



website

Playwright setting

For our testing setup, we will be using Playwright, an E2E testing framework. I like Playwright because it provides a great developer experience and good default settings, but you can achieve the same result with similar tools like Cypress or Selenium.

To install Playwright cd into our project and run the following command:

npm init playwright

We’ll be prompted with a few different options (eg JavaScript vs. TypeScript, etc.). When prompted, we need to add a GitHub Actions workflow to run our CI tests easily:

✔ Add a GitHub Actions workflow? (y/N) · true

Once complete, Playwright will add a sample test ./tests/example.spec.tssample E2E file in ./tests-examples/demo-todo-app.spec.ts (which we can ignore) and the Playwright config file in the ./playwright.config.ts.

✔ Success! Created a Playwright Test project at ~/your-project-dir

Inside that directory, you can run several commands:

  npx playwright test
    Runs the end-to-end tests.

  npx playwright test --project=chromium
    Runs the tests only on Desktop Chrome.

  npx playwright test example
    Runs the tests in a specific file.

  npx playwright test --debug
    Runs the tests in debug mode.

  npx playwright codegen
    Auto generate tests with Codegen.

We suggest that you begin by typing:

    npx playwright test

And check out the following files:
  - ./tests/example.spec.ts - Example end-to-end test
  - ./tests-examples/demo-todo-app.spec.ts - Demo Todo App end-to-end tests
  - ./playwright.config.ts - Playwright Test configuration

Visit https://playwright.dev/docs/intro for more information. ✨

The Playwright configuration file generated by the scaffolding process (./playwright.config.ts) provides some good default settings, but I recommend applying a few changes.

First, to simplify our setup, update projects list for running our Chromium-only tests:


projects: [
  
    name: "chromium",
    use: 
      ...devices["Desktop Chrome"],
    ,
  ,

We may update the list of projects later to include more browsers/devices if needed.

Then configure webServer section so that Playwright can automatically serve our application when we run our tests:


webServer: 
  command: 'npm run dev --port 8080',
  port: 8080,
  reuseExistingServer: true
,

Generate the initial snapshots

The sample test generated by Playwright (./tests/example.spec.ts) is not a visual regression test, so let’s delete its content and add a small test that suits our needs:

import  test, expect  from "@playwright/test";

test("example test", async ( page ) => 
  await page.goto("https://mmazzarolo.com/"); 
  await expect(page).toHaveScreenshot();
);

For visual regression testing, Playwright includes the ability to produce and visually compare snapshots using await expect(page).toHaveScreenshot(). On first run, the Playwright test will generate reference snapshots. Subsequent runs will be compared to the reference.

We are finally ready to run your test:

The test taker will say something like:

Error: example.spec.ts-snapshots/example-test-1-chromium-darwin.png is missing in snapshots, writing actual.

This is because there was no base image yet, so this method took a bunch of snapshots until two consecutive snapshots matched, and saved the last snapshot to the file system. It is now ready to be added to the repository (c tests/example.spec.ts-snapshots/example-test-1-chromium-darwin.png):



The image generated by our first test
The image generated by our first test

Now if we run our test again:

Now the test should succeed because the current UI matches the UI of the reference snapshot generated in the previous run.

Update snapshots locally

Let’s get to the interesting part.

If we make some visual change to the UI and rerun our tests, they will fail and Playwright will show us a nice difference between the “actual” and the “expected” snapshot:

In cases like this, when we want to make some voluntary changes to a page, we need to update the reference snapshots. We can do this with --update-snapshots flag:

npx playwright test --update-snapshots
[chromium] › example.spec.ts:3:1 › example test
tests/example.spec.ts-snapshots/example-test-1-chromium-darwin.png is re-generated, writing actual.

Now that we’ve seen how to run visual tests and update snapshots locally, we’re ready to move on to the CI flow.

Running the tests in CI using GitHub Actions

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that lets you automate your build, test, and deployment pipeline. With GitHub Actions, we can create workflows that build and test each pull request to a repository.

A good starting point for setting up visual regression testing is to run the tests every time a pull request is created and updated. Fortunately, Playwright already generates a handy GitHub Actions workflow for running tests for this particular use case in .github/workflows/playwright.yml.
This workflow should work perfectly out of the box, but I recommend modifying it a bit to:

  1. Install only the browsers we target with our tests (--with deps chromium);
  2. Save test result artifacts only when tests fail to avoid storing unnecessary artifacts (if: failure()).
 name: Playwright Tests

 on:
   push:
     branches: [main, master]
   pull_request:
     branches: [main, master]

 jobs:
   test:
     timeout-minutes: 60
     runs-on: ubuntu-latest
     steps:
       
       - uses: actions/checkout@v2
       - uses: actions/setup-node@v2
         with:
           node-version: "14.x"
       - name: Install dependencies
         run: npm install
       - name: Install Playwright Browsers
-        run: npx playwright install --with-deps
+        run: npx playwright install --with-deps chromium
       
       - name: Run Playwright tests
         run: npx playwright test
       
       - uses: actions/upload-artifact@v2
-        if: always()
+        if: failure()
         with:
           name: playwright-report
           path: playwright-report/
           retention-days: 30

Once we commit this workflow to the GitHub repo, submitting a new pull request will trigger the Playwright tests.

However, our test will probably fail at this point because the reference snapshots in our repo are taken in an environment that differs from the one used in CI (unless we created them on a machine running Ubuntu).

For more information about the test result, we can check the GitHub Action output or the test artifacts:



github action artifacts

For example, if we’ve generated the reference macOS snapshots, we’ll get an error that the Linux snapshots were not found:

Error: tests/example.spec.ts-snapshots/example-test-1-chromium-linux.png is missing in snapshots

Let’s see how we can update the reference snapshots for the CI environment.

Generate the reference snapshots locally using the --update-snapshots was a piece of cake, but for CI it’s a different story because we have to decide where, how and when to store them.

There are many ways we can handle this flow, but for the sake of simplicity, let’s start with the simple one.
One pattern that works well for me is to use a GitHub Action workflow to update the reference snapshots when a specific comment with the text “/update-snapshots” is posted in a pull request.
The idea is that whenever we make a pull request that we expect to affect the UI, we can post the “/update-snapshots” comment so that CI generates and commits the updated snapshots to the pull request branch.













name: Update Snapshots

on:
  
  
  
  
  issue_comment:
    types: [created]

jobs:
  updatesnapshots:
    
    
    if: $ github.event.issue.pull_request && github.event.comment.body == '/update-snapshots'
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
      
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0 
          token: $ secrets.GITHUB_TOKEN 
      
      
      
      - name: Get SHA and branch name
        id: get-branch-and-sha
        run: |
          sha_and_branch=$(\
            curl \
              -H 'authorization: Bearer $ secrets.GITHUB_TOKEN ' \
              https://api.github.com/repos/$ github.repository /pulls/$ github.event.issue.number  \
            | jq -r '.head.sha," ",.head.ref');
          echo "::set-output name=sha::$(echo $sha_and_branch | cut -d " " -f 1)";
          echo "::set-output name=branch::$(echo $sha_and_branch | cut -d " " -f 2)"
      
      - name: Fetch Branch
        run: git fetch
      - name: Checkout Branch
        run: git checkout $ steps.get-branch-and-sha.outputs.branch 
      
      - uses: actions/setup-node@v2
        with:
          node-version: "14.x"
      - name: Install dependencies
        run: npm install
      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium
      
      - name: Update snapshots
        run: npx playwright test --update-snapshots --reporter=list
      
      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: "[CI] Update Snapshots"

If we publish this workflow, create a new pull request and add the “/update-snapshots” comment, CI will take care of generating the reference snapshots.



snapshot update successful

OK!

That’s it — we have fully automated visual regression testing of our continuous integration. Whenever new changes are published in new pull requests, our tests will ensure that we don’t make any UI changes by mistake. We can also update our base snapshots by posting a comment in the pull request.

If your project supports preview links (like the auto-generated ones from Netlify, Vercel, etc.), read on. Otherwise, skip to the Conclusion section below.

Run Playwright tests against a deployment preview (Netlify, Vercel, etc.)

If your web application runs on platforms like Netlify and Vercel, you probably use their “Deploy Preview” feature: a way to preview changes to the pull request without affecting the web application in production. Deployment previews are enabled by default for GitHub pull requests, and they work by deploying them to a unique URL different from the one your production site uses.

If your web application uses deployment previews, we can integrate them into our visual regression testing workflows to compare snapshots against them instead of the local web server.
This approach has two advantages. First, we avoid running a local web server just to run the tests. Second, by running our tests against a preview link, we’ll create more reliable snapshots because preview links are a 1:1 representation of what we should see in production.

From a high-level view, there are three main changes we need to make to our codebase to run our Playwright tests against deployment previews:

  1. Allow passing the testing URL as a parameter to make our tests aware of the deployment preview link.
  2. Update our GitHub Action workflows to wait for the deployment preview to complete.
  3. Update our GitHub Action workflows to pass Playwright the deployment preview URL (as an environment variable).

Here’s how we can achieve this in Netlify (if you’re using Vercel or another platform that supports deployment visualization, the changes should be almost identical).

First, update use.baseURL value of playwright.config.ts to get the deployment URL as an environment variable (WEBSITE_URL):

use: 
  baseURL: process.env.WEBSITE_URL,

Also, let’s disable Playwright’s web server if a WEBSITE_URL an environment variable is provided:

webServer: process.env.WEBSITE_URL
  ? undefined
  : 
    command: "npm run dev --port 8080",
    port: 8080,
    reuseExistingServer: true,
  ,

Then update our playwright.yaml workflow for running tests against the deployment preview.
To wait for the Netlify deployment preview URL, we can use mmazzarolo/wait-for-netlify-action GitHub action.

mmazzarolo/wait-for-netlify-action is a fork of probablyup/wait-for-netlify-action. Default, probablyup/wait-for-netlify-action assumes that it runs within a workflow triggered by a pull request push. In our case, update-snapshots.yml workflow is triggered by a comment, so I forked this GitHub action to ensure that it works in any workflow, regardless of what triggered it.

The wait-for-netlify-action A GitHub Action requires two things:

  1. Setup a NETLIFY_TOKEN GitHub Action secret with Netlify Personal Access Token.
  2. Pass Netlify Site ID (from Netlify: Settings → Site Details → General) to site_id parameter in the workflow.
 name: Playwright Tests

 on:
   push:
     branches: [main, master]
   pull_request:
     branches: [main, master]
   workflow_run:
     workflows: ["Update Snapshots"]
     types:
       - completed

 jobs:
   test:
     timeout-minutes: 60
     runs-on: ubuntu-latest
     steps:
       
       - uses: actions/checkout@v2
       - uses: actions/setup-node@v2
         with:
           node-version: "14.x"
       - name: Install dependencies
         run: npm install
       - name: Install Playwright Browsers
         run: npx playwright install --with-deps chromium
+      
+      - name: Wait for Netlify Deploy
+        uses: mmazzarolo/wait-for-netlify-action@8a7a8d8cf5b313c916d805b76cc498380062d268
+        id: get-netlify-preview-url
+        with:
+          site_id: "YOUR_SITE_ID"
+        env:
+          NETLIFY_TOKEN: $ secrets.NETLIFY_TOKEN 
+      
+      
+      - name: Run Playwright tests
-        run: npx playwright test
+        run: WEBSITE_URL=$ steps.get-netlify-preview-url.outputs.url  npx playwright test
       
       - uses: actions/upload-artifact@v2
         if: failure()
         with:
           name: playwright-report
           path: playwright-report/
           retention-days: 30

We can finally update update-snapshots.yml workflow just like we did above.

 name: Update Snapshots

 on:
   
   
   
   issue_comment:
     types: [created]

 jobs:
   updatesnapshots:
     
     
     if: $ github.event.issue.pull_request && github.event.comment.body == '/update-snapshots'
     timeout-minutes: 60
     runs-on: ubuntu-latest
     steps:
       
       - uses: actions/checkout@v2
         with:
           fetch-depth: 0 
           token: $ secrets.GITHUB_TOKEN 
       
       
       
       - name: Get SHA and branch name
         id: get-branch-and-sha
         run: |
           sha_and_branch=$(\
             curl \
               -H 'authorization: Bearer $ secrets.GITHUB_TOKEN ' \
               https://api.github.com/repos/$ github.repository /pulls/$ github.event.issue.number  \
             | jq -r '.head.sha," ",.head.ref');
           echo "::set-output name=sha::$(echo $sha_and_branch | cut -d " " -f 1)";
           echo "::set-output name=branch::$(echo $sha_and_branch | cut -d " " -f 2)"
       
       - name: Fetch Branch
         run: git fetch
       - name: Checkout Branch
         run: git checkout $ steps.get-branch-and-sha.outputs.branch 
       
       - uses: actions/setup-node@v2
         with:
           node-version: "14.x"
       - name: Install dependencies
         run: npm install
+      
+      - name: Wait for Netlify Deploy
+        uses: mmazzarolo/wait-for-netlify-action@8a7a8d8cf5b313c916d805b76cc498380062d268
+        id: get-netlify-preview-url
+        with:
+          site_id: "YOUR_SITE_ID"
+          commit_sha: $ steps.get-branch-and-sha.outputs.sha 
+        env:
+          NETLIFY_TOKEN: $ secrets.NETLIFY_TOKEN 
       - name: Install Playwright browsers
         run: npx playwright install --with-deps chromium
+      
       - name: Update snapshots
-        run: npx playwright test --update-snapshots --reporter=list
+        run: WEBSITE_URL=$ steps.get-netlify-preview-url.outputs.url  npx playwright test --update-snapshots --reporter=list
       
       - uses: stefanzweifel/git-auto-commit-action@v4
         with:
           commit_message: "[CI] Update Snapshots"

That should do it. From now on, our Playwright visual regression tests will run against the deployment previews.

Conclusion

I hope this blog post has given you a solid foundation for building your visual testing setup. A few ideas how you can improve it further:

  • You’ll probably want to test full-screen snapshots using fullScreen parameter instead of just the visible viewport. And maybe taking mobile photos too.
  • If your web application loads parts of the UI asynchronously (eg images, videos), you’ll probably want to wait for them to load or skip them from the tests.
  • You can limit /update-snapshots command so that it can only be called by the owners of the repository.
  • You can trigger the snapshot update flow from a different source (eg webhooks) instead of relying on GitHub comments.
  • You can store the snapshots in a third-party storage solution.
  • If you use deployment previews, you can optimize workflows by parallelizing the step waiting for the preview connection with the rest of the workflow.

Acknowledgments

In this blog post, I have copied excerpts and paragraphs from the Playwright and Cypress “Functional VS visual testing” documentation.



Source link