Native Image Comparison Matcher For Vitest Browser Mode

As a developer using Vitest, you want to perform screenshot testing as part of your test suite to verify your UI components render correctly and prevent visual regressions. Currently, Vitest supports writing and comparing snapshots via the toMatchSnapshot matcher, but you would like to mirror this workflow with screenshots and pixel-wise comparison.

The Problem

Currently, Vitest fully supports writing and comparing snapshots via the toMatchSnapshot matcher. However, extending the matcher with jest-image-snapshot comes with some key challenges:

• jest-image-snapshot was built for Jest and is not being regularly updated, making it difficult to maintain Vitest's compatibility with it.
• jest-image-snapshot was designed for a Node environment and is not directly compatible with Vitest's browser mode. Using jest-image-snapshot in the Node + JSDOM/happy-dom environment requires manually managing the browser automation framework and passing the rendered output from the simulated DOM to the browser's DOM.
• The configuration for jest-image-snapshot is completely separate from Vitest's config, making the screenshot comparison aspect a black-box when rolling this kind of solution out to many devs working on the same project.

Suggested Solution

With Vitest's new browser mode already offering access to a native browser DOM and browser screenshot capabilities, we could provide a built-in image comparison matcher in Vitest that mimics the existing toMatchSnapshot matcher:

• If there is no screenshot file at the resolved path, write it and pass the test.
• If there is a screenshot file at the resolved path, compare it pixelwise against the input.
  • If the two match, pass the test
  • Otherwise, fail the test and generate a diff file showing the pixel-wise difference between the two images.

Proposal Example with Vue

import { render } from 'vitest-browser-vue'
import { expect, test } from 'vitest'
import { page } from '@vitest/browser/context'

test('counter button increments the count', async () => {
  const screen = render(Component, {
    props: {
      initialCount: 1,
    }
  })

  await screen.getByRole('button', { name: 'Increment' }).click()

  // Can be further simplified by allowing the matcher to take an Element or selector as input
  const { base64 } = await page.screenshot({ base64: true, path: "/tmp/screenshot.png" })
  await expect(base64).toMatchScreenshot()
})

Dependencies

• pixelmatch for image comparison and diff generation (works on the browser)
• pngJS (has a browser-compatible export, though much of the PNG API relies on the Node Buffer type. It can be made to work via this Buffer for browser implementation, but possibly pngJS can be circumvented altogether, since pixelmatch is the one doing the heavy lifting here)

Alternative Solutions

1. Building a "browser-friendly" version of jest-image-snapshot, either manually or via a tool like browserify. However, this would still be a separate dependency that could fall out of sync with Vitest's development and add friction when integrating into Vitest.
2. Using visual testing SaaS tools like Percy or Chromatic. These introduce external dependencies, costs, and don't provide the same level of developer control or integration with local workflows. Visual testing in the unit testing environment is faster, more stable, and adds more value (IMO) than doing it in an end-to-end fashion.
3. Continuing with custom solutions built on top of Playwright or similar. This requires separate test setups, increasing maintenance burden, fragmenting the testing ecosystem within projects, and leading to a lot of duplicated code across projects/organizations.

Additional Context

In my professional experience, screenshot testing has been an essential part of frontend testing, especially for component-based applications using Vue, React, or similar frameworks. Having spent much time in the past 2 years setting up different screenshot testing pipelines, including in Vitest with both JSDOM and browser mode environments, I believe that most companies and/or maintainers of bigger codebases will not be willing to invest the time necessary to get a custom solution working stably. Having this workflow supported natively in a major testing framework would essentially unlock visual testing for that share of users.

Some additional considerations:

• Screenshot testing becomes increasingly important as component libraries grow and UI consistency is critical
• A native solution could be optimized for Vitest's performance characteristics
• Proper integration with Vitest's watch mode and reporters could significantly improve developer experience when debugging visual regressions

Validations

• [x] Follow our Code of Conduct
• [x] Read the Contributing Guidelines.
• [x] Read the docs.
• [x] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.
  Native Image Comparison Matcher for Vitest Browser Mode: Q&A =====================================================

Q: What is the current state of screenshot testing in Vitest?

A: Currently, Vitest fully supports writing and comparing snapshots via the toMatchSnapshot matcher. However, extending the matcher with jest-image-snapshot comes with some key challenges, such as maintaining Vitest's compatibility with it, manually managing the browser automation framework, and dealing with separate configuration.

Q: Why is a native image comparison matcher needed in Vitest?

A: A native image comparison matcher is needed in Vitest to provide a seamless and efficient way to perform screenshot testing. This would eliminate the need for external dependencies, such as jest-image-snapshot, and provide a more integrated and optimized solution for Vitest's performance characteristics.

Q: What are the benefits of a native image comparison matcher in Vitest?

A: The benefits of a native image comparison matcher in Vitest include:

• Improved performance: A native solution could be optimized for Vitest's performance characteristics.
• Simplified configuration: Proper integration with Vitest's watch mode and reporters could significantly improve developer experience when debugging visual regressions.
• Increased adoption: Having this workflow supported natively in a major testing framework would essentially unlock visual testing for that share of users.

Q: What are the dependencies required for a native image comparison matcher in Vitest?

A: The dependencies required for a native image comparison matcher in Vitest include:

• pixelmatch for image comparison and diff generation (works on the browser)
• pngJS (has a browser-compatible export, though much of the PNG API relies on the Node Buffer type. It can be made to work via this Buffer for browser implementation, but possibly pngJS can be circumvented altogether, since pixelmatch is the one doing the heavy lifting here)

Q: What are the alternative solutions to a native image comparison matcher in Vitest?

A: The alternative solutions to a native image comparison matcher in Vitest include:

• Building a "browser-friendly" version of jest-image-snapshot, either manually or via a tool like browserify.
• Using visual testing SaaS tools like Percy or Chromatic.
• Continuing with custom solutions built on top of Playwright or similar.

Q: What are the considerations for implementing a native image comparison matcher in Vitest?

A: The considerations for implementing a native image comparison matcher in Vitest include:

• Screenshot testing becomes increasingly important as component libraries grow and UI consistency is critical.
• A native solution could be optimized for Vitest's performance characteristics.
• Proper integration with Vitest's watch mode and reporters could significantly improve experience when debugging visual regressions.

Q: How can I contribute to the development of a native image comparison matcher in Vitest?

A: To contribute to the development of a native image comparison matcher in Vitest, follow these steps:

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.