How to use Alchemist for Flutter golden tests

Golden testing is a method of testing the appearance of widgets using screenshot comparisons. It can be leveraged to help test purely visual components like charts or custom painters that might otherwise be really difficult to test. In this tutorial, you'll learn how to use both Flutter's built-in functionality for golden tests and the open source package Alchemist (developed by Betterment and VGV) to improve your testing experience.

The example we'll be using for our testing comes from DartPad. It is a Flutter project that uses a CustomPainter to create a sunflower image with a variable number of seeds. This is a prime example for using golden testing because the CustomPainter is painting directly to the canvas. We could alternatively test it using a mock Canvas and verifying that the correct painting functions are called in the correct order; however, this would be very fragile to implementation changes and would also fail to serve as documentation for the code being tested. Golden tests will be a lot easier for other engineers to maintain because the tests can be easily read and understood without knowing how the painting logic works.

‍

Using Flutter's built-in golden tests

Flutter already has built-in support for golden testing. The built-in golden test tooling is great because of how simple it is to start writing tests and how similar the syntax is to regular widget tests, but the downside is that they quickly become hard to read and maintain as you create more test variations. Additionally, there are issues with font rendering between platforms, so if you create your golden images on a Mac then the tests will fail on a Linux CI machine.

Generally, golden tests work the same way as normal Flutter widget tests. Each test case is contained in a testWidgets function which uses tester.pumpWidget to load the widget under test. Any desired interactions, such as tapping a button, can be performed once the widget is loaded. Finally, the matchesGoldenFile matcher within an expectLater call can be used to verify the rendered widget matches a previous render. For example, with our sunflower app it would look like this:

void main() {
  const rootKey = Key('root_sizedBox');

  Widget buildSubject(int seedCount) {
    return SizedBox(
      key: rootKey,
      width: 400,
      height: 400,
      child: CustomPaint(
        painter: SunflowerPainter(seedCount),
      ),
    );
  }

  group('SunflowerPainter with matchesGoldenFile', () {
    testWidgets('matches expected output with 100 seeds', (tester) async {
      await tester.pumpWidget(buildSubject(100));

      await expectLater(
        find.byKey(rootKey),
        matchesGoldenFile('sunflowerPainter_seedCount_100.png'),
      );
    });

    testWidgets('matches expected output with 20 seeds', (tester) async {
      await tester.pumpWidget(buildSubject(20));

      await expectLater(
        find.byKey(rootKey),
        matchesGoldenFile('sunflowerPainter_seedCount_20.png'),
      );
    });

    testWidgets('matches expected output with 2000 seeds', (tester) async {
      await tester.pumpWidget(buildSubject(2000));

      await expectLater(
        find.byKey(rootKey),
        matchesGoldenFile('sunflowerPainter_seedCount_2000.png'),
      );
    });
  });
}

When you run this test file the first time, you can pass the --update-goldens flag to flutter test to generate screenshots for each of the test cases. After updating golden tests, each subsequent test run will compare the rendered widgets to the generated images (golden files), failing the test if they don't match. When the widget changes and you want to regenerate the golden files, then you can use --update-goldens to generate new images and commit the updated golden files to your repository.

‍

Using Alchemist with golden tests

Alchemist is a framework designed to assist with golden testing. It addresses pain points found while using existing golden testing solutions on a large codebase, such as fixing continuous integration test flakes and simplifying injecting a custom theme. It's also designed to have a more declarative API, which should help improve test readability.

The golden tests using Alchemist have a different structure completely from typical flutter widget tests. They begin with a call to goldenTest. This function has a number of parameters you can use to customize the test. Every test needs a description, a filename, and a widget to test. Typically, the widget to test is a GoldenTestGroup. This is a special type of widget which organizes a number of labeled GoldenTestScenarios into a grid layout to keep the golden file organized. Here's what the completed goldenTest looks like:

group('SunflowerPainter with alchemist', () {
  goldenTest(
    'matches expected output',
    fileName: 'sunflowerPainter',
    widget: GoldenTestGroup(
      children: [
        GoldenTestScenario(
          name: 'with 100 seeds',
          child: buildSubject(100),
        ),
        GoldenTestScenario(
          name: 'with 20 seeds',
          child: buildSubject(20),
        ),
        GoldenTestScenario(
          name: 'with 2000 seeds',
          child: buildSubject(2000),
        ),
      ],
    ),
  );
});

Like before, running flutter test --update-goldens will make Alchemist generate the rendered widget images in the goldens/ directory adjacent to the test file. Inside will be a ci/ subdirectory, along with another subdirectory for each platform on which the tests run. Because CI golden files are not affected by font rendering inconsistencies, they are kept separate from the platform golden files (which can be omitted from source control, if desired). Golden tests running on CI can be generated on any platform and are guaranteed to pass on CI. The platform-specific goldens will not run on any other platform besides the one on which they were generated to avoid headaches caused by font-rendering inconsistencies.

The golden tests shown have been pretty basic, although very useful for verifying a specific type of widget.

‍

What's next

Golden tests can also be used with more advanced use cases, such as testing an app's UI library. Alchemist has some more built-in functionality to support this use case, like custom theming and gesture support. To get started with more advanced functionality, check out the Alchemist docs.

‍

Joanna May contributed to this article.