Supercharge your Flutter tests with Very Good CLI

Last week we made a number of exciting updates to Very Good CLI focused on improvements to testing Dart and Flutter apps 🎉.

Very Good CLI v0.6.0 now features a new test command that provides additional functionality on top of the existing command line tooling for dart test and flutter test.

Included are performance optimizations, improvements to the developer experience when measuring and enforcing tests coverage metrics, and improvements when working with multimodule monorepos.

‍

Overview 🚀

Very Good CLI's new test command includes the following flags and arguments. You can always see the most up-to-date list by running very_good test --help.

$ very_good test --help
Run tests in a Dart or Flutter project.

Usage: very_good test [arguments]
-h, --help                            Print this usage information.
    --coverage                        Whether to collect coverage information.
-r, --recursive                       Run tests recursively for all nested packages.
    --[no-]optimization               Whether to apply optimizations for test performance.
                                      (defaults to on)
    --exclude-coverage                A glob which will be used to exclude files that match from the coverage.
-x, --exclude-tags                    Run only tests that do not have the specified tags.
    --min-coverage                    Whether to enforce a minimum coverage percentage.
    --test-randomize-ordering-seed    The seed to randomize the execution order of test cases within test files.

Run "very_good help" to see global options.

‍

How its made 🛠

In order to build a custom test command, we first needed a way to programatically run tests that would fit nicely with our existing tooling for Very Good CLI. As a result, we created the Very Good Test Runner package which provides APIs to run Dart and Flutter tests programatically thanks to the JSON Reporter Protocol.

Internally, very_good_test_runner uses the Process class to spawn dart test and flutter test processes respectively. We specify the --reporter=json option to receive a machine-readable representation of the underlying test runner's progress. very_good_test_runner exposes a stream of TestEvent objects which consumers can subscribe to in order to build custom testing tooling.

import 'package:very_good_test_runner/very_good_test_runner.dart';

void main() {
  const arguments = ['--coverage'];
  const workingDirectory = 'path/to/project';

  // Run Dart tests and collect coverage in the working directory.
  dartTest(
    arguments: arguments,
    workingDirectory: workingDirectory,
  ).listen((TestEvent event) {
    // React to `TestEvent` instances.
  });

  // Run Flutter tests and collect coverage in the working directory.
  flutterTest(
    arguments: arguments,
    workingDirectory: workingDirectory,
  ).listen((TestEvent event) {
    // React to `TestEvent` instances.
  });
}

Custom testing tools can enable you to customize how test results are displayed, detect tests that are abnormally slow, re-run only the tests that have failed from previous test runs, and more. Another great example of a custom testing tool is the spec_cli from Invertase. The possibilities for custom test tooling are endless and we're very excited to see what other tools the community builds.

‍

Testing Improvements ✨

Performance Optimizations

Very Good CLI takes advantage of a known performance optimization to decrease test run times. From our experience, this is particularly significant in large codebases when collecting coverage metrics. As the number of tests increases, the cost of running the tests and collecting coverage metrics also increases so applying this optimization can reduce test runs quite dramatically (see below comparison).

This additional optimization step is performed by default but can be disabled with the --no-optimization flag. You may want to disable optimizations in order to improve the traceability of failing tests or for smaller codebases with fewer tests to reduce the initial overhead of performing the optimization step.

# run tests with performance optimizations
very_good test

# run tests without performance optimizations
very_good test --no-optimization

As part of the performance optimization step, Very Good CLI uses a mason brick to aggregate all tests and generate a single .test_runner.dart file.

// GENERATED CODE - DO NOT MODIFY BY HAND
// Consider adding this file to your .gitignore.

import 'package:test/test.dart';

import 'app/view/app_test.dart' as app_view_app_test_dart;
import 'counter/cubit/counter_cubit_test.dart' as counter_cubit_counter_cubit_test_dart;
import 'counter/view/counter_page_test.dart' as counter_view_counter_page_test_dart;

void main() {
  group('app_view_app_test_dart', app_view_app_test_dart.main);
  group('counter_cubit_counter_cubit_test_dart', counter_cubit_counter_cubit_test_dart.main);
  group('counter_view_counter_page_test_dart', counter_view_counter_page_test_dart.main);
}

Then, the test execution consists of executing the single .test_runner.dart file:

final generator = await MasonGenerator.fromBundle(testRunnerBundle);

...

await generator.generate(
  target,
  vars: vars,
  fileConflictResolution: FileConflictResolution.overwrite,
);

...

flutterTest(
  workingDirectory: workingDirectory,
  arguments: [
    ...
    if (optimizePerformance) p.join('test', '.test_runner.dart')
  ],
).listen((event) {...});

Enforcing Coverage

The new test command also provides an option to specify a minimum test coverage threshold. If the reported coverage drops below that threshold, the test run will exit with an error code.

# Run tests, collect coverage, and enforce 100% coverage
very_good test --coverage --min-coverage 100

This is particularly useful when running tests as part of your continuous integration because you can fail a build if test coverage drops below the accepted threshold (we recommend 100% 💯).

Excluding Coverage

In some cases, it's handy to be able to exclude certain files from test coverage.

For example, you may want to exclude code generated by build_runner from test coverage. The --exclude-coverage option allows you to specify a glob which will be used to exclude files from coverage.

# Run tests, collect coverage, exclude generated dart files from coverage, and enforce 100% coverage
very_good test --coverage --exclude-coverage "*.g.dart" --min-coverage 100

Run Recursively

When working on a monorepo with many packages, it can be quite tedious to run tests in all sub-packages. For example, after a large refactor you may want to verify that no regressions were introduced. Previously, you'd have to manually run the tests in each affected package one at a time. With the new test command in Very Good CLI, it's easy to run tests in all nested packages via the --recursive flag (-r for short).

# Run all tests in the current package as well as nested packages.
very_good test --recursive

❗️ Currently, the very_good test command must be executed from the root directory of a package so be sure to keep that in mind when using the --recursive flag.

‍

Summary

Very Good CLI now ships with some new testing capabilities that we hope will make running tests in Flutter/Dart even more enjoyable. We're excited to continue to expand the capabilities of Very Good CLI and plan to incorporate the testing improvements into very_good_workflows shortly.

Until next time, happy testing 🧪🦄!