Last week we made a number of exciting updates to Very Good CLI focused on improvements to testing Dart and Flutter apps 🎉.
Very Good CLI v0.6.0 now features a new test command that provides additional functionality on top of the existing command line tooling for dart test and flutter test.
Included are performance optimizations, improvements to the developer experience when measuring and enforcing tests coverage metrics, and improvements when working with multimodule monorepos.
Very Good CLI's new test command includes the following flags and arguments. You can always see the most up-to-date list by running very_good test --help.
How its made 🛠
In order to build a custom test command, we first needed a way to programatically run tests that would fit nicely with our existing tooling for Very Good CLI. As a result, we created the Very Good Test Runner package which provides APIs to run Dart and Flutter tests programatically thanks to the JSON Reporter Protocol.
Internally, very_good_test_runner uses the Process class to spawn dart test and flutter test processes respectively. We specify the --reporter=json option to receive a machine-readable representation of the underlying test runner's progress. very_good_test_runner exposes a stream of TestEvent objects which consumers can subscribe to in order to build custom testing tooling.
Custom testing tools can enable you to customize how test results are displayed, detect tests that are abnormally slow, re-run only the tests that have failed from previous test runs, and more. Another great example of a custom testing tool is the spec_cli from Invertase. The possibilities for custom test tooling are endless and we're very excited to see what other tools the community builds.
Testing Improvements ✨
Very Good CLI takes advantage of a known performance optimization to decrease test run times. From our experience, this is particularly significant in large codebases when collecting coverage metrics. As the number of tests increases, the cost of running the tests and collecting coverage metrics also increases so applying this optimization can reduce test runs quite dramatically (see below comparison).
This additional optimization step is performed by default but can be disabled with the --no-optimization flag. You may want to disable optimizations in order to improve the traceability of failing tests or for smaller codebases with fewer tests to reduce the initial overhead of performing the optimization step.
As part of the performance optimization step, Very Good CLI uses a mason brick to aggregate all tests and generate a single .test_runner.dart file.
Then, the test execution consists of executing the single .test_runner.dart file:
The new test command also provides an option to specify a minimum test coverage threshold. If the reported coverage drops below that threshold, the test run will exit with an error code.
This is particularly useful when running tests as part of your continuous integration because you can fail a build if test coverage drops below the accepted threshold (we recommend 100% 💯).
In some cases, it's handy to be able to exclude certain files from test coverage.
For example, you may want to exclude code generated by build_runner from test coverage. The --exclude-coverage option allows you to specify a glob which will be used to exclude files from coverage.
When working on a monorepo with many packages, it can be quite tedious to run tests in all sub-packages. For example, after a large refactor you may want to verify that no regressions were introduced. Previously, you'd have to manually run the tests in each affected package one at a time. With the new test command in Very Good CLI, it's easy to run tests in all nested packages via the --recursive flag (-r for short).
❗️ Currently, the very_good test command must be executed from the root directory of a package so be sure to keep that in mind when using the --recursive flag.
Very Good CLI now ships with some new testing capabilities that we hope will make running tests in Flutter/Dart even more enjoyable. We're excited to continue to expand the capabilities of Very Good CLI and plan to incorporate the testing improvements into very_good_workflows shortly.
Until next time, happy testing 🧪🦄!