Robot Testing: Code written by engineers and readable by well, everyone

Robot Testing is an end-to-end (E2E) technique that mimics a human interacting with software who can see and do anything a human can in an automated way. Its main advantage is that it focuses on the WHAT and not the HOW we are testing it. When a requirement changes or as our project or application evolves, we can continue to use the robot tests and its components.

Developers can build a Robot Testing scenario by grouping multiple robot tests that each correspond to one screen of the application. Non-technical stakeholders can simply glance over the different testing scenarios to obtain the necessary confidence to release the product, knowing that the project is tested E2E in an automated way.

‍

Benefits for Stakeholders

With Robot Testing, we gain confidence in the product because the engineers have control of the tests and can guarantee that they are written correctly. In addition, the tests are readable and understandable by non-technical stakeholders and teammates outside of the engineering team.

Another big benefit of robot tests is that since there are specific segmented “robots” or “tests” for specific components, the tests themselves can be reused and maintained as the project and codebase grows.

‍

Separating the “What” from the “How”

Often for Program Managers, one of the first questions we ask ourselves is: How can I make sure that what we are building behaves as expected? And when it comes to testing, the most important factor for us is “what” we are testing, which should match with the requirements for that feature or project.

Here’s one example of a PM caring about the “what.” The things we care about testing are:

If a user navigates from the home page to the login screen and enters only email and no password, they should receive an error message.
If a user enters email and password and they match, the user should receive a welcome message.

The first thing we are looking for is that we are testing all the steps/requirements in my story. Thanks to the Robot Pattern, we can look at an isolated section of the code, confirm that the engineers are testing what we need tested, and verify that the result is as expected.

When using robot tests, we can also benefit from visual assertion in the form of a video or screen recording that shows the code being run and a simulator showing what it looks like in a specific device (engineers can provide this).

When we separate the “what” from the “how,” we segment the test and are able to expose the composition listing what we are testing in a readable way.

‍

How to Use Robot Testing in a Flutter App

We’ve talked quite a bit already about how the Robot Testing pattern is all about separating the “what” from the “how.” In other words: in one file, the requirements of a certain feature must be presented in semi-natural language, and must not contain any implementation details of the testing tool we might be using (these testing scenarios should not have any code related to the WidgetTester nor from the FlutterDriver). This represents “what” we are testing.

In a different file, the “robot” code contains actions that could be executed on a given screen, and includes details about the different testing frameworks we might be using.

Let’s see a sample of a testing scenario with the Robot Pattern used to test a login screen:

group('Login', () {
  test('shows error message when login information is missing', () async {
    await homeRobot.navigateToLoginPage();
    await loginRobot.enterEmail('notvalid');
    await loginRobot.tapLoginButton();
    await loginRobot.checkInvalidCredentialsMessageIsShown();
  });

  test('authenticates a user with an email and password', () async {
    await loginRobot.enterEmail('email@email.com');
    await loginRobot.enterPassword('s3cr3etP4ssw0rd');
    await loginRobot.tapLoginButton();
    await loginRobot.checkWelcomeMessageIsShown('email@email.com');
  });
});

‍

As you can see, this testing scenario does not contain any implementation details about the testing framework used, and instead focuses on the feature requirements and the actions we need to execute to assert the behavior of our feature. This is the part we refer to as “what” we are testing, and it is highly valuable for stakeholders, as well as for engineers looking for ways to self-document their features.

Each one of the robots used will hide the implementation details of the testing framework. In our case, we use FlutterDriver to test our application — this it the “how” of Robot Testing:

import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';

class LoginRobot {
  const LoginRobot(this.driver);

  final FlutterDriver driver;

  Future enterEmail(String email) async {
    await driver.tap(find.byValueKey('emailTextField'));
    await driver.enterText(email);
  }

  Future enterPassword(String password) async {
    await driver.tap(find.byValueKey('passwordTextField'));
    await driver.enterText(password);
  }

  Future tapLoginButton() async {
    await driver.tap(find.text('Login'));
  }

  Future checkInvalidCredentialsMessageIsShown() async {
    final errorMessageFinder = find.byValueKey('snackbar');
    await driver.waitFor(errorMessageFinder);
  }

  Future checkWelcomeMessageIsShown(String email) async {
    final welcomeMessageFinder = find.text('Welcome $email');
    await driver.waitFor(welcomeMessageFinder);
    expect(await driver.getText(welcomeMessageFinder), 'Welcome $email');
  }
}

For more samples of the Robot testing pattern, you can visit our sample project.

‍

Robot Testing in the Real World

At VGV we aim for 100% test unit and widget test coverage, so these are baked into our Flutter development process. We can incorporate Robot Testing as an E2E testing strategy when it fits the project. We are happy to use it since it can help both our engineering team and our PM teams collaborate and communicate efficiently.

Robot testing is an additional step that can increase the confidence in your ability to release your product to the market, as well as decrease the cost and amount of manual testing needed, especially when there are regular releases that need to be tested.

Robot tests are just one of many other E2E tests that can be used in a project. At VGV, we love to let Robots test our Flutter Apps!

‍