How to Add Widget Tests with a QA Coding Agent for Flutter

Following our previous post where we set up GitHub Actions for automated testing, let's take the next step: creating actual widget tests using GitAuto, our QA Coding Agent. The goal? To demonstrate that we can create, validate, and merge test cases without even opening an IDE.

TL;DR - What's the story?

1. The Setup
2. Creating the Test Case
3. Analyzing GitAuto's Implementation
4. Test Execution and Self-Healing
5. Looking Forward

1. The Setup

Today's case study involves a Flutter UI widget (component) library project. Since the backend is managed by a different company in this case, we're focusing purely on UI components - making widget testing the perfect choice. We'll test the GFBadge widget (where GF stands for GitWidget Framework) which currently lacks test coverage. For this demonstration, I forked the original repository and intentionally removed its existing tests to simulate a real-world scenario where test coverage needs to be added.

2. Creating the Test Case

I created a GitHub issue titled "Add a widget test for lib/components/badge/gf_badge.dart":

Notice that I didn't specify any test requirements - let's see what GitAuto comes up with autonomously. Check the checkbox to assign the issue to GitAuto.

3. Analyzing GitAuto's Implementation

Here is the initial pull request and commit from GitAuto:

Without any specific requirements, GitAuto created three fundamental test cases:

1. Verify text display when text is provided
2. Verify child widget display when no text is provided
3. Verify default color and size values

For those new to Flutter testing (like myself), here's a quick explanation of the key components:

• testWidgets: Function for widget testing
• WidgetTester: Test environment class
• tester.pumpWidget: Renders the widget
• MaterialApp: Root widget (application foundation)
• Scaffold: Screen container

4. Test Execution and Self-Healing

The initial test run failed:

However, GitAuto demonstrated its self-healing capability by automatically creating a fix commit:

Interestingly, the fix was in an unrelated file. Looking at the error logs, I confirmed that the original error was indeed from that file. Thanks to our previous setup where we configured GitHub Actions to trigger on fix commits:

The tests were automatically re-run and passed successfully:

5. Looking Forward

This demonstration shows how we can go from test case creation to merge without opening an IDE. For projects that traditionally required multiple QA engineers, a single QA person could now manage extensive widget test creation by leveraging GitAuto.

Our next goal is to make GitAuto even more autonomous - similar to Dependabot, we want it to automatically discover untested widgets and create pull requests proactively. This could significantly improve how teams handle test coverage in Flutter projects.

Feel free to share your thoughts and feedback here.