Case Study: Adding an Automated Unit Test with GitAuto

We all know unit testing is important. But let me guess - while you agree with its importance, you haven't been able to prioritize it? There are customer-facing features waiting to be developed, critical bugs to be fixed, and it's hard to justify slowing down development to add tests? Well, what if we could delegate this task to an AI Coding Agent? If it works, your human engineers can continue focusing on their current priorities!

The Challenge

Let's start with a simple test case. I created an issue titled "Add unit tests to services/github/branch_manager.py". This file, services/github/branch_manager.py, is from GitAuto's own repository and handles GitHub branch operations. I'll admit - I developed this without unit tests initially. Let's see how GitAuto handles adding them.

First Attempt

I simply assigned the issue to GitAuto with just the title - no detailed requirements. Within a minute, it created this pull request. The result was surprisingly good, and it even taught me something about unit testing:

tests/services/github/test_branch_manager.py
+ @patch('services.github.branch_manager.requests.get')
+ @patch('services.github.branch_manager.create_headers')
+ def test_get_default_branch(mock_create_headers, mock_requests_get):

Interesting! It used the @patch decorator to mock external dependencies, isolating the unit test from external libraries and functions. It even correctly handled multiple decorators, maintaining the proper order between decorators and function arguments.

Iteration 1: Constants

While the implementation looked good, there was one issue:

tests/constants.py
+ owner = "test_owner"
+ repo = "test_repo"
+ token = "test_token"
+ expected_branch_name = "main"
+ expected_commit_sha = "abc123"

These hardcoded values would be frequently used across tests. For better maintainability, they should be in a separate constants file. I added this as a requirement to the original issue and had GitAuto try again.

Iteration 2: Class vs. Functions

The next GitAuto's attempt introduced a class-based approach:

tests/services/github/test_branch_manager.py
+ class TestBranchManager(unittest.TestCase):
+     @patch('services.github.branch_manager.requests.get')
+     def test_get_default_branch(self, mock_get):

This highlights an interesting aspect of current AI models - their non-deterministic nature can be both a blessing and a curse. In my repository, we avoid using classes for tests, but GitAuto couldn't know this without exploring other files (which we currently limit to maintain focus). This will eventually be configurable through GitAuto's homepage as a repository-specific coding style setting, for example, Prefer using classes or functions.

Iteration 3: Mocking Strategy

The third GitAuto's attempt looked promising but was missing the patch decorator:

tests/services/github/test_branch_manager.py
+
+ def test_get_default_branch_success(requests_mock):

This was actually my fault - I hadn't specified that we wanted to use patch decorators for mocking. Previous attempts included it by chance. Whether to use patch decorators or other mocking approaches is often a matter of preference and can depend on the repository's size and testing philosophy. I would go with patch decorators for this case so I added it to the requirements to the original issue.

Meeting the Requirements

The latest pull request met all our specified requirements. GitAuto even provided a clear explanation in the PR description:

## Where / How to code and why

1. Create constants.py in the tests directory: This file will store constants for future reuse, avoiding magic numbers or strings and enhancing maintainability.
2. Add new constants in tests/constants.py: Define necessary constants to be used across multiple test cases, promoting consistency and reducing duplication.
3. Implement unit tests without using classes: Keeping tests as simple functions improves readability and aligns with modern testing practices.
4. Use the @patch decorator for mocking requests: This approach isolates tests by mocking external HTTP requests, ensuring tests are fast, reliable, and do not depend on external services.

Test Execution

OK, the code looks good, so let's run the tests. Surprisingly, they failed! Wait! Looking at the error log, we see:

raise AssertionError(_error_message()) from cause
E AssertionError: expected call not found.

E Expected: get(
    url='https://api.github.com/repos/test_owner/test_repo/branches',
    headers={
      'Authorization': 'token test_token'
    },
    timeout=10
  )

E Actual: get(
    url='https://api.github.com/repos/test_owner/test_repo/branches',
    headers={
      'Accept': 'application/vnd.github.v3+json',
      'Authorization': '***',
      'User-Agent': '***',
      'X-GitHub-Api-Version': '2022-11-28'
    },
    timeout=120
  )

The issue was with the mocking implementation. GitAuto didn't properly account for our create_headers utility function, which adds additional header fields. This happened because we limited GitAuto's code exploration scope, so it didn't see the utility function's implementation.

Additionally, when I opened the code in my local IDE, I noticed a Flake8 warning:

'pytest' imported but unused by Flake8(F401)

Indeed, we're importing pytest but not using it directly. This is something we could improve by enabling Flake8 support in GitAuto's workflow.

After fixing these issues manually (see the commit), the tests passed successfully. Time to merge! You can see the complete journey in the final merged pull request.

The Bigger Picture

Can you see the potential? By creating multiple unit test tickets and assigning them to GitAuto, you can improve test coverage while only needing human engineers for review. This means you can enhance release safety without sacrificing development speed on your primary tasks.

If your team tracks Change Failure Rate (CFR), you might even see it decrease as this initiative progresses. This is particularly relevant for teams struggling with frequent bug introductions during releases - you know, the ones jokingly called "match-pump" teams because they seem to create as many problems as they solve.

This wasn't just a theoretical exercise - it's based on a real customer case. The results speak for themselves!

Want to ship 500x faster?

GitAuto is your AI coding agent that turns backlog tickets into pull requests in just 3 minutes for $10 - making it 500x faster and 99.5% cheaper.

Install GitAuto

It requires GitHub sign-in.