Test Execution

GitAuto runs every generated test file to verify the tests actually pass before committing them to the PR. It supports multiple ecosystems: Jest/Vitest for JavaScript/TypeScript, pytest for Python, and PHPUnit for PHP. Each runner collects output, identifies failures, and feeds errors back to the model for correction.

Tests That Compile but Fail Are Useless

A test file that compiles but fails at runtime provides no value. Running the tests before committing ensures the PR contains tests that actually work.

Without execution verification, reviewers would open the PR, see green type checks, then discover the tests fail when CI runs them. This wastes the reviewer's time and requires another iteration cycle to fix runtime failures.

Why Models Write Tests That Don't Pass

Models generate tests based on what the code should do, not what it actually does. If the model misunderstands an edge case, the test will assert the wrong behavior. Models also hallucinate API shapes - calling methods that don't exist or passing wrong argument types - because they're predicting based on the function name, not reading the actual implementation carefully enough. Code generation benchmarks evaluate the implementation code, not the test code, so models are heavily trained to write correct implementations but less practiced at writing correct tests that exercise real behavior.

How It Works

GitAuto detects which test runner the project uses by checking for configuration files and dependencies. For example, for JavaScript/TypeScript it detects Jest or Vitest. It runs the generated test file with the appropriate runner, passing flags to collect coverage data (--coverage) and target only the specific test file rather than the entire test suite.

The test tsconfig is used to ensure test-specific path aliases and compiler options are respected. If tests fail, the error output is fed back to the model for correction in the next iteration. Coverage data from passing tests is forwarded to the coverage enforcement step.

Node.js Version Detection

Native Node.js addons encode their ABI version at compile time. If the build environment compiles modules with one Node version but the test runner uses a different version, tests crash with NODE_MODULE_VERSION mismatch errors.

GitAuto detects each repo's required Node.js version from .nvmrc, .node-version, or package.json engines and switches to it at runtime before executing tests. If the upfront detection misses the right version, the AI agent can also switch reactively when it encounters ABI mismatch errors during test execution.

MongoDB Binary Pre-Caching

Projects using MongoMemoryServer need a MongoDB binary to run tests. By default, MongoMemoryServer downloads this binary (~100MB) at runtime on every CI run. GitAuto pre-caches the correct binary to S3 during the build phase, so tests start immediately without a network download.

The correct binary depends on the MongoDB server version, not the MongoMemoryServer package version. MongoDB 7.0+ publishes amazon2023 builds while 6.0.x only publishes amazon2builds. GitAuto detects the repo's MongoMemoryServer version, maps it to the default MongoDB server version, selects the correct distro, and constructs the full archive filename for pre-caching.

Related Features

• Coverage Enforcement - enforces coverage targets using data collected from test execution
• Type Checking - catches type errors before tests are executed
• Pytest Support - runs pytest for Python projects with dependency caching
• Snapshot Auto-Update - handles snapshot mismatches during test execution