Error Baselines

Before generating tests, GitAuto captures all pre-existing errors in the codebase by running the relevant compiler or linter (e.g., tsc --noEmit for TypeScript). After test generation, it compares the new error output against this baseline to separate errors introduced by the generated code from errors that already existed.

Why This Exists

Many real-world codebases carry pre-existing errors - suppressed warnings, incomplete type migrations, or intentionally loose checks in legacy modules. Without a baseline, the model tries to fix every error it sees, including ones that have nothing to do with the generated tests. This wastes agent iterations on unrelated issues and often makes things worse - the model might "fix" an error in production code that was intentionally left alone, or spend its entire iteration budget chasing pre-existing issues instead of refining the test it just wrote. Error baselines let GitAuto say: "ignore these 47 errors - they existed before you started. Focus only on the 3 new ones."

Why Models Can't Distinguish Old From New

When the model runs a verification command (e.g., tsc --noEmit), it gets back a flat list of errors with no indication of which ones are pre-existing. The model sees 50 errors and tries to fix all 50 - it has no way to know that 47 of them existed before it started. The verification output treats all errors equally, so the model does too. This wastes iterations on issues the model didn't cause and often makes things worse by "fixing" intentionally loose typing in legacy code that was left that way on purpose.

How It Works

GitAuto runs the relevant compiler or linter in check-only mode before the agent begins writing code. The error output is stored as the baseline. After the model generates or modifies files, GitAuto runs the same check again and diffs the two outputs. Only errors that appear in the second run but not the first are reported to the model as actionable issues. This filtering happens at the file and line level, so even if an existing file gains new errors from the generated code, only those new errors are surfaced.

Related Features

• Type Checking - the verification step that uses the baseline for comparison
• CI Log Cleaning - cleans noise from CI logs, a related denoising technique