Building an AI Test Pipeline with Antigravity Agents: Automating Quality Assurance in Production

"I don't have time to write tests" — whether you're a solo developer or part of a team, this is a pain point that never goes away. Every new feature pushes test coverage further behind, and you end up deploying with a knot in your stomach, hoping nothing breaks.

When I started using Antigravity's agent features, my entire approach to testing shifted. Instead of asking the AI to write individual tests, I switched to having the AI design the test pipeline itself. The result? Coverage jumped from 30% to 85%, and the maintenance cost barely increased.

What follows is the exact approach I used to build that production-grade test pipeline from scratch with Antigravity's multi-agent system, including the full implementation code.

Why AI-Generated Tests Often Feel "Off"

If you've asked an AI to write tests and thought "these aren't great," you're not alone. After analyzing the patterns, I found three root causes.

1. Insufficient context leads to shallow tests

The AI reads the surface structure of your code but doesn't grasp the business logic intent. For an authentication function, it might test "throws an exception when null is passed" but won't reach "refreshes via refresh token after JWT expiration."

2. Test interdependencies are ignored

AI tends to generate each test case independently. In production, however, execution order and database state management determine real quality.

3. Nobody validates the tests themselves

This is the most critical issue. Even when AI-generated tests pass, whether the tests themselves are correct is a separate question. Assertions that are too lenient don't catch bugs — they create false confidence.

To address all three, I structured Antigravity's agents into a three-layer architecture.

Three-Layer Test Agent Architecture

The pipeline separates testing into three agents — Generation, Validation, and Execution — each with distinct responsibilities. By having them verify each other's output, you catch mistakes that any single AI would miss.

// agents/test-pipeline/orchestrator.ts
// Orchestrator that coordinates the entire test pipeline
 
interface TestPipelineConfig {
  targetFiles: string[];
  testFramework: 'vitest' | 'jest';
  coverageThreshold: number;
  maxRetries: number;
}
 
interface PipelineResult {
  generated: number;
  validated: number;
  passed: number;
  failed: number;
  coverageDelta: number;
}
 
async function runTestPipeline(
  config: TestPipelineConfig
): Promise<PipelineResult> {
  const results: PipelineResult = {
    generated: 0,
    validated: 0,
    passed: 0,
    failed: 0,
    coverageDelta: 0,
  };
 
  for (const file of config.targetFiles) {
    // Phase 1: Test generation agent
    const generatedTests = await generateTestAgent(file, {
      framework: config.testFramework,
      includeEdgeCases: true,
      contextDepth: 3, // Trace callers up to 3 levels
    });
    results.generated += generatedTests.length;
 
    // Phase 2: Test validation agent
    const validatedTests = await validateTestAgent(generatedTests, {
      checkAssertionStrength: true,
      checkMockAccuracy: true,
      checkBoundaryConditions: true,
    });
    results.validated += validatedTests.length;
 
    // Phase 3: Test execution with feedback loop
    const executionResult = await executeTestAgent(validatedTests, {
      retries: config.maxRetries,
      fixOnFailure: true,
    });
    results.passed += executionResult.passed;
    results.failed += executionResult.failed;
  }
 
  return results;
}

The key is contextDepth: 3. Instead of only reading the target file, the agent also ingests the three levels of code that call the target. This gives the AI enough understanding of "how this function is actually used" to generate meaningful tests.

Implementing the Test Generation Agent

Here's the concrete implementation of the agent that generates tests. It leverages Antigravity's agents.md for fine-grained behavior control.

// agents/test-pipeline/generator.ts
// Test generation agent: analyzes source code and auto-generates test cases
 
import { readFileSync, existsSync } from 'fs';
import { resolve, dirname } from 'path';
 
interface TestCase {
  name: string;
  type: 'unit' | 'integration' | 'e2e';
  code: string;
  targetFunction: string;
  assertionCount: number;
}
 
interface GeneratorOptions {
  framework: 'vitest' | 'jest';
  includeEdgeCases: boolean;
  contextDepth: number;
}
 
async function generateTestAgent(
  filePath: string,
  options: GeneratorOptions
): Promise<TestCase[]> {
  // Read source code and analyze dependencies
  const sourceCode = readFileSync(filePath, 'utf-8');
  const imports = extractImports(sourceCode);
  const functions = extractExportedFunctions(sourceCode);
 
  // Collect context by recursively tracing callers
  const context = await collectContext(filePath, options.contextDepth);
 
  const testCases: TestCase[] = [];
 
  for (const fn of functions) {
    // Happy path tests
    const happyPath = await generateHappyPathTest(fn, context, options);
    testCases.push(happyPath);
 
    // Error case tests
    const errorCases = await generateErrorCases(fn, context, options);
    testCases.push(...errorCases);
 
    // Edge cases (optional)
    if (options.includeEdgeCases) {
      const edgeCases = await generateEdgeCases(fn, context, options);
      testCases.push(...edgeCases);
    }
  }
 
  return testCases;
}
 
// Recursively trace callers to collect context
async function collectContext(
  filePath: string,
  depth: number,
  visited: Set<string> = new Set()
): Promise<string[]> {
  if (depth <= 0 || visited.has(filePath)) return [];
  visited.add(filePath);
 
  const contextParts: string[] = [];
  const importers = await findImporters(filePath);
 
  for (const importer of importers) {
    const importerCode = readFileSync(importer, 'utf-8');
    contextParts.push(
      `// Context from: ${importer}\n${importerCode}`
    );
 
    const upperContext = await collectContext(
      importer,
      depth - 1,
      visited
    );
    contextParts.push(...upperContext);
  }
 
  return contextParts;
}
 
function extractImports(code: string): string[] {
  const importRegex = /import\s+.*?\s+from\s+['"](.+?)['"]/g;
  const matches: string[] = [];
  let match: RegExpExecArray | null;
  while ((match = importRegex.exec(code)) \!== null) {
    matches.push(match[1]);
  }
  return matches;
}
 
function extractExportedFunctions(code: string): string[] {
  const fnRegex = /export\s+(?:async\s+)?function\s+(\w+)/g;
  const matches: string[] = [];
  let match: RegExpExecArray | null;
  while ((match = fnRegex.exec(code)) \!== null) {
    matches.push(match[1]);
  }
  return matches;
}

The collectContext function is the linchpin. It recursively finds every file that imports the target and feeds that information to the agent. This alone makes a dramatic difference in how relevant the generated tests are.

The Validation Agent: AI Checking AI's Work

Not blindly trusting generated tests is the key to overall pipeline reliability. The validation agent evaluates tests from three angles.

// agents/test-pipeline/validator.ts
// Test validation agent: multi-dimensional quality checks on generated tests
 
interface ValidationResult {
  testCase: TestCase;
  isValid: boolean;
  score: number;         // Score from 0-100
  issues: ValidationIssue[];
}
 
interface ValidationIssue {
  type: 'weak-assertion' | 'missing-mock' | 'no-boundary' | 'redundant';
  severity: 'error' | 'warning';
  message: string;
  suggestion: string;
}
 
async function validateTestAgent(
  testCases: TestCase[],
  options: {
    checkAssertionStrength: boolean;
    checkMockAccuracy: boolean;
    checkBoundaryConditions: boolean;
  }
): Promise<TestCase[]> {
  const validatedTests: TestCase[] = [];
 
  for (const testCase of testCases) {
    const issues: ValidationIssue[] = [];
 
    // 1. Assertion strength check
    if (options.checkAssertionStrength) {
      const assertionIssues = checkAssertionStrength(testCase);
      issues.push(...assertionIssues);
    }
 
    // 2. Mock accuracy check
    if (options.checkMockAccuracy) {
      const mockIssues = checkMockAccuracy(testCase);
      issues.push(...mockIssues);
    }
 
    // 3. Boundary condition check
    if (options.checkBoundaryConditions) {
      const boundaryIssues = checkBoundaryConditions(testCase);
      issues.push(...boundaryIssues);
    }
 
    const errorCount = issues.filter(i => i.severity === 'error').length;
 
    if (errorCount === 0) {
      validatedTests.push(testCase);
    } else {
      // Attempt to fix problematic tests
      const fixedTest = await attemptFix(testCase, issues);
      if (fixedTest) {
        validatedTests.push(fixedTest);
      }
      // Unfixable tests are discarded — never compromise on quality
    }
  }
 
  return validatedTests;
}
 
// Check if assertions are too lenient
function checkAssertionStrength(testCase: TestCase): ValidationIssue[] {
  const issues: ValidationIssue[] = [];
  const code = testCase.code;
 
  // Tests relying only on toBeTruthy()/toBeFalsy() are weak
  const weakAssertions = code.match(/\.toBeTruthy\(\)|\.toBeFalsy\(\)/g);
  if (weakAssertions && weakAssertions.length > 0) {
    issues.push({
      type: 'weak-assertion',
      severity: 'warning',
      message: `toBeTruthy/toBeFalsy used in ${weakAssertions.length} places`,
      suggestion: 'Use toEqual() or toMatchObject() to verify specific values',
    });
  }
 
  // Too few expect() calls for the test case
  const expectCount = (code.match(/expect\(/g) || []).length;
  if (expectCount < 2) {
    issues.push({
      type: 'weak-assertion',
      severity: 'error',
      message: `Only ${expectCount} assertion(s) found`,
      suggestion: 'Verify side effects (DB writes, API calls) in addition to return values',
    });
  }
 
  return issues;
}
 
// Check that mocks match the real behavior
function checkMockAccuracy(testCase: TestCase): ValidationIssue[] {
  const issues: ValidationIssue[] = [];
  const code = testCase.code;
 
  // Empty vi.fn() mocks that return undefined
  const emptyMocks = code.match(/vi\.fn\(\)/g);
  if (emptyMocks && emptyMocks.length > 2) {
    issues.push({
      type: 'missing-mock',
      severity: 'warning',
      message: `${emptyMocks.length} empty vi.fn() mocks found`,
      suggestion: 'Use mockResolvedValue() or mockImplementation() to reproduce actual response shapes',
    });
  }
 
  return issues;
}

Here are the most common problems this validator catches:

Common mistake #1: Overusing toBeTruthy()

AI prioritizes "making the test pass," so it tends to use lenient assertions. expect(result).toBeTruthy() passes even when result is 0 or "". The validator flags these and suggests replacing them with toEqual() or toMatchObject().

Common mistake #2: Mass-producing empty mocks

Mocks created with bare vi.fn() pass type checks but may not match the actual API response structure. Tests go green, but in production an unexpected missing property causes a crash.

Common mistake #3: Only testing the happy path

AI excels at testing "code that works" but struggles with "code that breaks." Network errors, timeouts, invalid input — without deliberately adding these scenarios, your resilience in production won't improve.

The Execution Agent: Self-Healing Test Loop

The third layer runs validated tests and automatically repairs failures.

// agents/test-pipeline/executor.ts
// Test execution agent: run, analyze failures, and auto-repair loop
 
import { exec } from 'child_process';
import { promisify } from 'util';
 
const execAsync = promisify(exec);
 
interface ExecutionResult {
  passed: number;
  failed: number;
  fixedOnRetry: number;
  unfixable: string[];
}
 
async function executeTestAgent(
  testCases: TestCase[],
  options: {
    retries: number;
    fixOnFailure: boolean;
  }
): Promise<ExecutionResult> {
  const result: ExecutionResult = {
    passed: 0,
    failed: 0,
    fixedOnRetry: 0,
    unfixable: [],
  };
 
  for (const testCase of testCases) {
    let attempts = 0;
    let currentTest = testCase;
    let success = false;
 
    while (attempts < options.retries && \!success) {
      attempts++;
 
      try {
        const tempPath = writeTempTestFile(currentTest);
        const { stdout, stderr } = await execAsync(
          `npx vitest run ${tempPath} --reporter=json`,
          { timeout: 30000 }
        );
 
        const testResult = JSON.parse(stdout);
 
        if (testResult.numFailedTests === 0) {
          success = true;
          result.passed++;
          if (attempts > 1) {
            result.fixedOnRetry++;
          }
        } else if (options.fixOnFailure) {
          // Analyze failure and attempt repair
          const failureReason = extractFailureReason(testResult);
          currentTest = await repairTest(currentTest, failureReason);
        }
      } catch (error) {
        if (attempts >= options.retries) {
          result.failed++;
          result.unfixable.push(currentTest.name);
        }
      }
    }
 
    if (\!success) {
      result.failed++;
    }
  }
 
  return result;
}
 
// Structure the failure reason for analysis
function extractFailureReason(
  testResult: Record<string, unknown>
): FailureReason {
  const failures = (testResult as any).testResults
    ?.flatMap((r: any) => r.assertionResults)
    ?.filter((a: any) => a.status === 'failed') ?? [];
 
  if (failures.length === 0) {
    return { type: 'unknown', message: 'No assertion failures found' };
  }
 
  const firstFailure = failures[0];
  const message = firstFailure.failureMessages?.[0] ?? '';
 
  // Classify failure patterns
  if (message.includes('TypeError')) {
    return { type: 'type-error', message, hint: 'Mock type may not match implementation' };
  }
  if (message.includes('toEqual')) {
    return { type: 'value-mismatch', message, hint: 'Expected and actual values differ' };
  }
  if (message.includes('timeout')) {
    return { type: 'timeout', message, hint: 'Async timeout — possible missing await' };
  }
 
  return { type: 'assertion-failed', message, hint: 'Check assertion conditions' };
}
 
interface FailureReason {
  type: string;
  message: string;
  hint?: string;
}

The self-healing loop is surprisingly effective. In my projects, about 60% of initially failing tests were fixed on the second retry. The most common culprits are missing await on async operations and type mismatches in mock return values.

CI/CD Integration

Now let's wire these agents into GitHub Actions so tests are auto-generated and validated on every pull request.

# .github/workflows/ai-test-pipeline.yml
# AI Test Pipeline: auto-generate and run tests on every PR
 
name: AI Test Pipeline
on:
  pull_request:
    types: [opened, synchronize]
    paths:
      - 'src/**/*.ts'
      - 'src/**/*.tsx'
 
jobs:
  ai-test-generation:
    runs-on: ubuntu-latest
    timeout-minutes: 15
 
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for accurate diff
 
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
 
      - run: npm ci
 
      # Identify changed files
      - name: Detect changed files
        id: changes
        run: |
          CHANGED=$(git diff --name-only origin/main...HEAD -- 'src/**/*.ts' 'src/**/*.tsx' | grep -v '\.test\.' | grep -v '\.spec\.')
          echo "files<<EOF" >> $GITHUB_OUTPUT
          echo "$CHANGED" >> $GITHUB_OUTPUT
          echo "EOF" >> $GITHUB_OUTPUT
 
      # Run the AI test generation pipeline
      - name: Run AI Test Pipeline
        run: |
          npx tsx scripts/run-test-pipeline.ts \
            --files "${{ steps.changes.outputs.files }}" \
            --framework vitest \
            --coverage-threshold 80 \
            --max-retries 3
        env:
          ANTIGRAVITY_API_KEY: ${{ secrets.ANTIGRAVITY_API_KEY }}
 
      # Commit generated tests
      - name: Commit generated tests
        run: |
          git config user.name "AI Test Bot"
          git config user.email "ai-test@example.com"
          git add 'src/**/*.test.ts'
          git diff --cached --quiet || git commit -m "test: AI generated tests for changed files"
          git push
 
      # Coverage report
      - name: Run coverage check
        run: npx vitest run --coverage --reporter=json

One important detail: fetch-depth: 0 is specified to get the full git history. This is needed so git diff can accurately detect which files changed. With a shallow clone, the diff would fail and every file would be treated as changed.

Controlling Agent Behavior with agents.md

To keep all three agents consistent, define explicit rules in agents.md.

<\!-- .antigravity/agents.md -->
# Test Pipeline Agent Configuration
 
## Test Generation Agent
- Always reference caller context, not just the target function
- Include happy path, error cases, and edge cases for every function
- Minimum 2 expect() calls per test case
- When mocking, use values that match the real response types
- Name tests descriptively so failures are immediately understandable
 
## Test Validation Agent
- Flag standalone toBeTruthy()/toBeFalsy() assertions as warnings
- Flag more than 3 empty vi.fn() mocks as warnings
- Flag async function tests missing await as errors
- Flag tests that depend on other tests as errors
 
## Test Execution Agent
- Maximum 3 retries per test
- Treat TypeError as "mock type mismatch" and attempt auto-repair
- For timeout errors, check for missing await first
- Discard tests that fail after 3 retries, log the reason

These rules pay off constantly. For instance, the "descriptive test names" rule means that when a test fails, you immediately know what broke. Instead of should return correct value, you see User auth: refresh triggers after token expiration — much faster to debug.

Extending to E2E Tests with Playwright

Let's add E2E test generation using Antigravity's browser agent combined with Playwright.

// agents/test-pipeline/e2e-generator.ts
// E2E test generation agent: builds tests from user flow definitions
 
interface UserFlow {
  name: string;
  steps: FlowStep[];
  criticalPath: boolean;
}
 
interface FlowStep {
  action: 'navigate' | 'click' | 'type' | 'wait' | 'assert';
  target?: string;
  value?: string;
  timeout?: number;
}
 
function generatePlaywrightTest(flow: UserFlow): string {
  const steps = flow.steps.map(step => {
    switch (step.action) {
      case 'navigate':
        return `  await page.goto('${step.value}');`;
      case 'click':
        return `  await page.getByRole('button', { name: '${step.target}' }).click();`;
      case 'type':
        return `  await page.getByLabel('${step.target}').fill('${step.value}');`;
      case 'wait':
        return `  await page.waitForResponse(resp => resp.url().includes('${step.value}'));`;
      case 'assert':
        return `  await expect(page.getByText('${step.value}')).toBeVisible();`;
      default:
        return `  // Unknown action: ${step.action}`;
    }
  }).join('\n');
 
  return `
import { test, expect } from '@playwright/test';
 
test('${flow.name}', async ({ page }) => {
${steps}
});
`.trim();
}
 
// Example: login flow E2E test
const loginFlow: UserFlow = {
  name: 'User login: authenticate with email and password',
  steps: [
    { action: 'navigate', value: '/login' },
    { action: 'type', target: 'Email', value: 'test@example.com' },
    { action: 'type', target: 'Password', value: 'SecurePassword123\!' },
    { action: 'click', target: 'Sign in' },
    { action: 'wait', value: '/api/auth' },
    { action: 'assert', value: 'Dashboard' },
  ],
  criticalPath: true,
};
 
console.log(generatePlaywrightTest(loginFlow));
// Expected output:
// import { test, expect } from '@playwright/test';
//
// test('User login: authenticate with email and password', async ({ page }) => {
//   await page.goto('/login');
//   await page.getByLabel('Email').fill('test@example.com');
//   ...
// });

The E2E generator is designed to prefer getByRole and getByLabel over CSS selectors. Selector-dependent tests break whenever you refactor the UI. Following Playwright's recommended practices in AI-generated tests dramatically reduces maintenance costs.

Coverage Tracking and Feedback

To continuously measure the pipeline's impact, track coverage trends over time.

// scripts/coverage-tracker.ts
// Utility to record and visualize coverage trends
 
import { readFileSync, writeFileSync, existsSync } from 'fs';
 
interface CoverageEntry {
  date: string;
  commit: string;
  statements: number;
  branches: number;
  functions: number;
  lines: number;
  aiGenerated: number;
  manuallyWritten: number;
}
 
const HISTORY_FILE = '.coverage-history.json';
 
function recordCoverage(entry: CoverageEntry): void {
  const history: CoverageEntry[] = existsSync(HISTORY_FILE)
    ? JSON.parse(readFileSync(HISTORY_FILE, 'utf-8'))
    : [];
 
  history.push(entry);
 
  // Keep only the last 90 days
  const cutoff = new Date();
  cutoff.setDate(cutoff.getDate() - 90);
  const trimmed = history.filter(
    e => new Date(e.date) >= cutoff
  );
 
  writeFileSync(HISTORY_FILE, JSON.stringify(trimmed, null, 2));
}
 
function generateReport(history: CoverageEntry[]): string {
  if (history.length < 2) return 'Insufficient history';
 
  const latest = history[history.length - 1];
  const previous = history[history.length - 2];
  const delta = latest.lines - previous.lines;
  const direction = delta >= 0 ? '↑' : '↓';
 
  return [
    `## Coverage Report (${latest.date})`,
    '',
    `| Metric | Value | Change |`,
    `|--------|-------|--------|`,
    `| Statements | ${latest.statements}% | ${direction} |`,
    `| Branches | ${latest.branches}% | ${direction} |`,
    `| Functions | ${latest.functions}% | ${direction} |`,
    `| Lines | ${latest.lines}% | ${delta >= 0 ? '+' : ''}${delta}% |`,
    '',
    `AI-generated tests: ${latest.aiGenerated} / Manual tests: ${latest.manuallyWritten}`,
    `AI ratio: ${Math.round(latest.aiGenerated / (latest.aiGenerated + latest.manuallyWritten) * 100)}%`,
  ].join('\n');
}

In my projects, coverage climbed from 30% to 85% over three months, with AI-generated tests making up 72% of the total. The only tests I write manually are for complex business logic branching.

Common Pitfalls and How to Handle Them

Here are the pitfalls I encountered while operating this pipeline.

Pitfall #1: The AI tries to modify source code to make tests pass

Antigravity's agents are smart enough to realize they could change the implementation to fix a test. Add "Never modify source code during test repair" explicitly to agents.md.

Pitfall #2: Over-mocking removes all meaning from the test

For functions with many dependencies, the AI tends to mock everything. The result is "a test of whether the mocks work correctly" rather than a test of the actual function. I added a rule to the validator: "If mock count exceeds 5, escalate to an integration test."

Pitfall #3: Race conditions in async tests

The AI correctly inserts await for individual promises but doesn't account for race conditions when multiple async operations run in parallel — especially with Promise.all. This creates flaky tests that pass sometimes and fail others. I added conditional vi.useFakeTimers() injection to the execution agent to handle this.

Pitfall #4: Duplicate test names

When generating tests in bulk, names collide and Vitest warns about it. Enforcing a "function name + scenario + sequence number" format in the generation agent eliminates the issue.

Gradual Adoption Roadmap

You don't need to deploy the full pipeline at once. A phased approach works much better.

Weeks 1-2: Generation agent only

Run the agent manually and review generated tests yourself. Don't integrate with CI yet — just establish a quality baseline.

Weeks 3-4: Add the validation agent

Once you're comfortable with generated test quality, add the validator. This is where you get quantitative data on how many weak assertions and empty mocks exist.

Weeks 5-8: CI/CD integration

When the validator accuracy meets your standards, wire it into GitHub Actions. Start with "generate only, don't execute" mode, then enable auto-execution once stable.

This gradual approach prevents the dreaded "AI-generated tests broke CI and nobody noticed for a week" scenario.

What to Do Next

Start by picking the single file in your project with the worst test coverage and run the generation agent manually. Even adding just one rule to agents.md will show you the difference in output quality.

Test automation is hardest at the beginning. Once you cross that initial threshold, it accelerates rapidly. AI isn't great at "writing tests" per se — it's great at "learning the criteria for what makes a good test." Give it good rules and it returns proportionally good results.

For more advanced patterns like mutation testing and property-based testing with Antigravity, check out AI Test Generation Basics and TDD Mastery.