Giving AI Agents an Aesthetic Sense — Building a UI Quality Evaluation Pipeline with Antigravity × Gemini Vision

One of the moments I find most difficult when developing apps solo is answering the question: "Is this design good enough to ship?"

Whether a feature works can be verified with tests. Whether there are bugs can be caught in code review. But "Is this button placement intuitive?" "Is this font size readable?" "Does the overall tone match the app's concept?" — these aesthetic judgments are hard to codify, hard to test, and easy to overlook.

Starting in early 2026, I experimented with combining Antigravity's Agent Mode with Gemini Vision. What I found was that this "vague judgment" can be partially encoded into code. While we can't fully delegate subjective aesthetic decisions to AI, a system that quantitatively checks whether basic UI quality standards are met — and automatically generates improvement suggestions for problem areas — works at a practical level today.

In this article, I'm publishing the full design of the pipeline I'm actually running.

Why "Encode Aesthetic Sense into Code"?

Let me be upfront: I don't believe AI can fully replicate a human designer's sensibility. Coming from years of art practice, I hold the view that beauty is fundamentally a product of context and intention — something no algorithm can fully grasp.

But "is it beautiful?" and "is it usable?" are slightly different questions. The latter has standards that can be evaluated with reasonable universality.

Does text-to-background contrast meet WCAG 2.1 AA (4.5:1)?
Do touch targets maintain at least 44×44pt?
Does the visual flow guide eyes naturally from top-left to bottom-right?
Does critical information fit within the first viewport?

These can be quantified. Gemini Vision can read screenshots and perform pixel-level analysis. Antigravity can receive those analysis results and generate fix code.

Combining the three gives us the ability to automatically detect "passes tests but somehow feels clunky when you actually use it" problems — right in the CI/CD pipeline.

System Architecture Overview

The pipeline we're building flows like this:

[App Screen]
    ↓
[Screenshot capture via Playwright / Appium]
    ↓
[Evaluation request sent to Gemini Vision API]
    ↓
[Received as structured JSON scores]
    ↓
[Antigravity agent reads scores, generates fix code]
    ↓
[GitHub Actions auto-commits or creates PR]

Let's implement each step in order.

Step 1: Designing the Evaluation Rubric

To quantify aesthetic evaluation, we first need to articulate explicitly what we're evaluating. The rubric I use covers five axes:

// src/lib/ui-evaluator/rubric.ts
 
export type EvaluationAxis = {
  name: string;
  weight: number; // sum to 1.0
  criteria: string[];
};
 
export const UI_RUBRIC: EvaluationAxis[] = [
  {
    name: "accessibility",
    weight: 0.30,
    criteria: [
      "Does text-to-background contrast ratio meet 4.5:1 (WCAG AA)?",
      "Are interactive elements at least 44pt in size?",
      "Are focus indicators visible?",
    ],
  },
  {
    name: "visual_hierarchy",
    weight: 0.25,
    criteria: [
      "Is there a clear three-tier structure: heading / body / supplemental?",
      "Does the primary action stand out visually?",
      "Does whitespace help group information?",
    ],
  },
  {
    name: "consistency",
    weight: 0.20,
    criteria: [
      "Are the same types of elements using unified styles?",
      "Are brand colors applied consistently?",
      "Is the typographic scale unified?",
    ],
  },
  {
    name: "information_density",
    weight: 0.15,
    criteria: [
      "Is the amount of information on one screen within cognitive load limits?",
      "Does critical information fit within the first viewport?",
      "Can core functionality be understood without scrolling?",
    ],
  },
  {
    name: "motion_feedback",
    weight: 0.10,
    criteria: [
      "Is feedback for user actions clear?",
      "Are loading states displayed appropriately?",
    ],
  },
];

The key here is to convert all axes from qualitative descriptions into questions that AI can answer by looking at an image. Framing it as "is the contrast ratio sufficient?" rather than "is it beautiful?" makes Gemini Vision's job much easier and its responses more consistent.

Step 2: Automating Screenshot Capture

We use Playwright to capture screenshots of target screens. For mobile apps, Appium or the iOS Simulator screenshot API works well.

// src/lib/ui-evaluator/capture.ts
import { chromium, type Page } from "playwright";
import * as fs from "fs";
import * as path from "path";
 
export type ScreenConfig = {
  url: string;
  viewport: { width: number; height: number };
  screenshotPath: string;
  waitForSelector?: string;
  theme?: "light" | "dark";
};
 
export async function captureScreen(config: ScreenConfig): Promise<string> {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext({
    viewport: config.viewport,
    colorScheme: config.theme ?? "light",
    deviceScaleFactor: 2, // Capture at Retina resolution
  });
 
  const page: Page = await context.newPage();
  await page.goto(config.url, { waitUntil: "networkidle" });
 
  if (config.waitForSelector) {
    await page.waitForSelector(config.waitForSelector, { timeout: 5000 });
  }
 
  const dir = path.dirname(config.screenshotPath);
  if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
 
  await page.screenshot({
    path: config.screenshotPath,
    fullPage: false, // Evaluate first viewport only
  });
 
  await browser.close();
  return config.screenshotPath;
}
 
export async function captureAllScreens(
  screens: ScreenConfig[]
): Promise<string[]> {
  const results: string[] = [];
  for (const screen of screens) {
    const screenshotPath = await captureScreen(screen);
    results.push(screenshotPath);
    console.log(`✅ Captured: ${screen.url} → ${screenshotPath}`);
  }
  return results;
}

For iOS apps, here's a screenshot capture script via iOS Simulator:

#!/bin/bash
 
SIMULATOR_ID=$(xcrun simctl list --json | \
  python3 -c "import json,sys; d=json.load(sys.stdin); \
  [print(v['udid']) for runtimes in d['devices'].values() \
  for v in runtimes if v['state']=='Booted']" | head -1)
 
if [ -z "$SIMULATOR_ID" ]; then
  echo "❌ No booted simulator found"
  exit 1
fi
 
OUTPUT_DIR="./screenshots/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$OUTPUT_DIR"
 
xcrun simctl io "$SIMULATOR_ID" screenshot "$OUTPUT_DIR/screen.png"
echo "✅ Screenshot saved to $OUTPUT_DIR/screen.png"

Step 3: Building Evaluation Requests for Gemini Vision

The design of the prompt we send to Gemini Vision is the core of this pipeline. Rather than asking for general impressions, the critical thing is to make it return structured scores as JSON.

// src/lib/ui-evaluator/evaluator.ts
import { GoogleGenerativeAI } from "@google/generative-ai";
import * as fs from "fs";
import type { EvaluationAxis } from "./rubric";
import { UI_RUBRIC } from "./rubric";
 
const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY!);
 
export type AxisScore = {
  axis: string;
  score: number; // 0-10
  issues: string[];
  suggestions: string[];
};
 
export type UIEvaluationResult = {
  screenshotPath: string;
  totalScore: number; // 0-100 (weighted)
  axes: AxisScore[];
  criticalIssues: string[];
  summary: string;
  timestamp: string;
};
 
function buildEvaluationPrompt(rubric: EvaluationAxis[]): string {
  const axisDescriptions = rubric
    .map(
      (axis) =>
        `- ${axis.name} (weight: ${axis.weight}): ${axis.criteria.join(" / ")}`
    )
    .join("\n");
 
  return `
You are a UI/UX quality evaluator.
Evaluate the provided screenshot against the following rubric.
 
## Evaluation Axes and Criteria
${axisDescriptions}
 
## Output Format
Return JSON in exactly this format, without code blocks:
 
{
  "axes": [
    {
      "axis": "axis_name",
      "score": number from 0-10,
      "issues": ["issue found 1", "issue found 2"],
      "suggestions": ["specific improvement 1", "specific improvement 2"]
    }
  ],
  "criticalIssues": ["issues requiring immediate action (accessibility violations, etc.)"],
  "summary": "2-3 sentences summarizing the overall evaluation"
}
 
Evaluate objectively and technically. Minimize subjective aesthetic judgments.
Always report WCAG 2.1 accessibility standard violations as critical issues.
  `.trim();
}
 
export async function evaluateUIScreenshot(
  screenshotPath: string
): Promise<UIEvaluationResult> {
  const model = genAI.getGenerativeModel({
    model: "gemini-2.0-flash",
    generationConfig: {
      temperature: 0.1, // Low temperature for consistent scoring
    },
  });
 
  const imageBuffer = fs.readFileSync(screenshotPath);
  const base64Image = imageBuffer.toString("base64");
  const mimeType = screenshotPath.endsWith(".png") ? "image/png" : "image/jpeg";
 
  const prompt = buildEvaluationPrompt(UI_RUBRIC);
 
  const result = await model.generateContent([
    prompt,
    { inlineData: { mimeType, data: base64Image } },
  ]);
 
  const responseText = result.response.text();
 
  let parsedAxes: AxisScore[];
  let criticalIssues: string[];
  let summary: string;
 
  try {
    const parsed = JSON.parse(responseText);
    parsedAxes = parsed.axes;
    criticalIssues = parsed.criticalIssues;
    summary = parsed.summary;
  } catch (e) {
    parsedAxes = UI_RUBRIC.map((axis) => ({
      axis: axis.name,
      score: 5,
      issues: ["Failed to retrieve evaluation"],
      suggestions: [],
    }));
    criticalIssues = [];
    summary = "Failed to retrieve evaluation: " + String(e);
  }
 
  // Calculate weighted score
  const totalScore = parsedAxes.reduce((acc, axisScore) => {
    const rubricAxis = UI_RUBRIC.find((r) => r.name === axisScore.axis);
    const weight = rubricAxis?.weight ?? 0.2;
    return acc + axisScore.score * 10 * weight;
  }, 0);
 
  return {
    screenshotPath,
    totalScore: Math.round(totalScore),
    axes: parsedAxes,
    criticalIssues,
    summary,
    timestamp: new Date().toISOString(),
  };
}

Step 4: Passing Results to Antigravity to Request Fixes

This is the most interesting part of the pipeline. We build the mechanism for an Antigravity agent to read the JSON scores from Gemini Vision and actually fix the code.

Define the evaluation agent's behavior in AGENTS.md:

# AGENTS.md — UI Quality Fixer Agent Configuration
 
## Role of the ui-quality-fixer Agent
 
This agent reads `evaluation-results.json` and generates
code to fix UI/UX issues.
 
### Operating Principles
- For any axis with a score below 7.0, always make a specific fix
- Issues in criticalIssues are highest priority
- Fixes are made at component level; use minimal changes that don't affect other behavior
- Accessibility fixes must conform to WCAG 2.1 AA standards
 
### Actions After Fixing
1. Record a list of modified files in `fix-report.md`
2. Re-capture the same screen's screenshot and verify the evaluation score improved
3. If the score hasn't improved, retry with a different approach (max 3 attempts)
 
### What NOT to Do
- Change brand colors (requires verification against brand definition files)
- Make fundamental layout changes (requires design system alignment check)
- Add animations/transitions (due to performance impact)

Here's the script that passes evaluation results to the Antigravity agent:

// src/lib/ui-evaluator/fix-runner.ts
import * as fs from "fs";
import { type UIEvaluationResult } from "./evaluator";
 
export type FixRequest = {
  evaluationResult: UIEvaluationResult;
  sourceDir: string;
  targetComponent?: string;
};
 
export async function generateFixPrompt(request: FixRequest): Promise<string> {
  const { evaluationResult } = request;
 
  const lowScoreAxes = evaluationResult.axes.filter(
    (axis) => axis.score < 7.0
  );
 
  if (
    lowScoreAxes.length === 0 &&
    evaluationResult.criticalIssues.length === 0
  ) {
    return ""; // No fixes needed
  }
 
  const issueDetails = lowScoreAxes
    .map(
      (axis) =>
        `[${axis.axis} — Score: ${axis.score}/10]\n` +
        `  Issues: ${axis.issues.join(" / ")}\n` +
        `  Suggestions: ${axis.suggestions.join(" / ")}`
    )
    .join("\n\n");
 
  const criticalSection =
    evaluationResult.criticalIssues.length > 0
      ? `\n\n## ⚠️ Issues Requiring Immediate Action\n${evaluationResult.criticalIssues.map((i) => `- ${i}`).join("\n")}`
      : "";
 
  return `
Fix request from the UI quality evaluation agent.
 
## Evaluation Summary
Total Score: ${evaluationResult.totalScore}/100
Target Screen: ${evaluationResult.screenshotPath}
 
${criticalSection}
 
## Axes Requiring Improvement (Score Below 7.0)
 
${issueDetails}
 
## Target Source Directory
${request.sourceDir}
 
Please fix the issues above. Do not make changes listed under
"What NOT to Do" in AGENTS.md. Aim for maximum improvement
with minimal diff. After fixing, record changed files and
a description of changes in fix-report.md.
  `.trim();
}

Step 5: Real Before/After Results

Here's an example from applying this pipeline to one of my own apps (an affirmation iOS app).

Before

// Before: Light gray text with contrast ratio below standard
struct AffirmationCard: View {
    let text: String
    
    var body: some View {
        VStack {
            Text(text)
                .font(.body)
                .foregroundColor(Color.gray.opacity(0.6)) // ← Contrast ratio 2.1:1 (standard: 4.5:1)
                .padding()
        }
        .background(Color.white)
    }
}

Gemini Vision's evaluation result:

{
  "axes": [
    {
      "axis": "accessibility",
      "score": 3.0,
      "issues": [
        "Text '#999999' against background '#FFFFFF' has a contrast ratio of approximately 2.1:1, significantly below the WCAG AA standard of 4.5:1"
      ],
      "suggestions": [
        "Change text color to '#767676' or darker to achieve AA standard with minimal change",
        "Recommended: '#555555' achieves a contrast ratio of 7.4:1"
      ]
    }
  ],
  "criticalIssues": [
    "Accessibility violation: Main content contrast ratio does not meet WCAG 2.1 AA standard"
  ]
}

Fix code generated by the Antigravity agent (After):

// After: Automatically fixed by Antigravity agent
// Fix: Changed color to meet WCAG 2.1 AA contrast ratio (2.1:1 → 7.4:1)
struct AffirmationCard: View {
    let text: String
    
    var body: some View {
        VStack {
            Text(text)
                .font(.body)
                .foregroundColor(Color(hex: "#555555")) // ← Contrast ratio 7.4:1
                .padding()
        }
        .background(Color.white)
    }
}

The agent generated this fix about 40 seconds after reading the evaluation JSON. I didn't need to manually calculate contrast ratios or open a color picker.

Step 6: Integrating into GitHub Actions

This setup automatically runs UI evaluation whenever a PR is created, blocking the merge if scores fall below a threshold:

# .github/workflows/ui-quality-check.yml
name: UI Quality Check
 
on:
  pull_request:
    branches: [main]
    paths:
      - "src/components/**"
      - "src/screens/**"
      - "src/styles/**"
 
jobs:
  ui-evaluation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: "npm"
 
      - name: Install dependencies
        run: npm ci
 
      - name: Install Playwright browsers
        run: npx playwright install chromium
 
      - name: Start dev server
        run: npm run dev &
        env:
          PORT: 3000
 
      - name: Wait for server
        run: npx wait-on http://localhost:3000 --timeout 30000
 
      - name: Capture screenshots
        run: npx ts-node src/lib/ui-evaluator/capture-all.ts
 
      - name: Run UI evaluation
        run: npx ts-node src/lib/ui-evaluator/evaluate-all.ts
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
 
      - name: Check quality thresholds
        run: |
          node -e "
          const results = require('./evaluation-results.json');
          const failures = results.filter(r => r.totalScore < 65);
          const criticals = results.filter(r => r.criticalIssues.length > 0);
          
          if (criticals.length > 0) {
            console.error('❌ Critical accessibility issues found:');
            criticals.forEach(r => r.criticalIssues.forEach(i => console.error('  -', i)));
            process.exit(1);
          }
          
          if (failures.length > 0) {
            console.warn('⚠️ Low quality scores:');
            failures.forEach(r => console.warn(\`  \${r.screenshotPath}: \${r.totalScore}/100\`));
          }
          
          console.log('✅ UI quality check passed');
          "
 
      - name: Upload evaluation report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: ui-evaluation-report
          path: |
            evaluation-results.json
            screenshots/

What Three Months of Running This Pipeline Taught Me

After running this in production, I discovered a few things you won't find in official documentation or most technical articles.

1. There are evaluation axes that Gemini Vision struggles with

"Information density" evaluations are relatively stable, but "consistency" evaluations show high variance. Sending the same screenshot three times can yield scores anywhere from 6 to 9. Quantifiable axes like contrast ratio are stable, but context-dependent judgments like "does this feel coherent?" are subject to model sampling variability.

My fix: set temperature to 0.1 or lower, and evaluate each screen twice and average the results.

2. Costs accumulate quickly in CI

Gemini Vision has low per-request pricing, but if you're evaluating 20 screens per PR and running 50 PRs per day, that's 1,000 requests/day — a non-trivial monthly cost. I added paths: filters so the job only runs when UI-affecting files change.

3. Agents sometimes over-fix

Early on, with a loose "what NOT to do" definition in AGENTS.md, the agent would occasionally change brand colors or restructure layouts significantly. Explicitly stating "minimum changes for maximum improvement" in AGENTS.md and limiting change scope to the component level resolved this almost entirely.

Can AI Develop an Aesthetic Sense?

Let me step back for a moment before closing.

Building this pipeline meant continually confronting the reality that "AI can evaluate UI." I was skeptical at first. Coming from a background in art, I felt that aesthetic judgment lives in context and intention — things no algorithm can fully contain.

After running this in production, what I've come to feel is that a natural division of labor has emerged: AI judges quality based on rules; humans judge beauty based on intent.

Whether a contrast ratio meets the standard is something AI can determine more accurately and quickly than I can. But "does this whitespace fit the world this app is trying to create?" — that's still a question only a human can answer.

By delegating what AI can handle, we gain more time to focus on the questions humans should be asking. That's the nature of this pipeline, as I understand it.

I hope this approach is useful to anyone building products they care about making well.