Complete Guide to LLM Evaluation Frameworks with Antigravity: Systematic AI Output Quality Measurement and Continuous Improvement

Setup and context — Why LLM Evaluation Has Become Mission-Critical

Building AI-powered products today requires more than just writing code. When you integrate large language models into your systems, shipping to production without a robust evaluation framework is the equivalent of deploying code without tests — and the consequences can be just as painful.

Consider a real scenario: a SaaS team switched from Gemini 1.5 Pro to Gemini 2.0 Flash to reduce costs. The switch worked — costs dropped significantly — but support tickets about poor answer quality doubled within two weeks. A well-designed evaluation pipeline would have caught this regression in staging before a single user was affected.

This guide walks you through building a complete, production-grade LLM evaluation system centered around Antigravity, combining three powerful open-source tools: promptfoo for prompt testing, LangSmith for tracing and observability, and Ragas for RAG-specific evaluation. We'll cover everything from evaluation design principles to CI/CD automation and A/B testing for model selection, with real code throughout.

Foundations: A Framework for Thinking About LLM Quality

Before diving into tools, it helps to have a mental model for the three main categories of LLM evaluation.

Reference-based evaluation works when you have ground-truth answers — golden datasets where you know what the correct response should be. Classic metrics like BLEU, ROUGE, and Exact Match fall here, though modern practice combines these with semantic similarity scoring because two responses can express the same correct answer in very different words.

Reference-free evaluation is where the LLM-as-a-Judge pattern shines. You have a high-capability model (say, Gemini 1.5 Pro or Claude 3.7 Sonnet) evaluate the outputs of your production model against a rubric. Research consistently shows that strong judge models achieve correlation coefficients of 0.7–0.9 with human evaluators, making this approach practical even when assembling large golden datasets isn't feasible.

Task-specific evaluation applies when you're building something with well-defined quality dimensions: RAG systems, code generation, summarization, and so on. For RAG, metrics like Faithfulness (does the answer stick to the retrieved context?), Answer Relevancy (does it actually address the question?), and Context Precision (how much of what you retrieved was actually useful?) map directly to what users experience.

The most important design principle: always trace your evaluation metrics back to business outcomes. High technical scores that don't correlate with user satisfaction are a warning sign that you're measuring the wrong things.

Setting Up Your Evaluation Project with Antigravity

One of the most practical ways to use Antigravity for evaluation work is to have it scaffold your project structure from a single prompt. Try something like this in the Antigravity chat:

@Workspace Set up an LLM evaluation project with:
- promptfoo for prompt regression testing
- LangSmith for production tracing
- Ragas for RAG quality metrics
- GitHub Actions for CI automation

Use Node.js + TypeScript. Include package.json, config files,
and sample test cases.

Antigravity will generate a clean project structure along these lines:

llm-eval/
├── src/
│   ├── evaluators/
│   │   ├── faithfulness.ts      # RAG faithfulness scorer
│   │   ├── relevancy.ts         # Answer relevancy scorer
│   │   └── custom-judge.ts      # LLM-as-a-Judge implementation
│   ├── datasets/
│   │   ├── qa-golden.jsonl      # Q&A golden set
│   │   └── rag-testcases.jsonl  # RAG test cases
│   └── pipelines/
│       └── evaluation-runner.ts  # Main evaluation orchestrator
├── promptfooconfig.yaml          # promptfoo configuration
├── langsmith.config.ts           # LangSmith setup
├── .github/workflows/
│   └── llm-eval.yml             # CI/CD workflow
└── package.json

The core package.json scripts:

{
  "scripts": {
    "eval:promptfoo": "promptfoo eval --config promptfooconfig.yaml",
    "eval:rag": "ts-node src/pipelines/evaluation-runner.ts --type rag",
    "eval:full": "npm run eval:promptfoo && npm run eval:rag",
    "eval:ci": "npm run eval:full -- --output json > results/eval-$(date +%Y%m%d).json"
  },
  "dependencies": {
    "promptfoo": "^0.72.0",
    "langchain": "^0.3.0",
    "@langchain/core": "^0.3.0",
    "ragas": "^0.2.0"
  },
  "devDependencies": {
    "typescript": "^5.4.0",
    "ts-node": "^10.9.0"
  }
}

Building Automated Evaluation Pipelines with promptfoo

promptfoo is purpose-built for prompt version control and automated testing. Its signature feature is matrix evaluation — testing every combination of prompt variants, models, and parameters in a single run.

Designing Your promptfooconfig.yaml

# promptfooconfig.yaml
description: "Customer support AI evaluation suite"
 
# Providers (models) to evaluate
providers:
  - id: google:gemini-2.0-flash-exp
    config:
      temperature: 0.1
  - id: google:gemini-1.5-pro
    config:
      temperature: 0.1
  - id: openai:gpt-4o-mini
    config:
      temperature: 0.1
 
# Prompt variants to compare
prompts:
  - id: system-v1
    raw: |
      You are a support agent for {{company_name}}.
      Answer user questions concisely and accurately.
 
  - id: system-v2
    raw: |
      You are a customer support specialist at {{company_name}}.
      Understand the question fully, then respond step by step.
      - Tone: friendly and professional
      - Length: 100–200 words
 
# Test cases
tests:
  - vars:
      company_name: "TechCorp"
      question: "How do I reset my password?"
    assert:
      # LLM-as-a-Judge for helpfulness
      - type: llm-rubric
        value: "Does the response include concrete steps for resetting a password that a user can actually follow?"
      # Safety check
      - type: moderation
        value: none
      # Cost gate
      - type: cost
        threshold: 0.01
 
  - vars:
      company_name: "TechCorp"
      question: "What is your refund policy?"
    assert:
      - type: llm-rubric
        value: "Does the response address the refund policy, or appropriately direct the user when that information is unavailable?"
 
# Output configuration
outputPath: "./results/promptfoo-latest.json"

Implementing a Custom LLM-as-a-Judge

// src/evaluators/custom-judge.ts
import { GoogleGenerativeAI } from "@google/generative-ai";
 
interface JudgeResult {
  score: number;     // 0.0 to 1.0
  reasoning: string;
  pass: boolean;
}
 
export async function llmJudge(
  question: string,
  answer: string,
  criteria: string
): Promise<JudgeResult> {
  const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY!);
  const model = genAI.getGenerativeModel({ model: "gemini-2.0-flash-exp" });
 
  const judgePrompt = `
You are an expert evaluator of AI output quality.
Evaluate the following response against the provided criteria.
 
Criteria: ${criteria}
 
User question: ${question}
 
Response to evaluate: ${answer}
 
Return your evaluation as JSON (no code block):
{
  "score": number between 0.0 and 1.0,
  "reasoning": "concise reasoning in under 100 words",
  "pass": true if score >= 0.7, false otherwise
}
`;
 
  const result = await model.generateContent(judgePrompt);
  const text = result.response.text().trim();
 
  try {
    return JSON.parse(text) as JudgeResult;
  } catch {
    return {
      score: 0,
      reasoning: "Failed to parse judge output",
      pass: false,
    };
  }
}

The key insight with LLM-as-a-Judge is choosing the right judge model. Use a model from a different provider than the one you're evaluating to avoid self-serving bias — models tend to rate their own outputs more favorably.

LangSmith for Production Tracing and Quality Monitoring

LangSmith is LangChain's observability and debugging platform for LLM applications. It records every AI call in production, visualizes latency, cost, and error rates, and lets you build evaluated datasets from real user interactions.

Setting Up LangSmith Tracing

// src/config/langsmith.ts
import { Client } from "langsmith";
import { traceable } from "langsmith/traceable";
 
// Required environment variables:
// LANGCHAIN_TRACING_V2=true
// LANGCHAIN_API_KEY=lsv2_...
// LANGCHAIN_PROJECT=my-ai-app-prod
 
export const langsmithClient = new Client();
 
// Wrap any AI function with traceable for automatic logging
export const tracedAnswer = traceable(
  async function generateAnswer(question: string, context: string) {
    const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY!);
    const model = genAI.getGenerativeModel({ model: "gemini-2.0-flash-exp" });
 
    const prompt = `
Use the following context to answer the question.
 
Context: ${context}
Question: ${question}
`;
 
    const result = await model.generateContent(prompt);
    return result.response.text();
  },
  { name: "generate-answer", metadata: { version: "2.1.0" } }
);

Dataset Management and Programmatic Evaluation

LangSmith's dataset management feature lets you register Q&A pairs from production and re-run evaluation automatically whenever you update your model or prompts.

// src/pipelines/langsmith-evaluation.ts
import { Client, Run } from "langsmith";
import { evaluate } from "langsmith/evaluation";
 
const client = new Client();
 
async function createEvalDataset() {
  const dataset = await client.createDataset("customer-support-eval-v2", {
    description: "Customer support AI evaluation dataset",
  });
 
  const examples = [
    {
      inputs: { question: "How do I cancel my order?" },
      outputs: { answer: "Go to My Account > Order History and cancel within 24 hours." },
    },
    {
      inputs: { question: "How long does shipping take?" },
      outputs: { answer: "Standard shipping takes 3–5 business days." },
    },
  ];
 
  await client.createExamples({
    datasetId: dataset.id,
    inputs: examples.map((e) => e.inputs),
    outputs: examples.map((e) => e.outputs),
  });
 
  return dataset;
}
 
async function runEvaluation(datasetName: string) {
  const results = await evaluate(
    async (inputs: { question: string }) => ({
      answer: await tracedAnswer(inputs.question, ""),
    }),
    {
      data: datasetName,
      evaluators: [
        async (run: Run, example: any) => ({
          key: "answer_accuracy",
          score: await scoreAccuracy(run.outputs?.answer, example.outputs?.answer),
        }),
        async (run: Run) => ({
          key: "answer_conciseness",
          score: (run.outputs?.answer?.length ?? 0) < 200 ? 1.0 : 0.5,
        }),
      ],
      experimentPrefix: `eval-${new Date().toISOString().split("T")[0]}`,
    }
  );
 
  return results;
}
 
async function scoreAccuracy(answer: string, reference: string): Promise<number> {
  const keywords = reference.toLowerCase().split(/\s+/);
  const matchCount = keywords.filter((kw) =>
    answer.toLowerCase().includes(kw)
  ).length;
  return matchCount / keywords.length;
}

Ragas for RAG-Specific Quality Evaluation

If you're building a RAG system, Ragas provides specialized metrics that track the distinct ways RAG can go wrong — hallucinating beyond the source context, retrieving irrelevant chunks, or missing critical information.

The Four Core Ragas Metrics

Faithfulness: Does the generated answer rely only on information present in the retrieved context? Score close to 1.0 means no hallucination
Answer Relevancy: How directly does the response address the user's question? Verbose, off-topic answers score lower
Context Precision: Of the context chunks retrieved, what fraction were actually relevant to generating the answer?
Context Recall: Does the retrieved context contain all the information needed to produce a correct answer?

TypeScript Orchestrator with Python Ragas Backend

// src/evaluators/ragas-evaluation.ts
import { spawn } from "child_process";
import * as fs from "fs";
import * as path from "path";
 
interface RagasInputs {
  questions: string[];
  answers: string[];
  contexts: string[][];
  groundTruths: string[];
}
 
interface RagasResults {
  faithfulness: number;
  answer_relevancy: number;
  context_precision: number;
  context_recall: number;
}
 
export async function runRagasEvaluation(
  inputs: RagasInputs
): Promise<RagasResults> {
  const inputFile = path.join("/tmp", "ragas_input.json");
  fs.writeFileSync(inputFile, JSON.stringify(inputs));
 
  return new Promise((resolve, reject) => {
    const proc = spawn("python3", ["scripts/run_ragas.py", inputFile]);
    let output = "";
 
    proc.stdout.on("data", (data) => {
      output += data.toString();
    });
 
    proc.on("close", (code) => {
      if (code === 0) {
        try {
          resolve(JSON.parse(output) as RagasResults);
        } catch {
          reject(new Error("Failed to parse Ragas output"));
        }
      } else {
        reject(new Error(`Ragas process exited with code ${code}`));
      }
    });
  });
}

# scripts/run_ragas.py
import sys
import json
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
 
def main():
    with open(sys.argv[1]) as f:
        inputs = json.load(f)
 
    dataset = Dataset.from_dict({
        "question": inputs["questions"],
        "answer": inputs["answers"],
        "contexts": inputs["contexts"],
        "ground_truth": inputs["groundTruths"],
    })
 
    result = evaluate(
        dataset,
        metrics=[
            faithfulness,
            answer_relevancy,
            context_precision,
            context_recall,
        ],
    )
 
    print(json.dumps({
        "faithfulness": float(result["faithfulness"]),
        "answer_relevancy": float(result["answer_relevancy"]),
        "context_precision": float(result["context_precision"]),
        "context_recall": float(result["context_recall"]),
    }))
 
if __name__ == "__main__":
    main()

A practical Ragas workflow: run the four core metrics weekly on a fixed sample of recent production queries. A Faithfulness score below 0.75 is a strong signal that your retrieval or prompting is causing hallucinations, and Context Recall below 0.80 suggests your chunking strategy or embedding model needs work.

CI/CD Integration with GitHub Actions

The most impactful thing you can do with your evaluation framework is gate pull requests on quality. Here's a complete GitHub Actions workflow:

# .github/workflows/llm-eval.yml
name: LLM Quality Gate
 
on:
  pull_request:
    branches: [main]
    paths:
      - "src/prompts/**"
      - "src/models/**"
 
  schedule:
    - cron: "0 2 * * *"   # Nightly regression check
 
jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 30
 
    steps:
      - uses: actions/checkout@v4
 
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
 
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: "pip"
 
      - name: Install dependencies
        run: |
          npm ci
          pip install ragas datasets
 
      - name: Run evaluation
        env:
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: npm run eval:promptfoo -- --output json > results/promptfoo.json
 
      - name: Enforce quality gates
        run: |
          node scripts/check-quality-gates.js \
            --results results/promptfoo.json \
            --min-pass-rate 0.85 \
            --max-cost-per-run 0.50
 
      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: eval-results-${{ github.sha }}
          path: results/
          retention-days: 30
 
      - name: Comment on PR
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const results = require('./results/promptfoo.json');
            const passRate = results.stats.successes / results.stats.total;
            const body = `## 🤖 LLM Evaluation Results
 
            - **Pass rate**: ${(passRate * 100).toFixed(1)}%
            - **Tests run**: ${results.stats.total}
            - **Passed**: ${results.stats.successes} / **Failed**: ${results.stats.failures}
            - **Total cost**: $${results.stats.totalCost?.toFixed(4) ?? "N/A"}
 
            ${passRate >= 0.85 ? "✅ Quality gate passed" : "❌ Quality gate failed — review prompts or model config before merging."}
            `;
 
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body,
            });

The quality gate script that enforces pass/fail:

// scripts/check-quality-gates.js
const fs = require("fs");
 
function parseArgs() {
  const args = process.argv.slice(2);
  const opts = {};
  for (let i = 0; i < args.length; i += 2) {
    opts[args[i].replace("--", "")] = args[i + 1];
  }
  return opts;
}
 
async function main() {
  const opts = parseArgs();
  const results = JSON.parse(fs.readFileSync(opts.results, "utf-8"));
 
  const passRate = results.stats.successes / results.stats.total;
  const totalCost = results.stats.totalCost ?? 0;
  const minPassRate = parseFloat(opts["min-pass-rate"] ?? "0.85");
  const maxCost = parseFloat(opts["max-cost-per-run"] ?? "1.00");
 
  console.log(`Pass rate: ${(passRate * 100).toFixed(1)}% (threshold: ${minPassRate * 100}%)`);
  console.log(`Total cost: $${totalCost.toFixed(4)} (limit: $${maxCost})`);
 
  const failures = [];
  if (passRate < minPassRate) {
    failures.push(`Pass rate ${(passRate * 100).toFixed(1)}% below threshold ${minPassRate * 100}%`);
  }
  if (totalCost > maxCost) {
    failures.push(`Cost $${totalCost.toFixed(4)} exceeds limit $${maxCost}`);
  }
 
  if (failures.length > 0) {
    console.error("❌ Quality gate failed:");
    failures.forEach((f) => console.error(`  - ${f}`));
    process.exit(1);
  }
 
  console.log("✅ All quality gates passed");
}
 
main().catch((err) => {
  console.error(err);
  process.exit(1);
});

A/B Testing for Evidence-Based Model Selection

When you're considering switching models, don't flip the switch and hope for the best. Run structured A/B tests that give you real numbers.

Metrics to Compare

Here's what you want to measure for each model candidate:

Quality score: LLM-as-a-Judge helpfulness rating (0.0–1.0)
Latency: P50, P95, P99 response times in milliseconds
Cost: USD per 1,000 requests
Error rate: API errors and timeouts as a percentage
Context utilization: Actual tokens consumed / max context length

// src/pipelines/ab-test-runner.ts
interface ModelConfig {
  name: string;
  provider: string;
  model: string;
  temperature: number;
  maxTokens: number;
}
 
interface ABTestResult {
  model: ModelConfig;
  metrics: {
    qualityScore: number;
    avgLatencyMs: number;
    p95LatencyMs: number;
    costPer1000: number;
    errorRate: number;
  };
  sampleSize: number;
}
 
export async function runABTest(
  models: ModelConfig[],
  testCases: Array<{ question: string; reference?: string }>,
  runsPerModel = 50
): Promise<ABTestResult[]> {
  const results: ABTestResult[] = [];
 
  for (const model of models) {
    const latencies: number[] = [];
    const scores: number[] = [];
    let errors = 0;
    let totalCostUSD = 0;
 
    for (let i = 0; i < runsPerModel; i++) {
      const tc = testCases[i % testCases.length];
      const start = Date.now();
 
      try {
        const answer = await callModel(model, tc.question);
        latencies.push(Date.now() - start);
 
        const score = await llmJudge(
          tc.question,
          answer,
          "Does the response directly answer the user's question accurately and concisely?"
        );
        scores.push(score.score);
        totalCostUSD += estimateCost(model.model, tc.question.length / 4, answer.length / 4);
      } catch {
        errors++;
      }
    }
 
    latencies.sort((a, b) => a - b);
 
    results.push({
      model,
      metrics: {
        qualityScore: average(scores),
        avgLatencyMs: average(latencies),
        p95LatencyMs: latencies[Math.floor(latencies.length * 0.95)],
        costPer1000: (totalCostUSD / runsPerModel) * 1000,
        errorRate: errors / runsPerModel,
      },
      sampleSize: runsPerModel,
    });
  }
 
  return results;
}
 
function average(arr: number[]): number {
  return arr.reduce((a, b) => a + b, 0) / arr.length;
}
 
function estimateCost(modelId: string, inputTokens: number, outputTokens: number): number {
  const pricing: Record<string, { input: number; output: number }> = {
    "gemini-2.0-flash-exp": { input: 0.0001, output: 0.0004 },
    "gemini-1.5-pro": { input: 0.00125, output: 0.005 },
    "gpt-4o-mini": { input: 0.00015, output: 0.0006 },
  };
  const p = pricing[modelId] ?? { input: 0.001, output: 0.002 };
  return (inputTokens * p.input + outputTokens * p.output) / 1000;
}

Interpreting A/B Test Results

Once you have your comparison data, the decision isn't always obvious. A model that scores 8% higher on quality but costs 4x more may not be the right choice for every use case. Build a decision matrix:

For consumer-facing conversational features, quality and latency matter most. For internal tooling or batch processing, cost efficiency is often the primary concern. For RAG over sensitive documents, Faithfulness should be a hard minimum threshold regardless of cost.

Antigravity is especially useful here — describe your specific use case constraints and ask it to analyze the tradeoff data. It can quickly surface which model offers the best value for your particular scenario.

Maintaining and Evolving Your Evaluation Framework Over Time

An evaluation framework that you set up and forget will become a liability rather than an asset. The golden dataset becomes stale, quality thresholds stop reflecting current product requirements, and the judge model itself may become outdated relative to the models you're evaluating. Here's how to keep your framework healthy.

Scheduled Dataset Reviews

Block one hour per month to review your evaluation dataset. Look for three types of stale cases: questions that no longer reflect how users phrase things (language and product terminology evolves), cases where the "correct" answer has changed (product policy updates, new features), and cases that are too similar to each other (dense clusters that over-represent one scenario while leaving others untested). A dataset of 200 well-curated cases beats a dataset of 2,000 redundant ones every time.

Production Log Mining

Your production system generates evaluation data continuously — you just need a process to extract it. Set up a lightweight feedback loop where user signals (thumbs up/down, escalations to human agents, session abandonment) tag the corresponding AI interactions in your logging system. Then build a weekly job that surfaces the 10–15 most interesting cases (especially failures and edge cases) for potential addition to your dataset.

Antigravity makes this straightforward: point it at your LangSmith run history and ask it to write a script that filters runs by specific outcome signals and exports them in the format your test dataset expects.

Versioning Your Evaluation Assets

Treat your promptfooconfig.yaml, dataset files, and quality gate thresholds as first-class versioned artifacts. When you adjust a threshold (say, lowering the cost gate because you've switched to a cheaper model), commit that change with a message explaining why. Six months later, when someone wonders why the cost threshold is what it is, the git history tells the story. This discipline becomes especially valuable when onboarding new team members to the evaluation system.

Periodic Judge Model Calibration

The LLM you use as a judge is not a fixed standard — it gets updated, fine-tuned, and occasionally behaves differently across API versions. Run a calibration check quarterly: have three to five humans rate the same 50 outputs, compare to your judge's scores, and recalibrate your rubric if the correlation has drifted below your acceptable threshold. Document the calibration results alongside your dataset so you have a historical record of judge reliability.

Common Pitfalls and How to Avoid Them

Ignoring evaluation costs. Using a powerful judge model for every test case can make evaluation more expensive than inference itself. A two-tier approach works well: use Gemini 2.0 Flash as your default judge, and only escalate to Gemini 1.5 Pro for the cases that are borderline or particularly high-stakes.

Golden dataset decay. A dataset created six months ago may no longer reflect current user behavior or product capabilities. Build a lightweight process to review your dataset monthly, and automatically flag cases where your model's answers have changed significantly — those are the ones worth human review.

Self-serving bias in evaluation. If you use the same provider for inference and judging, the judge will tend to favor its own family's outputs. This isn't theoretical — it's been documented in multiple studies. Always choose a judge model from a different provider than the one being evaluated.

Production vs. test distribution mismatch. If your test cases don't mirror actual user inputs, strong evaluation results become meaningless. Make it a habit to sample from production logs and add those cases to your dataset — especially the ones that generated negative feedback signals.

Setting thresholds too strict from day one. CI quality gates that fail constantly are gates that get disabled. Start with achievable thresholds (85% pass rate is often a good starting point) and tighten them incrementally as your system improves.

Summary

Building a robust LLM evaluation framework is one of the highest-leverage investments you can make as an AI product engineer. The combination of promptfoo for structured prompt testing, LangSmith for production observability, and Ragas for RAG-specific metrics covers the full spectrum of quality concerns you'll encounter.

The right mental model is to treat evaluation as a living system, not a one-time setup. Start small, instrument your production system early, and let real user interactions inform which test cases matter most. Antigravity accelerates every stage of this process — from scaffolding your initial project structure to debugging evaluation failures and generating synthetic test data.

Once evaluation is part of your development workflow, every model update, prompt iteration, and architecture change becomes a decision backed by data rather than intuition. That shift — from gut-feel to evidence-based engineering — is what separates teams that ship reliable AI products from those perpetually firefighting regressions.

For related coverage, see our AI Quality Automation Pipeline with Sentry guide and the OpenTelemetry AI Observability Pipeline for production monitoring depth. For RAG-specific implementation detail, the RAG Pipeline and Vector Search Complete Guide is a natural next read.