Antigravity × Prometheus + Grafana — Build an Application Monitoring Stack with AI Agents

Why Application Monitoring Matters

After deploying your application to production, relying on the hope that "things are probably fine" is a recipe for trouble. In practice, performance degradation, memory leaks, and unexpected error spikes often go unnoticed until users start complaining — and by then, the damage is already done.

Prometheus and Grafana together form the industry-standard solution for real-time monitoring, enabling you to detect issues early and visualize the health of your systems at a glance. However, writing configuration files, crafting alert rules, and designing dashboards has traditionally required specialized knowledge and significant effort.

This is where Antigravity's AI agents shine. From generating configuration files to crafting alert rules and building Grafana dashboard JSON definitions, you can accomplish everything through natural language instructions to the AI agent.

Understanding the Prometheus + Grafana Architecture

Before diving into implementation, let's establish a clear picture of how the components fit together.

Prometheus is a pull-based metrics collection system. Your application exposes a /metrics endpoint, and Prometheus periodically scrapes (fetches) data from it. The collected metrics are stored in a time-series database and can be queried using PromQL, a powerful query language purpose-built for metrics analysis.

Grafana is a visualization platform that turns your metrics into actionable dashboards. It connects to Prometheus as a data source and lets you build panels with graphs, gauges, tables, and more.

Here's the typical data flow:

Your application exposes metrics via a /metrics endpoint
Prometheus scrapes the endpoint at regular intervals and stores the data
Grafana queries Prometheus using PromQL to render visualizations
Alertmanager triggers notifications (Slack, email, etc.) when thresholds are breached

Setting Up the Project in Antigravity

Let's start by creating the project structure in Antigravity's terminal.

# Create the project directory
mkdir monitoring-stack && cd monitoring-stack
 
# Prepare configuration directories
mkdir -p prometheus grafana/provisioning/datasources grafana/provisioning/dashboards

Next, ask Antigravity's AI agent to generate the Docker Compose file. Here's a sample prompt:

Create a Docker Compose file with Prometheus, Grafana, and Alertmanager.
Prometheus should be accessible on localhost:9090, Grafana on localhost:3000.
Use a bridge network called monitoring.

The agent will generate a docker-compose.yml like this:

# docker-compose.yml
# Monitoring stack: Prometheus + Grafana + Alertmanager
version: "3.8"
 
services:
  prometheus:
    image: prom/prometheus:v2.53.0
    container_name: prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/alert-rules.yml:/etc/prometheus/alert-rules.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - monitoring
    restart: unless-stopped
 
  grafana:
    image: grafana/grafana:11.1.0
    container_name: grafana
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=your-secure-password
    networks:
      - monitoring
    restart: unless-stopped
 
  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"
    networks:
      - monitoring
    restart: unless-stopped
 
volumes:
  prometheus_data:
  grafana_data:
 
networks:
  monitoring:
    driver: bridge

If you're new to containerized development, check out our guide on Docker-based development environments with Antigravity to get up to speed.

Instrumenting a Node.js Application with Metrics

The next step is adding a Prometheus metrics endpoint to your application. For Node.js with Express, the prom-client library is the go-to choice.

Give Antigravity's agent a prompt like this:

Add Prometheus metrics to my Express app using prom-client.
Track total HTTP requests, response time histogram, and active connections gauge.

Here's the generated metrics module:

// src/metrics.ts
// Prometheus metrics definitions and endpoint configuration
import { Registry, Counter, Histogram, Gauge, collectDefaultMetrics } from "prom-client";
 
// Create a custom registry (can be separated from default metrics)
const register = new Registry();
 
// Automatically collect Node.js default metrics (CPU, memory, GC, etc.)
collectDefaultMetrics({ register });
 
// HTTP request counter
export const httpRequestsTotal = new Counter({
  name: "http_requests_total",
  help: "Total number of HTTP requests",
  labelNames: ["method", "route", "status_code"],
  registers: [register],
});
 
// Response time histogram (in seconds)
export const httpRequestDuration = new Histogram({
  name: "http_request_duration_seconds",
  help: "Duration of HTTP requests in seconds",
  labelNames: ["method", "route", "status_code"],
  buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 2, 5],
  registers: [register],
});
 
// Active connections gauge
export const activeConnections = new Gauge({
  name: "active_connections",
  help: "Number of active connections",
  registers: [register],
});
 
export { register };

The middleware that records metrics for each request:

// src/middleware/metricsMiddleware.ts
// Middleware to record metrics per request
import { Request, Response, NextFunction } from "express";
import {
  httpRequestsTotal,
  httpRequestDuration,
  activeConnections,
} from "../metrics";
 
export function metricsMiddleware(
  req: Request,
  res: Response,
  next: NextFunction
) {
  // Increment active connections
  activeConnections.inc();
 
  // Start response time measurement
  const end = httpRequestDuration.startTimer();
 
  res.on("finish", () => {
    // Record metrics on request completion
    const route = req.route?.path || req.path;
    const labels = {
      method: req.method,
      route,
      status_code: res.statusCode.toString(),
    };
 
    httpRequestsTotal.inc(labels);
    end(labels); // Record response time in histogram
    activeConnections.dec();
  });
 
  next();
}

// src/app.ts
// Express application entry point
import express from "express";
import { register } from "./metrics";
import { metricsMiddleware } from "./middleware/metricsMiddleware";
 
const app = express();
 
// Apply metrics middleware to all routes
app.use(metricsMiddleware);
 
// /metrics endpoint (Prometheus scrapes this)
app.get("/metrics", async (_req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});
 
// Expected output: curl http://localhost:4000/metrics
// # HELP http_requests_total Total number of HTTP requests
// # TYPE http_requests_total counter
// http_requests_total{method="GET",route="/api/users",status_code="200"} 42
// ...
 
app.listen(4000, () => {
  console.log("Server running on port 4000");
});

Configuring Prometheus Scraping

Create the Prometheus configuration file that tells it where to find your application's metrics.

# prometheus/prometheus.yml
# Prometheus global config and scrape targets
global:
  scrape_interval: 15s      # Scrape metrics every 15 seconds
  evaluation_interval: 15s   # Evaluate alert rules every 15 seconds
 
# Load alert rule files
rule_files:
  - "alert-rules.yml"
 
# Alertmanager connection
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - "alertmanager:9093"
 
# Scrape target definitions
scrape_configs:
  # Monitor Prometheus itself
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
 
  # Monitor the Node.js application
  - job_name: "node-app"
    static_configs:
      - targets: ["host.docker.internal:4000"]
    metrics_path: "/metrics"
    scrape_interval: 10s  # Scrape the app every 10 seconds

The scrape_interval controls how frequently Prometheus fetches metrics. Shorter intervals give you more real-time data but increase load, so adjust based on your application's characteristics.

Setting Up Alert Rules

Alert rules are essential for catching issues before they impact users. Ask Antigravity's agent to "create common production alert rules" and it will generate a practical rule set.

# prometheus/alert-rules.yml
# Application monitoring alert rule definitions
groups:
  - name: application-alerts
    rules:
      # High error rate detection
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status_code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))
          > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "5xx error rate exceeds 5%"
          description: "Error rate over the last 5 minutes: {{ $value | humanizePercentage }}"
 
      # Response time degradation
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
          ) > 1.0
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "P95 response time exceeds 1 second"
          description: "Current P95 latency: {{ $value }}s"
 
      # Memory usage alert
      - alert: HighMemoryUsage
        expr: |
          process_resident_memory_bytes / (1024 * 1024) > 512
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Memory usage exceeds 512MB"
          description: "Current memory usage: {{ $value }}MB"

The for parameter specifies how long a condition must persist before firing. Setting it to 2–5 minutes prevents alerts from triggering on transient spikes — a best practice you'll want to follow.

Auto-Provisioning Grafana Dashboards

Manually configuring data sources and dashboards every time Grafana starts is inefficient. Provisioning files automate this entirely.

# grafana/provisioning/datasources/prometheus.yml
# Automatic Grafana data source configuration
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

You can also have Antigravity's agent generate the dashboard JSON definition. Here's a sample prompt:

Create a Grafana dashboard JSON with these panels:
1. HTTP request rate (req/s) time series graph
2. Error rate (5xx / total) gauge
3. P50 / P95 / P99 response time time series graph
4. Active connections stat panel

A portion of the generated dashboard definition looks like this:

{
  "dashboard": {
    "title": "Application Monitoring",
    "panels": [
      {
        "title": "Request Rate (req/s)",
        "type": "timeseries",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total[1m]))",
            "legendFormat": "Total Requests/s"
          }
        ],
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }
      },
      {
        "title": "Error Rate",
        "type": "gauge",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{status_code=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "thresholds": {
              "steps": [
                { "color": "green", "value": 0 },
                { "color": "yellow", "value": 1 },
                { "color": "red", "value": 5 }
              ]
            },
            "unit": "percent"
          }
        },
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }
      }
    ]
  }
}

For more advanced observability with distributed tracing and log aggregation, see Antigravity × OpenTelemetry: Building an AI-Driven Observability Pipeline.

Launching the Stack and Verifying Everything Works

With all configuration files in place, bring up the entire monitoring stack using Docker Compose.

# Launch the monitoring stack in the background
docker compose up -d
 
# Verify container status
docker compose ps
 
# Expected output:
# NAME          IMAGE                       STATUS
# prometheus    prom/prometheus:v2.53.0     Up 10 seconds
# grafana       grafana/grafana:11.1.0      Up 10 seconds
# alertmanager  prom/alertmanager:v0.27.0   Up 10 seconds

Once everything is running, you can access each service at these URLs:

Prometheus: http://localhost:9090 — Run PromQL queries, check target status
Grafana: http://localhost:3000 — View dashboards (default user: admin)
Alertmanager: http://localhost:9093 — Review alerts, configure silences

In Prometheus, navigate to "Status > Targets" and verify that the node-app target shows an UP state. This confirms that metrics collection is working correctly.

Looking back

By leveraging Antigravity's AI agents, you can dramatically streamline the process of building a Prometheus + Grafana monitoring stack. From generating configuration files and crafting alert rules to producing Grafana dashboard JSON definitions, the AI handles the heavy lifting — making production-grade monitoring accessible even if you're not a DevOps specialist.

Monitoring isn't a nice-to-have; it's the only way to know something is wrong before your users tell you. Start small with a Docker Compose stack in your development environment, and evolve your alert rules and dashboards as you learn what matters most for your application.

If you're looking to automate monitoring deployment as part of your CI/CD pipeline, check out Antigravity × GitHub Actions Advanced CI/CD Pipeline.