Articles/Integrations

⬡ Integrations/2026-03-27Advanced

Assembling Observability with Antigravity × OpenTelemetry — Instrumentation, Sampling, Cardinality, and AI Anomaly Detection in Production

A record of running OpenTelemetry in production on a Node.js backend: measured instrumentation overhead, tail sampling configuration, avoiding cardinality explosion, and operating AI anomaly detection behind an approval gate.

antigravity⁴⁶¹opentelemetryobservability¹⁹ monitoring⁵ tracing³ metrics⁴ production⁷¹

✦ Premium Article

The logs at 2 a.m. told me nothing

The API I run as an indie developer went down at two in the morning.

I opened the logs and found exactly one useful fact: a 500 had been returned. Which request, which downstream dependency, how long it had waited before giving up — none of that was there. It took me three hours to find the cause, and the cause turned out to be embarrassingly simple: a timeout on an external API set far too generously.

Those three hours were interest, charged by the absence of monitoring.

Observability isn't logging. It's the state of being able to ask a question after the incident and get an answer. A trace tells you the path a request took. Metrics tell you the system's temperature. Logs tell you the particulars. Only when all three sit on the same timeline does the question become answerable at all.

OpenTelemetry is the shared language for that. The catch is the setup cost — instrumenting code, configuring exporters, wiring backends. Done by hand, a few days disappear before you see a single span. Antigravity's agents take over that assembly work.

What follows is the configuration I arrived at after putting OpenTelemetry into a Node.js/TypeScript backend and actually running it in production, along with a record of the places it broke. Not just how to add instrumentation — what happens after you do.

Target audience: Backend engineers with Node.js/TypeScript experience who want to strengthen production monitoring.

The Three Pillars of OpenTelemetry and Antigravity's Role

OpenTelemetry (OTel) is a CNCF open standard that provides a unified approach to traces, metrics, and logs. Antigravity's agents serve as a powerful automation layer for instrumenting all three pillars.

Traces (Distributed Tracing)

Traces visualize how requests flow through microservices. They're essential for identifying processing bottlenecks and understanding service dependencies.

Metrics

Metrics capture numerical data over time — CPU usage, memory consumption, request rates, error rates. They form the foundation for alerting thresholds and capacity planning.

Logs

Logs provide detailed records of application behavior. When correlated with trace IDs, you can filter logs to a specific request's journey across your entire system.

Antigravity's agents analyze your codebase to suggest optimal instrumentation points and auto-generate OpenTelemetry SDK configuration. This ensures comprehensive coverage of endpoints and database queries that manual instrumentation often misses.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Measured auto-instrumentation overhead (p99 up ~1.9x, throughput down 24%), the tail-sampling config that recovers it, and the measured decision_wait/memory trade-off

✦A real cardinality explosion (42,000 series down to 380), plus the spanmetrics-after-sampling mistake that made request rates read 20x low, and the pipeline order that fixes it

✦A delegation table for agent vs. human decisions, and why auto-remediation stays approval-gated

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Environment Setup: OpenTelemetry SDK Configuration

Start by having Antigravity's agent analyze your project structure, then install the OpenTelemetry dependencies.

# Run in Antigravity's terminal
# Have the agent analyze your project structure
@agents analyze the project structure and identify all HTTP endpoints, 
database queries, and external API calls that need instrumentation
 
# Install OpenTelemetry packages
npm install @opentelemetry/sdk-node \
  @opentelemetry/api \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/exporter-metrics-otlp-http \
  @opentelemetry/sdk-metrics \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions

Next, have the Antigravity agent generate the OTel SDK initialization code.

// src/instrumentation.ts
// OTel initialization code generated by the Antigravity agent
 
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';
 
const resource = new Resource({
  [ATTR_SERVICE_NAME]: process.env.OTEL_SERVICE_NAME || 'my-api-service',
  [ATTR_SERVICE_VERSION]: process.env.npm_package_version || '1.0.0',
  'deployment.environment': process.env.NODE_ENV || 'development',
});
 
const traceExporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT + '/v1/traces',
});
 
const metricExporter = new OTLPMetricExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT + '/v1/metrics',
});
 
const sdk = new NodeSDK({
  resource,
  traceExporter,
  metricReader: new PeriodicExportingMetricReader({
    exporter: metricExporter,
    exportIntervalMillis: 30000, // Export metrics every 30 seconds
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingPaths: ['/health', '/ready'], // Exclude health checks
      },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-pg': { enabled: true },
      '@opentelemetry/instrumentation-redis': { enabled: true },
    }),
  ],
});
 
sdk.start();
 
// Graceful shutdown
process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('OTel SDK shut down successfully'))
    .catch((err) => console.error('Error shutting down OTel SDK', err))
    .finally(() => process.exit(0));
});
 
export default sdk;

Load this file before your application entry point to activate auto-instrumentation.

// Update scripts in package.json
{
  "scripts": {
    "start": "node --require ./src/instrumentation.ts src/index.ts",
    "dev": "tsx --require ./src/instrumentation.ts src/index.ts"
  }
}

Implementing Custom Spans and Metrics

Auto-instrumentation covers standard operations, but business-critical logic requires custom spans and metrics. Ask the Antigravity agent to "add custom instrumentation to this service's critical business processes" and it will insert code at the appropriate points.

// src/services/payment.service.ts
// Custom spans and metrics for payment processing
 
import { trace, metrics, SpanStatusCode } from '@opentelemetry/api';
 
const tracer = trace.getTracer('payment-service');
const meter = metrics.getMeter('payment-service');
 
// Define custom metrics
const paymentCounter = meter.createCounter('payments.processed', {
  description: 'Total number of processed payments',
  unit: '1',
});
 
const paymentDuration = meter.createHistogram('payments.duration', {
  description: 'Time taken to process payments',
  unit: 'ms',
});
 
const activePayments = meter.createUpDownCounter('payments.active', {
  description: 'Number of currently active payment processes',
});
 
export async function processPayment(
  userId: string,
  amount: number,
  currency: string
): Promise<PaymentResult> {
  // Start a custom span
  return tracer.startActiveSpan('payment.process', async (span) => {
    const startTime = Date.now();
    activePayments.add(1);
 
    try {
      // Add attributes to the span (searchable in traces)
      span.setAttributes({
        'payment.user_id': userId,
        'payment.amount': amount,
        'payment.currency': currency,
      });
 
      // Stripe API call (auto-instrumented as HTTP span)
      const result = await stripe.paymentIntents.create({
        amount: Math.round(amount * 100),
        currency,
        metadata: { userId },
      });
 
      // Record success metrics
      paymentCounter.add(1, {
        status: 'success',
        currency,
      });
 
      span.setStatus({ code: SpanStatusCode.OK });
      return { success: true, intentId: result.id };
 
    } catch (error) {
      // Record error metrics
      paymentCounter.add(1, {
        status: 'error',
        currency,
        error_type: error instanceof Error ? error.name : 'unknown',
      });
 
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error instanceof Error ? error.message : 'Payment failed',
      });
      span.recordException(error as Error);
 
      throw error;
    } finally {
      activePayments.add(-1);
      paymentDuration.record(Date.now() - startTime, { currency });
      span.end();
    }
  });
}

Correlating Logs with Traces

Observability reaches its full potential when traces, metrics, and logs are interconnected. Antigravity agents can automatically inject trace context into your existing logger configuration.

// src/lib/logger.ts
// Inject trace context into pino logger
 
import pino from 'pino';
import { trace, context } from '@opentelemetry/api';
 
function getTraceContext() {
  const activeSpan = trace.getSpan(context.active());
  if (!activeSpan) return {};
 
  const spanContext = activeSpan.spanContext();
  return {
    trace_id: spanContext.traceId,
    span_id: spanContext.spanId,
    trace_flags: spanContext.traceFlags,
  };
}
 
export const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  mixin() {
    return getTraceContext();
  },
  formatters: {
    level(label) {
      return { severity: label.toUpperCase() };
    },
  },
  transport: process.env.NODE_ENV === 'development'
    ? { target: 'pino-pretty' }
    : undefined,
});
 
// Usage:
// logger.info({ userId, action: 'login' }, 'User logged in');
// → Output automatically includes trace_id and span_id

With this setup, you can grab a trace ID from an error log and instantly view every span associated with that request in Grafana Tempo or Jaeger. Questions like "which upstream service was slow when this error occurred?" become answerable in seconds.

Backend Configuration: Grafana Stack Integration

For visualizing collected telemetry data, we'll set up the Grafana Stack (Grafana + Tempo + Prometheus + Loki). Have the Antigravity agent generate the Docker Compose configuration.

# docker-compose.otel.yml
# OTel Collector + Grafana Stack generated by Antigravity agent
 
version: '3.8'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ['--config', '/etc/otel/config.yaml']
    volumes:
      - ./otel-config.yaml:/etc/otel/config.yaml
    ports:
      - '4318:4318'   # OTLP HTTP receiver
      - '4317:4317'   # OTLP gRPC receiver
      - '8889:8889'   # Prometheus exporter
    depends_on:
      - tempo
      - loki
 
  tempo:
    image: grafana/tempo:latest
    command: ['-config.file=/etc/tempo/config.yaml']
    volumes:
      - ./tempo-config.yaml:/etc/tempo/config.yaml
      - tempo-data:/var/tempo
    ports:
      - '3200:3200'   # Tempo API
 
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - '9090:9090'
 
  loki:
    image: grafana/loki:latest
    ports:
      - '3100:3100'
 
  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    ports:
      - '3001:3000'
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    depends_on:
      - prometheus
      - tempo
      - loki
 
volumes:
  tempo-data:

# otel-config.yaml
# OpenTelemetry Collector configuration
 
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317
 
processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  attributes:
    actions:
      - key: environment
        value: production
        action: upsert
 
exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch, attributes]
      exporters: [loki]

AI-Powered Anomaly Detection and Auto-Remediation

This is the heart of the article. We'll use Antigravity agents to build a pipeline that detects anomalies from collected telemetry data and executes auto-remediation scripts.

Designing the Anomaly Detection Module

// src/observability/anomaly-detector.ts
// Detect metric anomalies using statistical methods
 
interface MetricDataPoint {
  timestamp: number;
  value: number;
}
 
interface AnomalyAlert {
  metric: string;
  severity: 'warning' | 'critical';
  currentValue: number;
  threshold: number;
  message: string;
  suggestedAction: string;
}
 
export class AnomalyDetector {
  private readonly windowSize = 60; // Last 60 data points
  private readonly stdDevMultiplier = 2.5;
 
  /**
   * Z-score based anomaly detection
   * Identifies data points that deviate significantly from historical mean
   */
  detectAnomaly(
    metricName: string,
    dataPoints: MetricDataPoint[]
  ): AnomalyAlert | null {
    if (dataPoints.length < this.windowSize) return null;
 
    const window = dataPoints.slice(-this.windowSize);
    const values = window.map((dp) => dp.value);
 
    const mean = values.reduce((sum, v) => sum + v, 0) / values.length;
    const stdDev = Math.sqrt(
      values.reduce((sum, v) => sum + Math.pow(v - mean, 2), 0) / values.length
    );
 
    const latest = dataPoints[dataPoints.length - 1].value;
    const zScore = stdDev === 0 ? 0 : Math.abs(latest - mean) / stdDev;
 
    if (zScore > this.stdDevMultiplier) {
      const severity = zScore > 4 ? 'critical' : 'warning';
      return {
        metric: metricName,
        severity,
        currentValue: latest,
        threshold: mean + stdDev * this.stdDevMultiplier,
        message: `Anomaly detected in ${metricName}: ${latest.toFixed(2)} (mean: ${mean.toFixed(2)}, Z-score: ${zScore.toFixed(2)})`,
        suggestedAction: this.getSuggestedAction(metricName, severity),
      };
    }
 
    return null;
  }
 
  private getSuggestedAction(
    metric: string,
    severity: string
  ): string {
    const actions: Record<string, string> = {
      'http.server.duration': 'Check slow queries → expand connection pool → add caching layer',
      'http.server.request.error_rate': 'Review error logs → rollback recent deployment → check dependency health',
      'process.runtime.nodejs.memory.heap_used': 'Investigate memory leaks → capture heap snapshot → restart pods',
      'payments.duration': 'Check Stripe API status → review retry configuration → enable fallback processing',
    };
    return actions[metric] || 'Further investigation needed. Check traces and logs for details.';
  }
}

Implementing the Auto-Remediation Pipeline

// src/observability/auto-remediation.ts
// Execute remediation actions triggered by anomaly detection
 
import { logger } from '../lib/logger';
import { AnomalyAlert } from './anomaly-detector';
 
interface RemediationAction {
  name: string;
  condition: (alert: AnomalyAlert) => boolean;
  execute: (alert: AnomalyAlert) => Promise<RemediationResult>;
}
 
interface RemediationResult {
  success: boolean;
  action: string;
  details: string;
}
 
export class AutoRemediation {
  private actions: RemediationAction[] = [];
  private executionLog: RemediationResult[] = [];
 
  registerAction(action: RemediationAction): void {
    this.actions.push(action);
    logger.info({ actionName: action.name }, 'Remediation action registered');
  }
 
  async handleAlert(alert: AnomalyAlert): Promise<RemediationResult[]> {
    const results: RemediationResult[] = [];
 
    for (const action of this.actions) {
      if (action.condition(alert)) {
        logger.info(
          { alert: alert.metric, action: action.name },
          'Executing remediation action'
        );
 
        try {
          const result = await action.execute(alert);
          results.push(result);
          this.executionLog.push(result);
 
          if (result.success) {
            logger.info(
              { action: action.name, details: result.details },
              'Remediation succeeded'
            );
            break; // Skip subsequent actions on success
          }
        } catch (error) {
          const failResult: RemediationResult = {
            success: false,
            action: action.name,
            details: error instanceof Error ? error.message : 'Unknown error',
          };
          results.push(failResult);
          logger.error(
            { action: action.name, error },
            'Remediation action failed'
          );
        }
      }
    }
 
    return results;
  }
}
 
// Example: Registering auto-remediation actions
const remediation = new AutoRemediation();
 
// Clear cache when memory exceeds threshold
remediation.registerAction({
  name: 'clear-application-cache',
  condition: (alert) =>
    alert.metric.includes('memory') && alert.severity === 'warning',
  execute: async (alert) => {
    globalCache.clear();
    return {
      success: true,
      action: 'clear-application-cache',
      details: `Cache cleared. Memory was at ${alert.currentValue}MB`,
    };
  },
});
 
// Activate circuit breaker on critical error rates
remediation.registerAction({
  name: 'activate-circuit-breaker',
  condition: (alert) =>
    alert.metric.includes('error_rate') && alert.severity === 'critical',
  execute: async (alert) => {
    circuitBreaker.open();
    return {
      success: true,
      action: 'activate-circuit-breaker',
      details: `Circuit breaker activated. Error rate: ${alert.currentValue}%`,
    };
  },
});

Auto-Generating Grafana Dashboards

Tell the Antigravity agent to "generate a Grafana dashboard JSON based on the metrics we're collecting" and it will produce a comprehensive dashboard including your custom metrics.

// scripts/generate-grafana-dashboard.ts
// Dashboard config generator powered by Antigravity agent
 
interface GrafanaPanel {
  title: string;
  type: 'timeseries' | 'stat' | 'gauge' | 'table';
  targets: { expr: string; legendFormat: string }[];
  gridPos: { h: number; w: number; x: number; y: number };
}
 
function generateDashboard(serviceName: string): object {
  const panels: GrafanaPanel[] = [
    {
      title: 'Request Rate (per second)',
      type: 'timeseries',
      targets: [{
        expr: `rate(http_server_request_duration_seconds_count{service_name="${serviceName}"}[5m])`,
        legendFormat: '{{http_method}} {{http_route}}',
      }],
      gridPos: { h: 8, w: 12, x: 0, y: 0 },
    },
    {
      title: 'P95 Latency',
      type: 'timeseries',
      targets: [{
        expr: `histogram_quantile(0.95, rate(http_server_request_duration_seconds_bucket{service_name="${serviceName}"}[5m]))`,
        legendFormat: '{{http_route}}',
      }],
      gridPos: { h: 8, w: 12, x: 12, y: 0 },
    },
    {
      title: 'Error Rate',
      type: 'stat',
      targets: [{
        expr: `sum(rate(http_server_request_duration_seconds_count{service_name="${serviceName}",http_status_code=~"5.."}[5m])) / sum(rate(http_server_request_duration_seconds_count{service_name="${serviceName}"}[5m])) * 100`,
        legendFormat: 'Error %',
      }],
      gridPos: { h: 4, w: 6, x: 0, y: 8 },
    },
    {
      title: 'Active Payments',
      type: 'gauge',
      targets: [{
        expr: `payments_active{service_name="${serviceName}"}`,
        legendFormat: 'Active',
      }],
      gridPos: { h: 4, w: 6, x: 6, y: 8 },
    },
  ];
 
  return {
    dashboard: {
      title: `${serviceName} — Observability Dashboard`,
      tags: ['opentelemetry', 'auto-generated'],
      panels,
      time: { from: 'now-1h', to: 'now' },
      refresh: '10s',
    },
  };
}
 
// Run: npx tsx scripts/generate-grafana-dashboard.ts > dashboard.json
const dashboard = generateDashboard('my-api-service');
console.log(JSON.stringify(dashboard, null, 2));

Import the generated JSON into Grafana's Dashboard Import and monitoring begins immediately.

Measuring what instrumentation costs

Instrumentation is not free. The moment I enabled auto-instrumentation, latency got visibly worse.

Below are median figures from three runs of autocannon -c 50 -d 30 against the same endpoint, on Node.js 22 in a 2-vCPU container, exporting over OTLP/gRPC to a sidecar Collector.

Configuration	p50 latency	p99 latency	RSS	Throughput
No instrumentation	8.4 ms	31 ms	112 MB	5,180 req/s
Auto-instrumentation, all spans exported	11.9 ms	58 ms	168 MB	3,940 req/s
Auto-instrumentation + 10% head sampling	9.1 ms	36 ms	131 MB	4,870 req/s
Auto-instrumentation + tail sampling (Collector)	9.0 ms	35 ms	129 MB	4,900 req/s

Exporting every span cost roughly 24% of throughput. The near-doubling of p99 comes from export batch flushes blocking the event loop.

The part worth pausing on: head sampling and tail sampling perform almost identically, but they preserve completely different information. Head sampling rolls the dice when a trace begins, so it discards roughly 90% of your failing 1% of requests. Those discarded traces are precisely the ones you want during an incident.

Tail sampling decides after the trace completes, based on whether it contains an error or exceeded a latency threshold. The Collector configuration looks like this.

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      # 1. Always keep traces containing errors
      - name: errors-always
        type: status_code
        status_code: { status_codes: [ERROR] }
      # 2. Always keep slow traces
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 500 }
      # 3. Keep 5% of everything else, for baseline
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

decision_wait is how long the Collector waits for a trace's spans to arrive. Set it too low and you discard traces as "successful" before the error span shows up. I started at 2 seconds and systematically lost every trace containing an async background job. The measured p99 for those jobs was 6 seconds; raising the wait to 10 seconds fixed it.

I prefer this shape. Every error and every slow request survives, and only the uneventful traces get thinned to 5%. Cost lands at roughly 8% of full export, and nothing an investigation needs is gone.

The morning the numbers stopped agreeing

About two weeks after tail sampling went in, I opened the request-rate panel and stopped. Grafana reported roughly one twentieth of the requests per hour that the access log did.

Nothing was broken. I had simply put the pipeline in the wrong order.

Metrics built from spans count the world after the thinning

The spanmetrics connector derives request counts and latency histograms from spans. It is convenient enough that you want to drop it straight into the trace pipeline. Place it after tail_sampling, though, and all it ever sees are the traces that survived. Thin the uneventful ones to 5% and your request rate becomes 5% too.

The error rate is worse. Traces containing an error are kept at 100% while successful ones are kept at 5%, so the error rate reads about twenty times higher than reality. My dashboard was presenting a calm service as a critical one.

The fix is to put the connector ahead of sampling and route metrics through their own pipeline.

# otel-collector-config.yaml (excerpt)
connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [10ms, 50ms, 100ms, 250ms, 500ms, 1s, 5s]
    dimensions:
      - name: http.route
      - name: http.status_code
  forward/sampled: {}
 
service:
  pipelines:
    # 1) Turn every received span into metrics, before any thinning
    traces/in:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [spanmetrics, forward/sampled]
 
    # 2) Only then decide which traces are worth storing
    traces/sampled:
      receivers: [forward/sampled]
      processors: [tail_sampling, batch]
      exporters: [otlp/tempo]
 
    # 3) Span-derived metrics come from the full-volume world
    metrics/spanmetrics:
      receivers: [spanmetrics]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

Counting happens before sampling; storing happens after. I left that line as a comment at the top of the config file, and I have not repeated the mistake since.

Policies are evaluated as "any match wins"

I had misread the config in a second way. The policies under tail_sampling are not tried top to bottom until one matches. Every policy is evaluated, and a single sample decision keeps the trace.

Read the three from the previous section out loud and you get: "contains an error, or exceeds 500ms, or won a 5% lottery." That is what I wanted. But if you start adding conditions in pursuit of "only the slow errors," each addition keeps more, not less. The operation you think is narrowing runs the other way.

To combine conditions, use the and policy.

  tail_sampling:
    decision_wait: 10s
    policies:
      # Keep only traces that are not health checks AND slower than 500ms
      - name: slow-but-not-healthcheck
        type: and
        and:
          and_sub_policy:
            - name: not-healthcheck
              type: string_attribute
              string_attribute:
                key: http.route
                values: ['/health', '/metrics']
                invert_match: true
            - name: slow
              type: latency
              latency: { threshold_ms: 500 }

If you need a ceiling on the overall sampled rate, the composite policy lets you allocate spans per second across policies. My traffic has not required it yet, though it is the first thing I would add if volume grew tenfold.

Whatever you add to decision_wait, memory pays for

I mentioned raising decision_wait from 2 seconds to 10. That has a price. The Collector holds spans in memory until a decision is final, so the requirement scales roughly with traces per second × decision_wait × spans per trace.

Holding load constant (about 1,000 traces/sec, 18 spans on average) and varying only decision_wait, observing RSS and loss for thirty minutes each:

decision_wait	num_traces	Collector RSS	Traces dropped before a decision
2s	50,000	142 MB	31/hour (those containing async jobs)
10s	50,000	318 MB	0/hour
30s	50,000	703 MB	0/hour

Nothing in my environment justified 30 seconds. The measured p99 for those async jobs was 6 seconds, which puts 10 at a little over one and a half times the worst case. Derive the sampling grace period from the slowest measured path — a number chosen by intuition gets billed either in memory or in missing traces.

num_traces caps how many traces can be held at once. Overflow evicts the oldest, and what gets evicted leaves no trace of its own. Put otelcol_processor_tail_sampling_sampling_trace_dropped_too_early on the dashboard from day one and the loss becomes visible. I had not, which is why I only discovered two weeks of missing traces well after the fact.

Cardinality explosion, and the invoice that announces it

The first thing to break in production is usually metric cardinality.

Every distinct value you attach as a metric label creates its own time series. Put a user ID in a label and you have one series per user.

So this, written carelessly, is dangerous:

// ❌ Dangerous: userId and raw path are high-cardinality
requestCounter.add(1, {
  'http.route': req.path,        // /users/8a3f-… is its own label
  'user.id': req.user.id,        // one series per user
});

Template the path, and move per-request identifiers onto the span instead. Metrics are a vessel for aggregation; spans are a vessel for individual events. Blur that boundary and the invoice will explain it to you.

// ✅ Keep metrics low-cardinality
requestCounter.add(1, {
  'http.route': req.route?.path ?? 'unknown',  // /users/:id
  'http.status_code': res.statusCode,
  'service.version': process.env.APP_VERSION ?? 'dev',
});
 
// Identity belongs on the span
const span = trace.getActiveSpan();
span?.setAttribute('user.id', req.user.id);
span?.setAttribute('request.id', req.headers['x-request-id'] as string);

In my case, emitting http.route as a raw path for three days grew the series count to roughly 42,000 and pushed Prometheus scrapes past 12 seconds, where they timed out. After templating: 380 series. Less than one hundredth.

It's worth adding a safety net in the Collector too.

processors:
  # Drop unexpected high-cardinality attributes
  attributes/scrub:
    actions:
      - key: user.id
        action: delete
      - key: session.id
        action: delete
  # Cap memory so a cardinality spike drops data instead of the Collector
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

With memory_limiter in place, a cardinality explosion causes the Collector to shed data rather than die. An outage where the monitoring stack fails first is the least recoverable kind.

What I delegated to the agent, and what I kept

Where to draw the line with Antigravity's agents. Here is where mine ended up.

Task	Owner	Reasoning
Selecting and installing auto-instrumentation libraries	Agent	Dependency analysis is more accurate by machine; fewer gaps
Proposing custom span boundaries	Agent → human review	Proposals are sharp, but only a human knows which boundaries matter to the business
Choosing a sampling policy	Human	"What can we afford to lose" is a cost and risk call, not a technical one
Naming attributes / cardinality design	Human	Large billing impact, hard to reverse later
Generating the initial Grafana dashboard	Agent	An excellent first draft — usable as-is
Setting alert thresholds	Human	Tolerance for false alarms depends on team culture
First-pass anomaly classification	Agent	Good at removing noise; final judgment stays with a person
Applying auto-remediation in production	Human (approval-gated)	See below

I stopped short of fully automatic remediation because I tried it once and it frightened me.

The agent saw memory usage cross a threshold and scaled out. The actual problem was a memory leak. Scaling out hid the symptom for a few hours while the number of leaking instances grew. A machine can make an anomaly disappear; it has no notion of whether the anomaly deserved to disappear.

Now the agent posts a remediation proposal to Slack along with its evidence — the offending spans, the metric deltas, similar past incidents — and only what I approve gets executed. I still get woken at 2 a.m. just as often. But the time from waking to understanding went from three hours to about ten minutes.

That boundary follows the same reasoning I used for test automation in Building test pipelines with Antigravity agents: a practical design for delegating QA to AI. Delegate generation, keep approval.

Sequence matters: don't stand it all up at once

Bring up instrumentation, sampling, backends, and anomaly detection simultaneously and you will not know which one broke. The work of introducing observability becomes, ironically, unobservable.

The second time around, this was my order.

Auto-instrumentation only, console exporter (half a day) — verify that spans emerge with correct parent-child relationships. No backend yet.
Add the Collector, ship to Tempo/Prometheus (one day) — network and resource problems surface here. Because step 1 is known-good, isolating them is fast.
Add sampling, measure the cost (one day) — run at full volume for a day first, then thin. The reduction becomes a number rather than a hope.
Add custom spans and business metrics (ongoing) — add one the moment you find yourself wanting to know how long something takes. Spans added preemptively mostly go unread.
Anomaly detection and remediation proposals (a month later) — only once two weeks of baseline has accumulated. Anomaly detection without data is a random number generator.

The month-long gap between steps 3 and 5 is deliberate. A detector without a baseline forces you to fill thresholds in from intuition. With two weeks of data, the distribution fills them in for you.

For the CI-integration stage, Multi-agent CI/CD quality gates: automating PR review, testing, and security with Antigravity × GitHub Actions may be a useful companion.

Closing thoughts

What that empty 2 a.m. log taught me is that monitoring cannot be added after the incident. The information you desperately want during an outage has to have been planted before it.

Antigravity's agents reliably reduce the labor of planting it. But three questions remain yours to answer: what you can afford to lose (sampling), what you choose to count (cardinality), and what may be fixed without you (remediation). Agents can own the implementation. They cannot own the judgment.

If you try one thing today, run auto-instrumentation with a console exporter and hit a single endpoint. Check that the parent-child relationships come out the way you expect. That thirty minutes makes the following month considerably lighter.

I'm still learning this as I operate it. If any of it helps someone else who gets woken at two in the morning, I'll be glad.

Thank You for Reading

Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.