Keeping Unattended Agent Run Logs Long Enough to Debug — Without Filling the Disk

Retain in three tiers: hot, warm, cold

The structure I settled on has three tiers.

1. Hot tier (last 7 days, full text): complete records including stdout, exception stacks, and an input summary. Recent failures get diagnosed from here.
2. Warm tier (8–30 days, summary only): the full text is dropped and each run is compacted to a one-line summary record — success/failure, duration, item count, exit code, and just the first line of any error.
3. Cold tier (31 days and older): deleted by default. If you need trend analysis, push only daily aggregate numbers to a separate small file.

In concrete numbers, my four-site setup produces 30–50 MB of raw logs per day in total. Holding seven days in the hot tier caps it at roughly 350 MB at steady state. A warm summary is about 200 bytes per run, so even 30 days fits in a few megabytes. The result is that total log footprint stabilizes around 400 MB and no longer grows without bound.

You can also draw the tier boundaries by run count instead of days, but I recommend days. You can search backward from the date of an incident, which makes your intuition work for you during an investigation.

Protect failure logs past their expiry

Tiered retention needs exactly one exception added to it: failed records must not be dropped from the hot tier even after the retention window passes.

The reason is simple — the rarer a failure, the more valuable it is. A job that breaks once a month under a specific condition will have its evidence erased by a 7-day window before it recurs. You'd be left waiting for a repeat with no idea of the cause, which is the single worst position to be in for unattended operation.

In practice, when you select what to compact or delete, only evict records that are both successful and past the window, and keep failures on a separate, longer schedule. I keep failed records in full for up to 90 days, and only thin the oldest ones if they somehow keep growing. Failures rarely exceed a few dozen megabytes in 90 days, so the impact on disk is negligible.

// Eviction candidates: only successful records past the window
type RunRecord = {
  schema: number;        // schema version (see below)
  runId: string;
  site: string;
  startedAt: string;     // ISO8601
  ok: boolean;
  exitCode: number;
  durationMs: number;
  itemCount: number;
  errorHead?: string;    // failures only: first line of the error
};
 
const DAY = 24 * 60 * 60 * 1000;
 
function isEvictable(r: RunRecord, now: number): boolean {
  const ageDays = (now - Date.parse(r.startedAt)) / DAY;
  if (!r.ok) {
    // Failures are protected for 90 days; never evicted on age alone
    return ageDays > 90;
  }
  // Successes leave the hot tier after 7 days
  return ageDays > 7;
}

This one rule alone goes a long way toward preventing the "it recurred, but last time's log is gone" situation.

Give every record a schema version

A pitfall that bites in long-term operation is changing the log format. As you keep running, you will inevitably hit a moment of "I wish I'd recorded that field too." The instant you add or rename a field, the aggregation script that reads old logs throws on the older lines and stops.

To avoid this, put an integer schema version at the head of every record. The reader branches on the version, refuses to crash on a version it doesn't know, and interprets only the fields it understands.

// Reader: survive unknown schema versions, parse only what you know
function parseRecord(line: string): RunRecord | null {
  let raw: any;
  try {
    raw = JSON.parse(line);
  } catch {
    return null; // skip broken lines silently (e.g. partial-write debris)
  }
  const schema = typeof raw.schema === "number" ? raw.schema : 1;
  if (schema > 2) {
    // Ignore future fields; pull out only the common ones
    return {
      schema, runId: raw.runId, site: raw.site,
      startedAt: raw.startedAt, ok: !!raw.ok,
      exitCode: raw.exitCode ?? -1,
      durationMs: raw.durationMs ?? 0,
      itemCount: raw.itemCount ?? 0,
      errorHead: raw.errorHead,
    };
  }
  // v1 had no itemCount → fill a default
  return { itemCount: 0, ...raw, schema };
}

The version number costs almost nothing and pays off later: it spares you from bulk-converting all your past logs whenever the schema changes.

Run a single compaction job

Tiered retention does not happen on its own. The work of demoting hot to warm and expiring warm should run as a separate, small job once a day. The key is not to make the main agent do this cleanup. If you bury the cleanup inside the main process, log rotation stops on any day the main process fails — so logs pile up most on exactly the day you most need them.

#!/usr/bin/env bash
# rotate-logs.sh — run once a day, separately from the main job
set -euo pipefail
LOG_DIR="${HOME}/agent-logs"
HOT="${LOG_DIR}/hot.jsonl"
WARM="${LOG_DIR}/warm.jsonl"
TMP="$(mktemp "${LOG_DIR}/rotate.XXXXXX")"
 
node "${LOG_DIR}/compact.mjs" "$HOT" "$WARM" "$TMP"
 
# Swap atomically (the original stays intact if this dies midway)
mv "$TMP" "$HOT"
 
# Check free disk; thin further if it's tight
FREE_MB=$(df "$LOG_DIR" --output=avail -m | tail -1 | tr -d ' ')
if [ "${FREE_MB:-0}" -lt 300 ]; then
  echo "low disk: ${FREE_MB}MB — running emergency eviction"
  node "${LOG_DIR}/evict.mjs" "$WARM"
fi

Writing to a temp file and replacing with mv keeps the original hot log intact if the job dies mid-compaction. If you trim a file in place, an interruption during the trim can lose the log itself. Having the cleanup job destroy the very thing it's meant to maintain defeats the purpose, so I recommend the atomic swap here.

A priority eviction order under disk pressure

The day will come when capacity simply runs short. Deciding in advance what to delete first prevents the accident of panic-deleting something important. The order I use is:

1. Full text of successful records (hot tier). The summary survives in warm, so drop the full text first.
2. Summary records from the old end of the warm tier, oldest date first.
3. If that's still not enough, the oldest failed records past 90 days.

The spine of this order is one principle: discard the details of success first, and guard the evidence of failure to the very end. The logs you actually re-read in unattended operation are the failures, not the successes. Evict in the opposite order and you'll free up disk but be left with no clues when it matters.

Note that deletion should write the lines you want to keep into a new file and swap it in, rather than truncating in place. The reason is the same as the compaction job: the original survives even if you're interrupted partway.

A small check to fold into your routine

Finally, a short morning check so this doesn't become "write it and forget it." Because the warm tier is one summary line per run, you can grasp last night's outcomes just by reading it.

# List last night's runs with pass/fail (readable in 30 seconds)
node -e '
const fs=require("fs");
const lines=fs.readFileSync(process.env.HOME+"/agent-logs/warm.jsonl","utf8").trim().split("\n");
const since=Date.now()-24*60*60*1000;
for(const l of lines){const r=JSON.parse(l);
  if(Date.parse(r.startedAt)<since) continue;
  console.log(`${r.ok?"OK ":"NG "} ${r.site} ${r.itemCount} items ${r.errorHead??""}`);}
'

Once glancing at this list becomes a habit, failures almost stop slipping away unnoticed. The goal of log design isn't to save space — it's to reach the reason something broke by the shortest path. Since switching to this three-tier retention, the time I spend on root-cause investigation has dropped by more than half, by feel.

Unattended logs are a handoff note to your future self. Don't hoard the full text forever, and don't keep only the latest while throwing away the past. Vary the granularity by freshness, and guard the evidence of failure for the long haul. Draw that line up front and you get both a flat disk and peace of mind. I hope this helps anyone else running several jobs unattended.