When A2A Tasks Sat Stuck in 'working' — Field Notes on Tracing State Transitions to Detect Stalls

Building a Monitored A2A Client

Below is a minimal build that inserts transition recording into both send and receive, and has a watchdog name any task that lingers too long in a non-terminal state. Records go to a lightweight store (memory here; Redis or the like in production).

# env: A2A_STALL_FACTOR="3"
import asyncio
import time
import uuid
from dataclasses import dataclass, field
from collections import defaultdict
from antigravity import A2AClient, TaskRequest, Message
 
TERMINAL = {"completed", "failed", "cancelled"}
 
@dataclass
class Transition:
    task_id: str
    agent: str
    from_state: str
    to_state: str
    at: float
    dwell: float
 
@dataclass
class TraceStore:
    events: list[Transition] = field(default_factory=list)
    _last: dict = field(default_factory=dict)  # task_id -> (state, at)
 
    def record(self, task_id: str, agent: str, to_state: str) -> None:
        now = time.monotonic()
        prev_state, prev_at = self._last.get(task_id, ("<start>", now))
        self.events.append(Transition(
            task_id=task_id, agent=agent,
            from_state=prev_state, to_state=to_state,
            at=now, dwell=now - prev_at))
        self._last[task_id] = (to_state, now)
 
    def open_tasks(self) -> list[tuple[str, str, str, float]]:
        """Return tasks lingering in a non-terminal state as (task_id, agent, state, elapsed)."""
        now = time.monotonic()
        out = []
        for task_id, (state, at) in self._last.items():
            if state not in TERMINAL:
                agent = next((e.agent for e in reversed(self.events)
                              if e.task_id == task_id), "?")
                out.append((task_id, agent, state, now - at))
        return out
 
    def edge_p95(self) -> dict[tuple[str, str], float]:
        """p95 dwell time per edge; used to derive the stall threshold."""
        buckets = defaultdict(list)
        for e in self.events:
            buckets[(e.from_state, e.to_state)].append(e.dwell)
        p95 = {}
        for edge, vals in buckets.items():
            vals.sort()
            p95[edge] = vals[max(0, int(len(vals) * 0.95) - 1)]
        return p95

With the recorder in place, call record at the key points of sending and receiving. Even if the peer forgets to assert completed, the watchdog can still detect it as "a task lingering in working," which means the skipped transition itself becomes observable from the outside.

store = TraceStore()
 
class MonitoredClient(A2AClient):
    async def send_task(self, target, task: TaskRequest):
        store.record(task.id, self.my_agent_id, "submitted")
        resp = await super().send_task(target, task)
        store.record(task.id, target, "working")
        return resp
 
    async def on_state(self, task_id: str, agent: str, state: str):
        store.record(task_id, agent, state)
 
async def watchdog(stall_factor: float = 3.0, interval: float = 10.0):
    """Flag stalls at edge p95 x factor and warn by name."""
    while True:
        await asyncio.sleep(interval)
        p95 = store.edge_p95()
        for task_id, agent, state, elapsed in store.open_tasks():
            budgets = [v for (f, _), v in p95.items() if f == state]
            budget = max(budgets) * stall_factor if budgets else 60.0
            if elapsed > budget:
                print(f"STALL task={task_id[:8]} agent={agent} "
                      f"state={state} elapsed={elapsed:.0f}s budget={budget:.0f}s")

open_tasks emits tasks lingering in non-terminal states, and edge_p95 gives the standard duration for each edge. The watchdog reconciles the two and names any task sitting far longer than standard, with its agent and stalled state attached. Narrowing "three tasks froze" down to "summarizer has not fired working → completed within 3x its p95" changes the starting speed of the investigation completely.

Derive the Stall Threshold from Dwell Time

Applying one fixed timeout to every task is crude for agents that call an LLM internally. Summarization and image analysis differ in duration by an order of magnitude. So set the threshold from the p95 of dwell time per edge, times a constant.

In my own runs I start with a factor of 3. Agent response times have a long right tail, so basing the threshold on p95 rather than the mean stays stable. Assume the distribution itself moves; do not trust the factor as fixed, and tune it while watching the false-positive rate.

Choosing a Shape That Stalls Less

Once measurement makes stalls visible, lean toward a shape that stalls less to begin with. In A2A, a cycle where agent A calls B, B calls C, and C calls A again makes the causality of a stall untraceable. I decide as follows.

1. Up to mid scale, use a DAG (an orchestrator calling workers in order) and prioritize traceability above all
2. Require the peer to always return an explicit completed / failed transition, leaving no path that replies silently
3. Split auth tokens per agent to bound both the blast radius of a stall and the impact of a leak

A DAG makes the center prone to becoming a bottleneck, but during a production incident it tells you where things froze in seconds. I myself pick a DAG in almost every case, prioritizing observability over scalability, and I strongly recommend that call. Hard-to-visualize choreography is safe to adopt only once you have a way to funnel transition logs into one place via an event bus.

What I Kept Tripping Over

I stepped in the same holes more than once, so I will write them down.

Make the watchdog interval too short and you misflag legitimately long tasks as stalls, and the warnings overflow. Summarization routinely takes tens of seconds, so a fixed threshold that ignores p95 exhausts the team every time it fires. Do not skip the small step of deriving the threshold from the distribution.

If you assign a trace_id on the sender side but reset it on the receiver, the edge timeline splits into two and the stall becomes untraceable. The more you hit the low-level API directly, the more care it takes to carry the envelope's trace_id unbroken through the whole chain.

And the biggest lesson: stall detection only tells you that something froze. Why it froze is settled in the final step, where a person reads the logs of the named agent. Measurement is a tool for pointing at the cause, not the cause itself. Miss this and you end up with a handsome dashboard and no fewer incidents.

The Next Step

If you have a setup where two or more agents coordinate, start by recording nothing but the state transitions. The moment a working task shows up by name in open_tasks while the error logs say nothing, the point of looking on the time axis lands.

A silent stall, left alone, only piles up the assumption that "it must be running." Turn state transitions into one timeline you can read by edge, and you can peel that assumption away one layer at a time. I am still tuning the factor myself, but I hope it helps with your own agent operations.