When the Android CLI Got 3x Faster and Cut Tokens by ~70%, the Right Move Was More Verification Per Change — Not More Parallelism

Set the parallelism cap from verification throughput, not the downstream

So how do you set the cap? Not from a downstream like an external API rate limit, but from the narrowest later stage — which, in most solo setups, is human review time.

Use Little's Law directly. Pick a target lead time W_target, measure the later-stage throughput λ, and the admissible WIP cap falls out.

WIP_max = λ_review × W_target

Example: if review plus gates reliably clear 6 changes/day (λ_review = 6/day)
         and you target a lead time of 1 day (W_target = 1 day):
    WIP_max = 6 × 1 = 6

→ No matter how fast generation is, allow at most 6 changes "in progress" at once.

It matters that λ_review is measured, not wished. I counted merged changes daily for about two weeks and took the median. Use the real value — including interrupted and off days — not the "I could do more if I tried" number. Set it optimistically and the queue overflows and you are back where you started.

Enforce the WIP cap with an admission controller (Python)

Once the cap is set, enforce it mechanically. Put a single admission controller just before the scheduler dispatches a task: if in-progress work has hit WIP_max, hold the new task. No clever control is needed — one gate that increments and decrements a count atomically is enough.

# admission_controller.py
# Keep in-progress (WIP) changes under WIP_MAX.
# State lives in one file, updated atomically with flock (safe under concurrent starts).
import fcntl
import json
import os
import time
from contextlib import contextmanager
from dataclasses import dataclass
 
STATE_PATH = os.environ.get("WIP_STATE", "/var/agent/wip_state.json")
 
 
@dataclass(frozen=True)
class WipConfig:
    wip_max: int          # = round(lambda_review_per_day * W_target_days)
    stale_after_sec: int  # if still in-flight past this, reclaim as stuck
 
 
@contextmanager
def _locked_state(path: str):
    """Read/write the state file under an exclusive lock."""
    os.makedirs(os.path.dirname(path), exist_ok=True)
    fd = os.open(path, os.O_RDWR | os.O_CREAT, 0o644)
    try:
        fcntl.flock(fd, fcntl.LOCK_EX)
        raw = os.read(fd, 1_000_000).decode() or "{}"
        state = json.loads(raw)
        state.setdefault("inflight", {})  # change_id -> started_at(epoch)
        yield state
        os.lseek(fd, 0, os.SEEK_SET)
        os.ftruncate(fd, 0)
        os.write(fd, json.dumps(state).encode())
    finally:
        fcntl.flock(fd, fcntl.LOCK_UN)
        os.close(fd)
 
 
def _reap_stale(state: dict, stale_after_sec: int) -> None:
    """Drop changes stuck with no verify/review progress (keeps the count honest)."""
    now = time.time()
    for cid, started in list(state["inflight"].items()):
        if now - started > stale_after_sec:
            del state["inflight"][cid]
 
 
def try_admit(change_id: str, cfg: WipConfig) -> bool:
    """Register in-flight and return True if there is room; False if full."""
    with _locked_state(STATE_PATH) as state:
        _reap_stale(state, cfg.stale_after_sec)
        if change_id in state["inflight"]:
            return True  # idempotent: never start the same change twice
        if len(state["inflight"]) >= cfg.wip_max:
            return False
        state["inflight"][change_id] = time.time()
        return True
 
 
def release(change_id: str) -> None:
    """Once merged or rejected, free the slot."""
    with _locked_state(STATE_PATH) as state:
        state["inflight"].pop(change_id, None)
 
 
if __name__ == "__main__":
    # λ_review = 6/day, W_target = 1 day → WIP_MAX = 6
    cfg = WipConfig(wip_max=6, stale_after_sec=36 * 3600)
    cid = os.environ["CHANGE_ID"]
    if try_admit(cid, cfg):
        print(f"admit {cid}")  # only now let the agent start
    else:
        print(f"defer {cid}")  # hold until a later stage frees up
        raise SystemExit(75)   # EX_TEMPFAIL: tell the scheduler to retry later

Three things matter. First, flock makes the update atomic, so concurrent scheduled runs never skew the count. Second, try_admit is idempotent per change_id, so retries never double-start. Third, _reap_stale evicts changes stuck without progress after a set time; without it, a failed change holds its slot forever and healthy changes can never start.

The agent launcher runs only what try_admit allows and calls release when done. However fast generation gets, changes in progress never exceed WIP_MAX.

Spend the freed token budget on depth, not breadth

Capping parallelism leaves you with spare speed and tokens. I chose to put the 3x/70% surplus into depth (more verification per change), not breadth (more changes). Since the later stages set delivered, breadth wouldn't move it anyway.

Concretely, per change I added:

1. A second self-review pass. Have the agent review its own diff under a separate prompt, listing mismatches against the spec and unhandled edge cases. If issues surface, rewrite before a human sees it.
2. Targeted extra evals. Beyond tests for the touched area, run one extra set of golden outputs for places that broke on similar past fixes.
3. A richer review summary. Attach the intent, the risk, and the points to check, structured so a human reads it in 30 seconds. This one actually raises λ_review.

The third is different in kind. Items 1 and 2 invest in per-change quality; item 3 invests in the later-stage throughput λ itself. Faster, more accurate review lets you raise WIP_max — one of the few moves that helps both depth and breadth.

How to measure it: delivered/week and escape rate

After changing the design, confirm with numbers, not impressions. I track just two.

• delivered/week: changes actually merged. If parallelism rises but the later stages are unchanged, this should not move — test whether that prediction holds.
• escape rate: the share of merged changes where a defect that review should have caught turns up afterward. Raise parallelism and thin out review, and this always worsens.

My two-week, two-condition comparison — same later-stage capacity, one run parallelism-first and one WIP-capped with deeper verification — came out roughly like this:

delivered/week barely moved — as predicted, because the later stages are the constraint. Meanwhile lead time fell to a third and escape rate to under a third. Tokens stayed at half the baseline even after adding the second self-review. In short, routing the speed-and-token surplus into depth kept output steady while improving both speed (lead time) and certainty (escape rate) at once.

These are one environment's numbers and shift with your later-stage throughput and change difficulty. What matters is less the absolute values than whether the structure — "parallelism-first doesn't grow delivered but does grow risk" — reproduces in your own data.

The order to roll this out (your next move)

If you start tomorrow, do one thing first. For two weeks, count the changes you actually merge each day, take the median for λ_review, and compute WIP_max = λ_review × W_target. Put it in the admission controller to physically cap parallelism, and route the freed speed and tokens into "a second self-review pass per change." Reverse the order — raise parallelism first — and you'll be scrambling to add a cap only after the queue has already stretched.

When a faster tool lands, I've come to believe the first question is not "how many can I run at once" but "where is the narrowest stage right now." Speed the entrance without widening the narrow part and the line only grows. Speed belongs not in parallelism but in making each change certain — and that judgment is what quietly holds up unattended operations.