After Generating Several Candidates, Which One Do You Adopt? Designing Best-of-N That Arbitrates by Verification

Order the Verification Gates by Stage

Running every gate on every candidate is wasteful. There is no point running tests on a candidate that fails type checking. Order the gates from cheapest to most expensive and eliminate failing candidates where they fall.

With this order, obviously broken candidates disappear before reaching tests or the smoke run. Even if you generate five candidates with Flash, often only one or two actually reach the smoke stage, so total verification cost does not scale linearly with candidate count.

Implementing the Arbiter

This is the core: run candidates through gates, score them, and return an adoption. Shown in TypeScript. Swap the verification functions for whatever your environment uses (type-check command, test runner).

type Candidate = {
  id: string;
  apply: () => Promise<void>;   // reflect the candidate into the worktree
  revert: () => Promise<void>;  // roll the reflection back
};
 
type GateResult = { passed: boolean; penalty: number; note: string };
type Gate = {
  name: string;
  run: () => Promise<GateResult>;
  fatal: boolean; // true: eliminate on failure / false: deduct only
};
 
type Verdict =
  | { kind: "adopt"; id: string; score: number; trail: string[] }
  | { kind: "all_failed"; trail: string[] }
  | { kind: "budget_exhausted"; trail: string[] };
 
async function arbitrate(
  candidates: Candidate[],
  gates: Gate[],
  opts: { maxVerifyMs: number },
): Promise<Verdict> {
  const trail: string[] = [];
  const start = Date.now();
  let best: { id: string; score: number } | null = null;
 
  for (const c of candidates) {
    if (Date.now() - start > opts.maxVerifyMs) {
      trail.push(`budget: stopped before verifying ${c.id}`);
      return best
        ? { kind: "adopt", id: best.id, score: best.score, trail }
        : { kind: "budget_exhausted", trail };
    }
 
    await c.apply();
    let score = 100;
    let eliminated = false;
 
    for (const g of gates) {
      const r = await g.run();
      trail.push(`${c.id}/${g.name}: ${r.passed ? "ok" : "ng"} ${r.note}`);
      if (!r.passed && g.fatal) { eliminated = true; break; }
      if (!r.passed) score -= r.penalty;
    }
 
    await c.revert();
    if (eliminated) continue;
 
    // Ties go to the first arrival (pass candidates in a stable order for determinism)
    if (!best || score > best.score) best = { id: c.id, score };
  }
 
  if (!best) return { kind: "all_failed", trail };
  return { kind: "adopt", id: best.id, score: best.score, trail };
}

There are three design points. First, always cycle apply → verify → revert per candidate so candidates do not pollute each other's state. Second, eliminate early with fatal gates to skip wasted verification. Third, record every decision into trail. Being able to reproduce later why a candidate was chosen (or dropped) is a lifeline in autonomous operation.

How to Fold Up Ties, Total Failure, and Budget Exhaustion

The unhappy paths matter more in design than the happy one.

On a tie, you may be tempted to fall back to majority vote, but here we take the first arrival in stable order. Choosing at random makes the result change between runs for the same input, breaking reproducibility. Always pass candidates in the same order and keep arbitration deterministic.

When every candidate fails (all_failed), do not auto-adopt. Push the candidate with the smallest penalty onto a "waiting for human review" queue and keep it out of the mainline. Forcing one through here leads to the worst accident: a broken change merged automatically.

Budget exhaustion (budget_exhausted) is insurance for when verification takes too long. Once maxVerifyMs is exceeded, take the provisional best so far, or abort if there is none. If you run parallel agents overnight, capping per-task verification is essential — otherwise one slow run stalls the whole batch.

How Many Candidates Should You Generate?

Larger N is not always better. Verification cost and adoption quality both have diminishing returns. Across the routine fix tasks I observed, the trend looked roughly like this.

The jump from one to three candidates is large; three to five is small. I default to N=3 for routine tasks and only raise it to N=5 for task types that have failed often in the past. Because Flash keeps the generation side roughly flat, what really matters is not letting the verification side balloon — which is what staged gates are for.

Three Checks Before Putting It Into Operation

Before wiring this into an autonomous pipeline, I always confirm these three points.

1. Are the verification gates truly independent of generation? If you have the same agent write the type checks or tests themselves, independence collapses. Fix the gates by hand and let the agent handle only generation.
2. Does revert reliably roll back? If the worktree stays dirty when you apply the next candidate, the score gets dragged by the previous one. Isolating each candidate with git worktree is the solid choice.
3. Is the path designed so total failure never enters the mainline? This is the one place to give up on automation and hand off to a human; not breaking that design is your safety valve for long-term operation.

Rather than adding more candidates, build verification to be independent and staged. In an era where generation has become fast and cheap, I believe what pays off is the design of the side that chooses. I hope this gives a useful handhold to anyone wrestling with the same problem.