Implementation Notes: Running AdMob Mediation A/B Tests with Antigravity Parallel Agents

Orchestrator Agent — Hypothesis Generation

The Orchestrator runs weekly, pulls last week's AdMob report, identifies four dimensions with improvement headroom, and emits one hypothesis per dimension. Its job is solely to feed four Experimenters.

# orchestrator.py
from antigravity import Agent
from admob import fetch_weekly_report
import json
 
ORCHESTRATOR_PROMPT = """You are the AdMob mediation optimization lead.
Look at last week's numbers and form 4 hypotheses with improvement headroom.
 
Input data:
{report_summary}
 
Evaluation axes:
- eCPM (effective CPM): meaningful at $0.50 granularity
- ARPU (average revenue per user): evaluate per segment
- fill rate: any network below 90% is suspicious
 
Hypothesis output format (JSON array):
[
  {
    "id": "exp_001",
    "hypothesis": "Promoting Meta Audience Network to top for banners should raise eCPM",
    "format": "banner",
    "change": {"network_order": ["meta", "applovin", "admob", "unity"]},
    "target_segment": "iOS / Japan / 30+",
    "expected_outcome": "eCPM +5% / fill rate stable",
  },
  ...
]
"""
 
def orchestrate(week_start: str) -> list[dict]:
    report = fetch_weekly_report(week_start)
    report_summary = json.dumps(report, ensure_ascii=False, indent=2)
    
    agent = Agent(
        model="gemini-3-pro",  # Pro recommended for hypothesis generation
        system_prompt=ORCHESTRATOR_PROMPT.format(report_summary=report_summary),
    )
    
    response = agent.run("Output 4 hypotheses as JSON for last week's report.")
    hypotheses = json.loads(response.text)
    
    assert len(hypotheses) == 4, "Orchestrator must return exactly 4 hypotheses"
    return hypotheses

I use Gemini 3 Pro for Orchestration. Hypothesis generation is a "read multiple numbers, combine them" task, and Flash produces weaker hypothesis quality than I want at the top of the pipeline. Once per week with ~2,000 tokens out is roughly $1/month even at Pro pricing. Not a place to save money.

Hypothesis quality multiplies through the rest of the loop, so this is the highest-leverage place to spend on model quality.

Experimenter Agent — Remote Config Five-Percent Rollout

The Experimenter takes one hypothesis and launches it as a Firebase Remote Config A/B test. Conditions + Variants is the standard pattern: "this 5% of users gets the new order."

# experimenter.py
from firebase_admin import remote_config
from antigravity import Agent
 
EXPERIMENTER_PROMPT = """You execute AdMob mediation experiments.
Launch the given hypothesis as a Firebase Remote Config A/B test.
 
Hypothesis:
{hypothesis}
 
Generate:
1. Remote Config parameter key and value
2. Conditions (5% rollout)
3. Rollback thresholds (eCPM -20% / ARPU -10%)
"""
 
def launch_experiment(hypothesis: dict) -> dict:
    agent = Agent(
        model="gemini-3-flash",
        system_prompt=EXPERIMENTER_PROMPT.format(
            hypothesis=json.dumps(hypothesis)
        ),
    )
    
    config = json.loads(agent.run("Output Remote Config as JSON").text)
    
    template = remote_config.get_template()
    template.parameters[f"network_order_{hypothesis['format']}"] = \
        remote_config.Parameter(
            default_value=remote_config.ParameterValue(
                value=json.dumps(config["control_value"])
            ),
            conditional_values={
                "experiment_5pct": remote_config.ParameterValue(
                    value=json.dumps(config["variant_value"])
                ),
            },
        )
    
    template.conditions.append(remote_config.Condition(
        name="experiment_5pct",
        expression=f"percent({hypothesis['id']}) <= 5",
        tag_color="GREEN",
    ))
    
    remote_config.publish_template(template)
    return config

Why 5%? With 50M cumulative downloads and ~80K DAU, four experiments at 5% each puts 16,000 users into experiment groups while leaving 84% in control — comfortable sample sizes on both sides.

App-Side Implementation — Swap Ad Order via Remote Config

The iOS and Android apps read Remote Config keys like network_order_banner at launch and reconfigure AdMob mediation order accordingly.

// iOS (Swift) — AdMobMediator.swift
import GoogleMobileAds
import FirebaseRemoteConfig
 
final class AdMobMediator {
    static let shared = AdMobMediator()
    private let remoteConfig = RemoteConfig.remoteConfig()
    
    func setupBanner(rootViewController: UIViewController) -> GADBannerView {
        let order = networkOrder(for: "banner")
        // Mediation lives in AdMob console; we approximate the swap
        // by switching UnitID per pre-configured experiment variant.
        let unitId = adUnitId(for: "banner", order: order)
        
        let bannerView = GADBannerView(adSize: GADAdSizeBanner)
        bannerView.adUnitID = unitId
        bannerView.rootViewController = rootViewController
        bannerView.load(GADRequest())
        return bannerView
    }
    
    private func networkOrder(for format: String) -> [String] {
        let key = "network_order_\(format)"
        guard let json = remoteConfig.configValue(forKey: key).stringValue,
              let data = json.data(using: .utf8),
              let order = try? JSONDecoder().decode([String].self, from: data)
        else {
            return ["admob"]  // fallback
        }
        return order
    }
    
    private func adUnitId(for format: String, order: [String]) -> String {
        let key = "\(format)_\(order.joined(separator: "_"))"
        return adUnitMapping[key] ?? defaultUnitId(format: format)
    }
}

The load-bearing trick is "map to pre-created Unit IDs." AdMob does not expose an API to mutate mediation programmatically, so we approximate it by swapping the Unit ID and pre-creating each variant's Unit ID up front. Once the inventory of Unit IDs is in place, Remote Config alone handles the switch.

Implementing this once each in iOS and Android closes the loop; everything else lives in Antigravity from then on.

Evaluator Agent — Look at ARPU, Not Just eCPM

Seven days later the Evaluator wakes up and compares the four experiments. The most important implementation decision in the whole pipeline is "do not look at eCPM alone."

# evaluator.py
EVALUATOR_PROMPT = """You evaluate AdMob A/B experiment results.
Assess the 4 experiments over the 7-day period.
 
Experiment data:
{experiments_data}
 
Rules (important):
- Roll back if ARPU drops 5%+ even if eCPM is up 5%+
- Roll back if fill rate drops below 90%, no matter how good eCPM looks
- Statistical significance: require p < 0.05 over 16,000 users x 7 days
- Roll out only experiments with confirmed improvement
 
Output (JSON):
{
  "results": [
    {
      "experiment_id": "exp_001",
      "outcome": "rollout" | "rollback" | "extend",
      "rationale": "eCPM +8%, ARPU +6%, fill rate 92% — net positive",
      "next_action": "Expand experiment_5pct condition to 100%",
    },
    ...
  ]
}
"""
 
def evaluate(experiments: list[dict]) -> list[dict]:
    data = []
    for exp in experiments:
        metrics = fetch_experiment_metrics(exp["id"], days=7)
        data.append({"experiment": exp, "metrics": metrics})
    
    agent = Agent(
        model="gemini-3-pro",
        system_prompt=EVALUATOR_PROMPT.format(
            experiments_data=json.dumps(data, indent=2)
        ),
    )
    
    return json.loads(agent.run("Output per-experiment decisions as JSON").text)["results"]

Chasing eCPM alone produces the classic disaster: "high-paying ads are firing, but fill rate dropped, total impressions dropped, revenue dropped." I learned this manually. In 2024 a change that pushed eCPM +20% looked like a clear win until fill rate slid from 95% to 80% and ARPU ended -8%. Net monthly cost: a few thousand dollars.

Since then I evaluate eCPM x fill rate ≈ ARPU. The Evaluator's rules encode that explicitly.

Auto-Rollback — The Safety Net

When the Evaluator decides "rollback," it reverts Remote Config immediately. Force Refresh propagates to all clients within an hour.

def rollback(experiment_id: str):
    template = remote_config.get_template()
    
    template.conditions = [
        c for c in template.conditions if c.name != f"experiment_{experiment_id}"
    ]
    
    for key, param in template.parameters.items():
        if f"experiment_{experiment_id}" in param.conditional_values:
            del param.conditional_values[f"experiment_{experiment_id}"]
    
    remote_config.publish_template(template)
    print(f"Rolled back {experiment_id}")

"Can roll back instantly" is the psychological backbone for running experiments aggressively. In the manual era, every bad change came with the cost of "half a day to revert," which made me operate conservatively. Auto-rollback lowers the cost of "let me just try" — and improvement total = experiment count x expected value, so what really matters is being able to run more experiments, not running them more cautiously.

Four Gotchas Worth Knowing for Parallel-Agent Operation

Things that bit during implementation.

First, keep parallel experiments independent. Four Experimenters fighting over the same Remote Config parameter cancel each other's effects. Allocate one parameter per experiment (network_order_banner vs network_order_interstitial) to avoid collisions.

Second, AdMob Reporting API data lag. The realtime endpoint has small lag but is not aggregated for daily comparison; the historical endpoint lags up to 48 hours. Schedule the Evaluator for 2 days after experiment end.

Third, Gemini API rate limits. Four agents in parallel — especially the Evaluator running the same prompt in succession — hit 429s readily. Cap concurrent calls with asyncio.Semaphore(2) and implement exponential backoff for retries.

Fourth, fill rate below 90% is a sign of failure, not a tradeoff to negotiate. The "rollback if below 90%" rule comes from production experience: drops below this threshold mean more ad-less sessions, which damages the non-ad UX. Banner / interstitial / rewarded each warrant their own thresholds.

Outcomes After Six Months of Operation

Quantitative comparison, before vs after:

• Monthly avg eCPM: $2.10 → $2.65 (+26%)
• Monthly avg ARPU: $0.14 → $0.18 (+29%)
• A/B experiment count: 2-3/month manual → 16/month automated
• Time per experiment: 4-6 hours → 5 minutes (review only)

A 29% ARPU lift at this download volume translates to mid-single-digit thousands of dollars per month annualized. In the manual era I throttled experimentation out of "what if this one regresses" anxiety; with auto-rollback I can afford to try more, and capture more wins on average.

Equally valuable: winning patterns become documented. The Evaluator's rationale field captures the reasoning behind every decision, so over months a pattern library builds up. Reviewing six months of logs surfaced rules like "in Japan, raising the Meta Audience Network floor from $0.30 to $0.40 reliably grows banner eCPM" — knowledge that previously lived only as a hazy hunch.

What I Plan to Add Next

Three open items.

First, per-country optimization automation. Today the loop centers on Japan; US and Southeast Asia are still coarsely tuned. Splitting Orchestrators per country and running them in parallel is in design.

Second, integration with non-ad revenue. StoreKit 2 in-app purchase and affiliate revenue should join the AdMob report so I can quantify tradeoffs like "stronger ads → lower IAP conversion."

Third, LLM output traceability. Persisting the inter-agent communication logs would let me replay "why did the Evaluator recommend rollback?" later. Compliance pressure is low for indie ops, but it raises the quality of my own decision-making to be able to audit it.

AdMob optimization is unglamorous, but compounding a +29% ARPU over six months is far from trivial. I hope these notes help other solo developers running app portfolios. The patterns here generalize beyond mediation — IAP pricing A/Bs and onboarding A/Bs use exactly the same Antigravity skeleton.