Running iOS Push Notification A/B Tests Weekly With Antigravity Agent — A Self-Improving Loop For Copy, Timing, And Segments

Bridging FCM And Antigravity Agent

The FCM HTTP v1 API needs an OAuth2 access token via a service account, which is a bit fiddly to invoke from the agent directly. I push that complexity into a Cloudflare Worker.

Before, my single-developer script kept the service account key in a local env file and called FCM straight from a script:

// Before: a hobby-grade script
import { JWT } from "google-auth-library";
 
const key = require("./service-account.json"); // stored in local env
const jwt = new JWT({
  email: key.client_email,
  key: key.private_key,
  scopes: ["https://www.googleapis.com/auth/firebase.messaging"],
});
 
async function sendPush(token: string, title: string, body: string) {
  const { access_token } = await jwt.authorize();
  await fetch(
    `https://fcm.googleapis.com/v1/projects/${PROJECT_ID}/messages:send`,
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${access_token}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ message: { token, notification: { title, body } } }),
    },
  );
}

Three problems with this. The key is tied to one machine. There is no retry path or rate control. And nothing stops an agent from blasting everyone if it later learns this endpoint exists.

After, the bridge lives in a Worker and exposes only a "schedule" verb to the agent:

// After: the Cloudflare Worker (excerpt)
import { Hono } from "hono";
 
const app = new Hono<{ Bindings: Env }>();
 
app.post("/api/push/schedule", async (c) => {
  const auth = c.req.header("Authorization");
  if (auth !== `Bearer ${c.env.AGENT_TOKEN}`) {
    return c.json({ ok: false, reason: "unauthorized" }, 401);
  }
 
  const payload = await c.req.json();
  const { app_id, segment, title, body, scheduled_at, ab_arm } = payload;
 
  // Guard 1: per-app daily cap
  const todayCount = await c.env.PUSH_KV.get(`count:${app_id}:${todayKey()}`);
  if (Number(todayCount ?? 0) >= MAX_PUSH_PER_DAY[app_id]) {
    return c.json({ ok: false, reason: "daily_cap_exceeded" }, 429);
  }
 
  // Guard 2: schedule-only, reject immediate sends
  const lead = new Date(scheduled_at).getTime() - Date.now();
  if (lead < 30 * 60 * 1000) {
    return c.json({ ok: false, reason: "lead_time_too_short" }, 400);
  }
 
  await c.env.PUSH_DB.prepare(
    `INSERT INTO scheduled_pushes (app_id, segment, title, body, scheduled_at, ab_arm, status)
     VALUES (?, ?, ?, ?, ?, ?, 'pending')`,
  )
    .bind(app_id, segment, title, body, scheduled_at, ab_arm)
    .run();
 
  return c.json({ ok: true });
});
 
export default app;

The two guards are the heart of the design. Even if Antigravity decides "push more," the worker physically refuses to let it. Lead time below thirty minutes is rejected so an accidental immediate dispatch cannot happen.

Keep The MCP Surface Small

The MCP that Antigravity calls accepts only six fields. I deliberately do not expose user lists, token lookups, or worker internals. The agent is allowed to write copy and queue a send; everything else lives behind the worker.

Prompting Antigravity For Copy

Copy generation prompts live in YAML files per app. The agent reads them at the start of each run. Three things matter:

First, an explicit worldview line. For the wallpaper apps it reads "quiet, never pushy"; for the wellness apps it reads "gentle, never declarative, matches the reader's pace." Skip this and the agent picks up trending social media voice and drifts toward clickbait.

Second, include both winning and losing copy from the last twelve weeks. Showing only winners narrows the search; showing five winners and five losers and asking for "five candidates outside both buckets" produces the most interesting angles.

Third, declare the success metric and the banned phrases explicitly. The primary metric is "total active seconds within sixty minutes of open," not open rate. Banned phrases include "now," "limited," and "last chance," because optimizing for them works short-term and erodes trust.

# prompts/wallpaper-app/push-generation.yaml
app_world:
  tone: "quiet, never pushy"
  brand_voice: "evoke the urge to softly change a wallpaper at night"
  avoid:
    - "urgency words like now / limited / last chance"
    - "exaggerated qualifiers like god-tier / amazing"
    - "emoji overuse"
 
evaluation:
  primary: "sum of active_time_sec within 60 minutes of tap"
  secondary: "D7 retention"
  tertiary: "AdMob impressions x eCPM"
  hard_caps:
    - "max one push per user per day"
    - "no sends between 23:00-08:00 local"
 
ab_test:
  arms_per_week: 2
  min_sample_per_arm: 3000

The agent reads this and the previous twelve weeks of champions_losers.json, then proposes four arms. Output lands in a GitHub PR, I review, merge, and the worker picks up the new YAML on its next cron tick. The PR trail means I can later go back and ask "why was that week's copy strong?"

A Small Prompting Habit That Helps

One prompt shape I now lean on: "for the cohort where D7 is down 2 points or more year over year, give me three short lines that read fine even if the user does not tap." Naming the cohort and giving permission to not be tapped keeps the agent from drifting toward open-rate maximization.

Segment Design And Timing

Do not over-segment. After several attempts I settled on five buckets: D0-D7, D8-D30, D31-D90, D91+, and paying users. Cutting finer leaves single arms with too few users to draw a conclusion.

Timing is local. About 70% of my users are in Japan, but 25% are English-speaking and the rest scattered. Sending in UTC means American users get pinged at 3 AM. The worker picks each user's local hour before dispatching:

-- Cron picks up the queue and joins users on segment + local time
SELECT
  u.fcm_token,
  s.title,
  s.body,
  s.ab_arm
FROM scheduled_pushes s
JOIN users u
  ON u.segment = s.segment
WHERE s.status = 'pending'
  AND s.scheduled_at <= datetime('now', '+5 minutes')
  AND (
    CAST(strftime('%H', datetime('now', u.tz_offset || ' hours')) AS INTEGER)
    BETWEEN 19 AND 22
  )
LIMIT 5000;

The single most surprising lesson: for wellness apps the strongest window was consistently around 21:30 local, presumably the pre-sleep moment. For wallpaper apps it sat at 18:30-19:30, the post-commute slice. App worldview and send time correlate strongly, so when the agent suggests a late-night window for a wellness app I reject it on instinct.

Measuring Performance In BigQuery

Stream Analytics into BigQuery and you can answer the arm-by-arm questions via SQL. The agent has read-only BigQuery access and a weekly cron runs roughly this query:

WITH push_events AS (
  SELECT
    user_pseudo_id,
    event_timestamp,
    (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'ab_arm') AS ab_arm,
    (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'segment') AS segment
  FROM `analytics_xxx.events_*`
  WHERE event_name = 'push_received'
    AND _TABLE_SUFFIX BETWEEN
      FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY))
      AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
),
opens AS (
  SELECT user_pseudo_id, event_timestamp AS open_ts
  FROM `analytics_xxx.events_*`
  WHERE event_name = 'push_opened'
)
SELECT
  p.ab_arm,
  p.segment,
  COUNT(DISTINCT p.user_pseudo_id) AS sent_users,
  COUNT(DISTINCT o.user_pseudo_id) AS opened_users,
  SAFE_DIVIDE(COUNT(DISTINCT o.user_pseudo_id), COUNT(DISTINCT p.user_pseudo_id)) AS open_rate
FROM push_events p
LEFT JOIN opens o
  ON o.user_pseudo_id = p.user_pseudo_id
  AND o.open_ts BETWEEN p.event_timestamp AND p.event_timestamp + 3600 * 1000000
GROUP BY p.ab_arm, p.segment
ORDER BY open_rate DESC;

SAFE_DIVIDE matters; you do not want the table to fall over when an arm got zero opens. More importantly, I always report open_rate alongside post_open_60m_active_seconds. Open rate alone is a trap.

Minimum Sample Discipline

Three thousand sends per arm before declaring a winner. Below that, a friendly segment can carry a noisy result. I encode this rule in the YAML so the agent does not over-claim.

Asked where the three thousand came from, though, I had no answer. I had been shipping four candidates a week and crowning the strongest one for over a year without ever checking how often that crown landed on the right head.

Measuring How Often The Weekly Winner Is Actually The Best Arm

So I wrote a small dependency-free simulator and ran it locally. It assumes an 8% baseline open rate, makes exactly one of the arms genuinely stronger, and counts — across 200,000 trials — how often the arm that looked best really was best. It also records how far the winner's observed rate overstates its true rate, and how often the winner clears a significance bar against the runner-up.

// push-ab-power.mjs — no dependencies. Run with: node push-ab-power.mjs
const TRIALS = 200000, BASE = 0.08;
const rng = ((s) => () => (s ^= s << 13, s >>>= 0, s ^= s >>> 17, s ^= s << 5, s >>>= 0, s / 4294967296))(20260903);
const binom = (n, p) => {
  const u1 = Math.max(rng(), 1e-12), u2 = rng();
  const z = Math.sqrt(-2 * Math.log(u1)) * Math.cos(2 * Math.PI * u2);
  return Math.min(n, Math.max(0, Math.round(n * p + Math.sqrt(n * p * (1 - p)) * z)));
};
const z2 = (x1, n1, x2, n2) => {
  const p = (x1 + x2) / (n1 + n2), se = Math.sqrt(p * (1 - p) * (1 / n1 + 1 / n2));
  return se ? (x1 / n1 - x2 / n2) / se : 0;
};
function run(arms, n, lift) {
  const truth = Array.from({ length: arms }, (_, i) => BASE + (i === 0 ? lift : 0));
  let hit = 0, curse = 0, declared = 0;
  for (let t = 0; t < TRIALS; t++) {
    const obs = truth.map((p) => binom(n, p));
    const [[, best], [, second]] = obs.map((v, i) => [v, i]).sort((a, b) => b[0] - a[0]);
    if (best === 0) hit++;                       // arm 0 is the genuinely better one
    curse += obs[best] / n - truth[best];        // winner's curse
    if (z2(obs[best], n, obs[second], n) >= 1.96) declared++;
  }
  return { pick: hit / TRIALS, curse: (curse / TRIALS) * 100, declare: declared / TRIALS };
}
for (const lift of [0.01, 0.004, 0]) {
  console.log(`\nTrue lift +${(lift * 100).toFixed(1)}pt`);
  for (const arms of [4, 2]) for (const n of [3000, 6000, 12000, 25000]) {
    const r = run(arms, n, lift);
    console.log(`  arms=${arms} n=${String(n).padStart(5)}  correct ${(r.pick * 100).toFixed(1)}%` +
      `  curse +${r.curse.toFixed(2)}pt  declared ${(r.declare * 100).toFixed(1)}%`);
  }
}

The realistic lift from a copy rewrite, in my own data, is somewhere around 0.3 to 0.5 points absolute. Here is what four arms at three thousand sends does with a +0.4pt difference.

Four arms at three thousand sends is right 49.9% of the time. That is a coin toss. Worse, the winner's observed rate overstates its true rate by 0.46 points — larger than the difference I was chasing. When that champion sagged the following week, it was not fatigue in the copy. It had never been the better arm.

The null case is harder to read.

With no real difference at all, the weekly winner still looks 0.51 points better. My digest was formatted so that a week of no progress still read like progress. I may be missing something, but the agent was not the problem here and neither was the copy — the selection rule itself was built to harvest noise.

Read the first table sideways and the fix appears. Doubling to six thousand sends per arm only reaches 60.2%, while dropping to two candidates gets 72.0% at the original volume. More sends costs your users attention; fewer candidates costs only me.

Send fewer variants before you send more notifications. If accuracy can come from my side of the loop rather than from the people who opened my app, I would rather take it from my side.

Rewriting The Rules To Match The Measurement

Three changes came out of this. Arms per week went from four to two. A winner now has to clear a two-proportion z of 1.96 against the runner-up, and when it does not, the reigning champion simply stays. And Antigravity is told explicitly that "no winner this week" is a valid answer.

ab_test:
  arms_per_week: 2          # was 4 — same volume, 49.9% -> 72.0% correct picks
  min_sample_per_arm: 3000
  decision:
    rule: "two_proportion_z"
    min_z_vs_runner_up: 1.96
    on_tie: "keep_champion"
  report:
    must_include: ["observed delta", "z value", "explicit no-winner call"]

The decision code changed from a one-line max to something that can return a tie.

type Arm = { id: string; sent: number; opened: number };
 
export function declareWinner(arms: Arm[], champion: string): { winner: string; tie: boolean } {
  const sorted = [...arms].sort((a, b) => b.opened / b.sent - a.opened / a.sent);
  const [top, next] = sorted;
  const p = (top.opened + next.opened) / (top.sent + next.sent);
  const se = Math.sqrt(p * (1 - p) * (1 / top.sent + 1 / next.sent));
  const z = se ? (top.opened / top.sent - next.opened / next.sent) / se : 0;
 
  // No significant gap, no new winner. The champion stays.
  return z >= 1.96 ? { winner: top.id, tie: false } : { winner: champion, tie: true };
}

As the significance column predicts, even two arms at three thousand sends only produce a callable winner about once every twelve weeks. That felt thin at first. Then I noticed that the other eleven weeks were not weeks of indecision — they were weeks I did not buy noise.

A Push Outage Of My Own Making — Losing 4.8 Points Of D7

The first month I ran this loop I made a clean mistake. The agent's copy was winning on open rate, so I raised the cap from "one per user per day" to "two." Short-term open rate climbed. Three weeks later D7 retention was down 4.8 points across the app.

Cause: the second daily push slipped into late-evening windows and a chunk of users disabled notifications outright in iOS settings. AdMob impressions fell about 12% and roughly thirty thousand JPY of monthly revenue evaporated.

Recovery looked like this:

1. Reset MAX_PUSH_PER_DAY to 1 in the worker and ship within the hour.
2. Pause sends to anyone who had received two pushes in the previous fourteen days for one full week.
3. Show a quiet in-app banner — never a popup — explaining how to re-enable notifications. No urgency language.
4. Rewrite the Antigravity prompt to state explicitly: "the hard cap exists to protect D7 retention; never propose raising it."

After that I made two structural changes I now treat as load-bearing: short-term and medium-term KPIs always travel together to the agent, and hard caps live in the worker, not in the prompt. The agent is good at short-term optimization. It is the human's job to install the structures that surface medium-term decay.

The Weekly Cadence — What The Agent Does And What I Do

Monday morning: Antigravity ships a Slack digest summarizing last week's arms — naming a winner only when it clears the significance bar, otherwise keeping the champion — and proposing five candidates for the coming week. This is fully scripted via cron.

I spend roughly thirty minutes reviewing. I check whether the winners respect the worldview, whether D7 is holding, and whether AdMob is moving in the expected direction.

I narrow the five candidates to two, occasionally rewriting one by hand, then open a GitHub PR. Narrowing is harder work than it used to be, and the measurement says it is the part that pays. Merging it lets the worker enqueue next week's sends in sequence.

End of month: I ask Antigravity for "trends across the past four weeks and segment ideas worth trying next month." This is scaffolding for my own thinking; I always make the final call.

Three rules are non-negotiable: dispatch, KPI definition, and hard caps stay with the human. The agent owns copy candidates and hypothesis generation. The moment that boundary blurs, the cap-relaxation story above repeats.

Areas I Have Not Touched Yet, And What I Want To Try Next

A few things on my own list, mostly as a way of closing this article rather than padding it.

First, per-user recommendation pushes. Today I send by segment; Firestore already has each user's recent wallpaper history, so I could ask Antigravity for short copy that hints at a new release in a category the user has opened five-plus times in thirty days. Privacy framing matters — whether to say "based on what you have been viewing" or not is a design question, not a technical one.

Second, silent push for pre-load. The idea is to pre-fetch the next wallpaper to the device a few minutes before the visible push fires, so the opening tap feels instant. The agent's job here shifts from copy to batch planning.

Third, AdMob-aware send volume. Weekend eCPM tends to dip, so concentrating sends on weekday evenings might smooth monthly revenue. I expect to run this experiment this quarter.

I hope this is useful if you are sketching an agent-assisted push loop of your own. Push is a small surface where worldview must fit into one line, and that constraint is why I think agents shine here in tandem with a careful human reviewer. Thanks for reading.