Running iOS Push Notification A/B Tests Weekly With Antigravity Agent — A Self-Improving Loop For Copy, Timing, And Segments

Bridging FCM And Antigravity Agent

The FCM HTTP v1 API needs an OAuth2 access token via a service account, which is a bit fiddly to invoke from the agent directly. I push that complexity into a Cloudflare Worker.

Before, my single-developer script kept the service account key in a local env file and called FCM straight from a script:

// Before: a hobby-grade script
import { JWT } from "google-auth-library";
 
const key = require("./service-account.json"); // stored in local env
const jwt = new JWT({
  email: key.client_email,
  key: key.private_key,
  scopes: ["https://www.googleapis.com/auth/firebase.messaging"],
});
 
async function sendPush(token: string, title: string, body: string) {
  const { access_token } = await jwt.authorize();
  await fetch(
    `https://fcm.googleapis.com/v1/projects/${PROJECT_ID}/messages:send`,
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${access_token}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ message: { token, notification: { title, body } } }),
    },
  );
}

Three problems with this. The key is tied to one machine. There is no retry path or rate control. And nothing stops an agent from blasting everyone if it later learns this endpoint exists.

After, the bridge lives in a Worker and exposes only a "schedule" verb to the agent:

// After: the Cloudflare Worker (excerpt)
import { Hono } from "hono";
 
const app = new Hono<{ Bindings: Env }>();
 
app.post("/api/push/schedule", async (c) => {
  const auth = c.req.header("Authorization");
  if (auth !== `Bearer ${c.env.AGENT_TOKEN}`) {
    return c.json({ ok: false, reason: "unauthorized" }, 401);
  }
 
  const payload = await c.req.json();
  const { app_id, segment, title, body, scheduled_at, ab_arm } = payload;
 
  // Guard 1: per-app daily cap
  const todayCount = await c.env.PUSH_KV.get(`count:${app_id}:${todayKey()}`);
  if (Number(todayCount ?? 0) >= MAX_PUSH_PER_DAY[app_id]) {
    return c.json({ ok: false, reason: "daily_cap_exceeded" }, 429);
  }
 
  // Guard 2: schedule-only, reject immediate sends
  const lead = new Date(scheduled_at).getTime() - Date.now();
  if (lead < 30 * 60 * 1000) {
    return c.json({ ok: false, reason: "lead_time_too_short" }, 400);
  }
 
  await c.env.PUSH_DB.prepare(
    `INSERT INTO scheduled_pushes (app_id, segment, title, body, scheduled_at, ab_arm, status)
     VALUES (?, ?, ?, ?, ?, ?, 'pending')`,
  )
    .bind(app_id, segment, title, body, scheduled_at, ab_arm)
    .run();
 
  return c.json({ ok: true });
});
 
export default app;

The two guards are the heart of the design. Even if Antigravity decides "push more," the worker physically refuses to let it. Lead time below thirty minutes is rejected so an accidental immediate dispatch cannot happen.

Keep The MCP Surface Small

The MCP that Antigravity calls accepts only six fields. I deliberately do not expose user lists, token lookups, or worker internals. The agent is allowed to write copy and queue a send; everything else lives behind the worker.

Prompting Antigravity For Copy

Copy generation prompts live in YAML files per app. The agent reads them at the start of each run. Three things matter:

First, an explicit worldview line. For the wallpaper apps it reads "quiet, never pushy"; for the wellness apps it reads "gentle, never declarative, matches the reader's pace." Skip this and the agent picks up trending social media voice and drifts toward clickbait.

Second, include both winning and losing copy from the last twelve weeks. Showing only winners narrows the search; showing five winners and five losers and asking for "five candidates outside both buckets" produces the most interesting angles.

Third, declare the success metric and the banned phrases explicitly. The primary metric is "total active seconds within sixty minutes of open," not open rate. Banned phrases include "now," "limited," and "last chance," because optimizing for them works short-term and erodes trust.

# prompts/wallpaper-app/push-generation.yaml
app_world:
  tone: "quiet, never pushy"
  brand_voice: "evoke the urge to softly change a wallpaper at night"
  avoid:
    - "urgency words like now / limited / last chance"
    - "exaggerated qualifiers like god-tier / amazing"
    - "emoji overuse"
 
evaluation:
  primary: "sum of active_time_sec within 60 minutes of tap"
  secondary: "D7 retention"
  tertiary: "AdMob impressions x eCPM"
  hard_caps:
    - "max one push per user per day"
    - "no sends between 23:00-08:00 local"
 
ab_test:
  arms_per_week: 4
  min_sample_per_arm: 3000

The agent reads this and the previous twelve weeks of champions_losers.json, then proposes four arms. Output lands in a GitHub PR, I review, merge, and the worker picks up the new YAML on its next cron tick. The PR trail means I can later go back and ask "why was that week's copy strong?"

A Small Prompting Habit That Helps

One prompt shape I now lean on: "for the cohort where D7 is down 2 points or more year over year, give me three short lines that read fine even if the user does not tap." Naming the cohort and giving permission to not be tapped keeps the agent from drifting toward open-rate maximization.

Segment Design And Timing

Do not over-segment. After several attempts I settled on five buckets: D0-D7, D8-D30, D31-D90, D91+, and paying users. Cutting finer leaves single arms with too few users to draw a conclusion.

Timing is local. About 70% of my users are in Japan, but 25% are English-speaking and the rest scattered. Sending in UTC means American users get pinged at 3 AM. The worker picks each user's local hour before dispatching:

-- Cron picks up the queue and joins users on segment + local time
SELECT
  u.fcm_token,
  s.title,
  s.body,
  s.ab_arm
FROM scheduled_pushes s
JOIN users u
  ON u.segment = s.segment
WHERE s.status = 'pending'
  AND s.scheduled_at <= datetime('now', '+5 minutes')
  AND (
    CAST(strftime('%H', datetime('now', u.tz_offset || ' hours')) AS INTEGER)
    BETWEEN 19 AND 22
  )
LIMIT 5000;

The single most surprising lesson: for wellness apps the strongest window was consistently around 21:30 local, presumably the pre-sleep moment. For wallpaper apps it sat at 18:30-19:30, the post-commute slice. App worldview and send time correlate strongly, so when the agent suggests a late-night window for a wellness app I reject it on instinct.

Measuring Performance In BigQuery

Stream Analytics into BigQuery and you can answer the arm-by-arm questions via SQL. The agent has read-only BigQuery access and a weekly cron runs roughly this query:

WITH push_events AS (
  SELECT
    user_pseudo_id,
    event_timestamp,
    (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'ab_arm') AS ab_arm,
    (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'segment') AS segment
  FROM `analytics_xxx.events_*`
  WHERE event_name = 'push_received'
    AND _TABLE_SUFFIX BETWEEN
      FORMAT_DATE('%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY))
      AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
),
opens AS (
  SELECT user_pseudo_id, event_timestamp AS open_ts
  FROM `analytics_xxx.events_*`
  WHERE event_name = 'push_opened'
)
SELECT
  p.ab_arm,
  p.segment,
  COUNT(DISTINCT p.user_pseudo_id) AS sent_users,
  COUNT(DISTINCT o.user_pseudo_id) AS opened_users,
  SAFE_DIVIDE(COUNT(DISTINCT o.user_pseudo_id), COUNT(DISTINCT p.user_pseudo_id)) AS open_rate
FROM push_events p
LEFT JOIN opens o
  ON o.user_pseudo_id = p.user_pseudo_id
  AND o.open_ts BETWEEN p.event_timestamp AND p.event_timestamp + 3600 * 1000000
GROUP BY p.ab_arm, p.segment
ORDER BY open_rate DESC;

SAFE_DIVIDE matters; you do not want the table to fall over when an arm got zero opens. More importantly, I always report open_rate alongside post_open_60m_active_seconds. Open rate alone is a trap.

Minimum Sample Discipline

Three thousand sends per arm before declaring a winner. Below that, a friendly segment can carry a noisy result. I encode this rule in the YAML so the agent does not over-claim.

A Push Outage Of My Own Making — Losing 4.8 Points Of D7

The first month I ran this loop I made a clean mistake. The agent's copy was winning on open rate, so I raised the cap from "one per user per day" to "two." Short-term open rate climbed. Three weeks later D7 retention was down 4.8 points across the app.

Cause: the second daily push slipped into late-evening windows and a chunk of users disabled notifications outright in iOS settings. AdMob impressions fell about 12% and roughly thirty thousand JPY of monthly revenue evaporated.

Recovery looked like this:

1. Reset MAX_PUSH_PER_DAY to 1 in the worker and ship within the hour.
2. Pause sends to anyone who had received two pushes in the previous fourteen days for one full week.
3. Show a quiet in-app banner — never a popup — explaining how to re-enable notifications. No urgency language.
4. Rewrite the Antigravity prompt to state explicitly: "the hard cap exists to protect D7 retention; never propose raising it."

After that I made two structural changes I now treat as load-bearing: short-term and medium-term KPIs always travel together to the agent, and hard caps live in the worker, not in the prompt. The agent is good at short-term optimization. It is the human's job to install the structures that surface medium-term decay.

The Weekly Cadence — What The Agent Does And What I Do

Monday morning: Antigravity ships a Slack digest summarizing last week's arms, declaring winners and losers, and proposing five candidates for the coming week. This is fully scripted via cron.

I spend roughly thirty minutes reviewing. I check whether the winners respect the worldview, whether D7 is holding, and whether AdMob is moving in the expected direction.

I narrow the five candidates to four, occasionally rewriting one by hand, then open a GitHub PR. Merging it lets the worker enqueue next week's sends in sequence.

End of month: I ask Antigravity for "trends across the past four weeks and segment ideas worth trying next month." This is scaffolding for my own thinking; I always make the final call.

Three rules are non-negotiable: dispatch, KPI definition, and hard caps stay with the human. The agent owns copy candidates and hypothesis generation. The moment that boundary blurs, the cap-relaxation story above repeats.

Areas I Have Not Touched Yet, And What I Want To Try Next

A few things on my own list, mostly as a way of closing this article rather than padding it.

First, per-user recommendation pushes. Today I send by segment; Firestore already has each user's recent wallpaper history, so I could ask Antigravity for short copy that hints at a new release in a category the user has opened five-plus times in thirty days. Privacy framing matters — whether to say "based on what you have been viewing" or not is a design question, not a technical one.

Second, silent push for pre-load. The idea is to pre-fetch the next wallpaper to the device a few minutes before the visible push fires, so the opening tap feels instant. The agent's job here shifts from copy to batch planning.

Third, AdMob-aware send volume. Weekend eCPM tends to dip, so concentrating sends on weekday evenings might smooth monthly revenue. I expect to run this experiment this quarter.

I hope this is useful if you are sketching an agent-assisted push loop of your own. Push is a small surface where worldview must fit into one line, and that constraint is why I think agents shine here in tandem with a careful human reviewer. Thanks for reading.