Antigravity × Gemma 4: Building an On-Prem PII Masking Agent — Production Design That Keeps Sensitive Data In-House

Layer 1: Regex Prescreen That Holds Up in Production

The first job is to wipe out everything you can decide deterministically. The design rule here is "tolerate over-detection so long as you never miss." The LLM downstream will sanity-check the context anyway, so a few extra hits are cheap insurance.

# pii_prescreen.py — deterministic PII detector
# Behavior: locate every span we can flag with confidence
# so the downstream LLM never even sees them as raw PII.
 
import re
from dataclasses import dataclass
from typing import Iterable
 
@dataclass(frozen=True)
class PIIMatch:
    kind: str          # "email" | "phone_jp" | "credit_card" | "ipv4"
    start: int
    end: int
    value: str
 
# Email — tolerant but practical (we accept some over-matching)
EMAIL_RE = re.compile(
    r"[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}"
)
 
# Japanese phone numbers (2–5 digit area codes, hyphen optional)
PHONE_JP_RE = re.compile(
    r"(?<!\d)(?:0\d{1,4}[-(\s]?\d{1,4}[-)\s]?\d{3,4})(?!\d)"
)
 
# Credit cards — accept candidates here, validate Luhn after
CREDIT_RE = re.compile(r"(?<!\d)(?:\d[ -]?){13,19}(?!\d)")
 
IPV4_RE = re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b")
 
def luhn_valid(num: str) -> bool:
    """Quick Luhn check on a candidate card-number string."""
    digits = [int(c) for c in num if c.isdigit()]
    if not 13 <= len(digits) <= 19:
        return False
    checksum = 0
    for i, d in enumerate(reversed(digits)):
        if i % 2 == 1:
            d *= 2
            if d > 9:
                d -= 9
        checksum += d
    return checksum % 10 == 0
 
def find_pii(text: str) -> Iterable[PIIMatch]:
    for m in EMAIL_RE.finditer(text):
        yield PIIMatch("email", m.start(), m.end(), m.group())
    for m in PHONE_JP_RE.finditer(text):
        yield PIIMatch("phone_jp", m.start(), m.end(), m.group())
    for m in CREDIT_RE.finditer(text):
        if luhn_valid(m.group()):
            yield PIIMatch("credit_card", m.start(), m.end(), m.group())
    for m in IPV4_RE.finditer(text):
        yield PIIMatch("ipv4", m.start(), m.end(), m.group())
 
if __name__ == "__main__":
    sample = "Reach me at ichiro@example.co.jp or 090-1234-5678. Test card: 4111 1111 1111 1111"
    for m in find_pii(sample):
        print(f"{m.kind}: {m.value!r} at [{m.start}:{m.end}]")
    # Expected output:
    # email: 'ichiro@example.co.jp' at [12:32]
    # phone_jp: '090-1234-5678' at [36:49]
    # credit_card: '4111 1111 1111 1111' at [62:81]

Why is Luhn validation here and not in the regex itself? Long digit strings show up in legitimate places — order numbers, tracking IDs, internal product codes. Treating only Luhn-valid candidates as "definitely PII" cut my false-positive rate by an order of magnitude. That decision came from an outage where masking an order number broke a support reply. Real production tells you which corners actually need cutting.

Layer 2: Gemma 4 for Context-Aware Entity Extraction

This is the heart of the pipeline. Gemma 4 runs inside the Antigravity-managed environment and extracts person names, locations, and organizations.

# gemma_ner.py — Gemma 4 NER extractor
# Behavior: extract context-dependent PII the regex layer
# cannot reach without a model.
 
from llama_cpp import Llama
import json
from typing import List, TypedDict
 
class Entity(TypedDict):
    kind: str   # "person" | "location" | "org"
    text: str
    confidence: float
 
NER_PROMPT = """\
Extract named entities from the text below.
Output JSON only. Each item: { "kind": "person|location|org", "text": "...", "confidence": 0.0-1.0 } .
If `text` is not a literal substring of the input, do not output it.
 
Text:
\"\"\"
{text}
\"\"\"
 
JSON:"""
 
class GemmaNER:
    def __init__(self, model_path: str, n_ctx: int = 8192):
        self.llm = Llama(
            model_path=model_path,
            n_ctx=n_ctx,
            n_threads=8,
            verbose=False,
        )
 
    def extract(self, text: str, min_confidence: float = 0.6) -> List[Entity]:
        if not text.strip():
            return []
        prompt = NER_PROMPT.format(text=text[:4000])  # safety cap
        try:
            resp = self.llm(
                prompt,
                max_tokens=512,
                temperature=0.0,   # reproducibility wins
                top_p=0.95,
                stop=["\n\n"],
            )
            raw = resp["choices"][0]["text"].strip()
            # Extract JSON in case the model adds a preamble
            start = raw.find("[")
            end = raw.rfind("]") + 1
            if start == -1 or end == 0:
                return []
            entities = json.loads(raw[start:end])
        except (json.JSONDecodeError, KeyError, IndexError) as e:
            # Bubble up — never silently swallow in production
            raise RuntimeError(f"NER failed: {e}") from e
 
        # Filter: confidence threshold + literal-substring check (no hallucinations)
        return [
            e for e in entities
            if e.get("confidence", 0) >= min_confidence
            and e.get("text") in text
        ]
 
if __name__ == "__main__":
    ner = GemmaNER("models/gemma-2-7b-it.Q4_K_M.gguf")
    text = "Mr. Masaki Hirokawa from Dolice Inc. will visit the Tokyo office."
    for e in ner.extract(text):
        print(e)
    # Example output:
    # {'kind': 'person', 'text': 'Masaki Hirokawa', 'confidence': 0.92}
    # {'kind': 'org', 'text': 'Dolice Inc.', 'confidence': 0.88}
    # {'kind': 'location', 'text': 'Tokyo office', 'confidence': 0.71}

Three design points to call out. First, temperature=0.0 is non-negotiable. Without reproducibility you cannot test or debug, full stop. Second, the literal-substring check after parsing is your hallucination shield — LLMs occasionally invent entities that never appeared in the input. Third, the 0.6 confidence threshold came from tuning against ~10,000 internal samples to maximize F1. Your domain will tune to a different number; do the measurement.

For Gemma 4 setup itself — quantization choice, model paths, and the Antigravity-side configuration — see the Gemma 4 local LLM production guide. I assume that ground is already set in the code above.

Layer 3: Reversible vs Irreversible Masking — Pick Per Purpose

This is the layer most tutorials wave away. In production you cannot. "Mask it" is a verb that hides at least three different operations.

Reversible masking (tokenization) replaces the value with a token you can decrypt later — Masaki Hirokawa → [PERSON_a8f3]. Useful when a support agent has to call the customer back. Irreversible masking (redaction) destroys the original — useful for analytics or archives where re-identification is a liability. A purpose-aware router on top of both keeps the calling code clean.

# masking_router.py — strategy router by purpose
from cryptography.fernet import Fernet, InvalidToken
from dataclasses import dataclass
 
@dataclass
class MaskedSpan:
    original_kind: str
    placeholder: str
    encrypted: bytes | None  # set only when reversible
 
class MaskingRouter:
    """
    Switches between reversible and irreversible per purpose.
    - 'support': reversible (decrypt later for customer follow-up)
    - 'analytics': irreversible (must not be re-identifiable)
    - 'archive': irreversible + hash for deterministic dedup
    """
    def __init__(self, key: bytes | None = None):
        self.fernet = Fernet(key) if key else None
 
    def mask(self, value: str, kind: str, purpose: str) -> MaskedSpan:
        if purpose == "support":
            if self.fernet is None:
                raise RuntimeError("Reversible masking requires a key")
            token = self.fernet.encrypt(value.encode("utf-8"))
            placeholder = f"[{kind.upper()}_{token[-8:].decode('utf-8', 'ignore')}]"
            return MaskedSpan(kind, placeholder, token)
        elif purpose == "analytics":
            placeholder = f"[{kind.upper()}_REDACTED]"
            return MaskedSpan(kind, placeholder, None)
        elif purpose == "archive":
            import hashlib
            digest = hashlib.sha256(value.encode("utf-8")).hexdigest()[:12]
            return MaskedSpan(kind, f"[{kind.upper()}_{digest}]", None)
        else:
            raise ValueError(f"unknown purpose: {purpose}")
 
    def unmask(self, span: MaskedSpan) -> str:
        if span.encrypted is None or self.fernet is None:
            raise InvalidToken("This span is irreversible — cannot decrypt")
        return self.fernet.decrypt(span.encrypted).decode("utf-8")
 
if __name__ == "__main__":
    key = Fernet.generate_key()
    router = MaskingRouter(key)
    span = router.mask("Masaki Hirokawa", "person", "support")
    print(span.placeholder)         # e.g. [PERSON_xY3kFq8M]
    print(router.unmask(span))      # Masaki Hirokawa

The obvious question is "where do I keep the key?" In my setup, the key lives in an HSM (hardware security module), and decryption only happens via a dedicated, audit-logged API. A key sitting next to the application code is, strictly speaking, not really masking. Build the key separation in from day one — retrofitting it later is twice the work.

Layer 4: Audit Logging So You Can Answer SOC2 in Real Time

The audit log is the part most teams underinvest in. "When, who, what, why" of every mask and unmask must land in WORM (Write Once Read Many) storage, otherwise the compliance team will block the next renewal. I have had a contract get held up for exactly this reason. Twice.

# audit_log.py — WORM-style append log
import json
import time
import hashlib
from pathlib import Path
 
class WORMAuditLog:
    """
    Append-only log. Each entry includes the previous entry's hash,
    so any silent edit breaks the chain (mini blockchain).
    """
    def __init__(self, path: Path):
        self.path = Path(path)
        self.path.parent.mkdir(parents=True, exist_ok=True)
 
    def _last_hash(self) -> str:
        if not self.path.exists() or self.path.stat().st_size == 0:
            return "0" * 64
        with self.path.open("rb") as f:
            f.seek(0, 2)
            size = f.tell()
            f.seek(max(0, size - 4096))
            tail = f.read().decode("utf-8", errors="ignore").strip().splitlines()
            if not tail:
                return "0" * 64
            return json.loads(tail[-1])["hash"]
 
    def append(self, event: dict) -> str:
        prev = self._last_hash()
        entry = {
            "ts": time.time(),
            "prev": prev,
            "event": event,
        }
        payload = json.dumps(entry, separators=(",", ":"), sort_keys=True)
        digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()
        line = json.dumps({**entry, "hash": digest}, ensure_ascii=False)
        with self.path.open("a", encoding="utf-8") as f:
            f.write(line + "\n")
        return digest
 
if __name__ == "__main__":
    log = WORMAuditLog(Path("/var/log/pii_audit.jsonl"))
    log.append({"action": "mask", "kind": "person", "purpose": "support"})
    log.append({"action": "unmask", "user": "support_ticket_42"})

Hash-chaining means tampering ("let me just delete this one entry") breaks the chain visibly. In production I pair this with S3 Object Lock or GCS retention policies, but the file-append version is enough to start. Don't try to land the perfect cloud WORM setup on day one — graduate from local append, harden once the rest of the pipeline is stable.

Three Production Pitfalls I Actually Hit

First, "Gemma 4 sometimes folds Japanese titles into the person entity." Senior Director Tanaka-san came out as a single person span, so the masked output read [PERSON_xxx]-san and the title Senior Director got vaporized. Context broke. Fix: a small post-processing step strips a known title list (CEO, CTO, Director, Manager, etc.) off detected person spans. The general pattern — clean the LLM output with light deterministic rules — applies everywhere in production.

Second, "quoted thread blocks get masked twice." When > Mr. Masaki Hirokawa appears in a reply, the pipeline treated the quoted line as a fresh input and assigned a different placeholder than the original message. Same person, two tokens, broken thread. Fix: split off quoted blocks (lines starting with >) before the pipeline, and keep a per-thread cache that reuses the original placeholder on subsequent appearances.

Third, "the LoRA inference cost estimate was wrong." I sized 7B + LoRA at 70 ms based on short benchmark inputs. Real-world emails are longer, and average latency landed at 220 ms. Even with parallelism on Manager Surface, that is 3× the budget. I switched the email path from "real-time mask" to "near-sync within 30 s" and moved to batch processing. If you only measure latency on toy inputs, your future self will pay for it.

Throughput vs Latency — How to Decide on Parallelism

PII masking design splits cleanly along one question: do you need streaming? If a chatbot has to mask in-line during a streaming response, run Layer 2 on Gemma 4 7B with Q4_K_M quantization, and pipeline the filter alongside the response stream. If the work is batch-style — overnight email classification — Gemma 4 27B with bigger context wins on accuracy and the cost is fine.

In my setup, the real-time path (chat) and the async path (email) use different models and different pipelines. The pull to unify them is real, but the SLOs and cost shapes are different enough that unification produces two mediocre paths instead of one good one. Owning the split is the inflection point where the operation stops being painful.

For the parallelism implementation and the Manager Surface job-queue design, see Antigravity parallel agents production architecture. It pairs naturally with this article and gives you the orchestration layer this pipeline plugs into.

Cost Modeling — What This Pipeline Actually Spends

A premium guide that ducks the cost conversation is incomplete. Here is the back-of-envelope math I run for clients, with the numbers I see on my own infrastructure.

A single 7B Gemma 4 instance at Q4_K_M quantization fits comfortably in 24 GB of GPU VRAM (an L4 or RTX A5000 class card). At the average input size I see — roughly 600 tokens — the median throughput is around 35 requests per second per GPU, with tail latency at p99 of about 280 ms. Layer 1 (regex) runs on CPU and is essentially free. Layer 3 (masking router) is also CPU-bound and finishes in under 5 ms even at high load. Layer 4 (audit log) is bottlenecked by disk fsync, which on a modern NVMe is well under 2 ms.

Translate that into monthly numbers and the picture is striking. For a workload of one million masked documents per day, you need about three GPUs running 24/7 to absorb peak load, plus a small CPU fleet for the surrounding layers. Compare that to sending the same volume to a leading cloud LLM at standard input/output token pricing and the cloud option is between three and five times more expensive — before you factor in egress, audit infrastructure, and the legal cost of handling a data-residency exception.

The trade-off, of course, is upfront capital and operational complexity. You own the GPU cluster. You own the patching. You own the on-call rotation. For a single-developer SaaS, this might be the wrong economics. For a regulated B2B SaaS where the cloud option is closed off entirely, it is often the only economics that actually works. I have stopped pretending the answer is universal.

Operational Observability — What You Actually Watch in Production

You cannot run this pipeline by feel. Three signals deserve a permanent place on your dashboard.

The first is layer-by-layer latency distribution. If Layer 2 latency suddenly spikes from 70 ms median to 200 ms median, it almost always means input size has grown — either users are sending longer messages, or an upstream system started concatenating threads. Knowing the layer immediately tells you where to look. I emit a per-layer histogram via Prometheus and alert on p95 deviation from a 7-day rolling baseline.

The second is detection rate by entity kind. If person detections drop by 30% over a single day, something has shifted in the input distribution and you may be missing PII that should be masked. False negatives are the dangerous direction here, and they are silent unless you watch the rate. I sample 1% of inputs into a manual review queue and compare extracted entities against human labels weekly.

The third is decryption requests over time, on the audit log side. A sudden spike in unmask calls usually means a support flow changed, or — more rarely — a credential leak. Either case demands attention. Tying decryption rate to user identity in the audit log makes anomalies actionable instead of generic.

For the underlying instrumentation, I lean on the patterns from Antigravity agent trace observability design, which covers OpenTelemetry trace propagation across multi-stage agent pipelines. The PII pipeline is exactly the kind of multi-stage agent where end-to-end traces save you on the bad day.

Failure Modes I Did Not Cover Above

A few smaller pitfalls deserve mention because they cost me hours when I first hit them.

Unicode normalization across layers. The regex layer, the LLM layer, and the masking router each have their own ideas about what "the same string" means. If Layer 1 normalizes to NFC and Layer 2 leaves NFD intact, your literal-substring guard in the NER step will reject correct extractions. Pin the normalization form at pipeline entry and never let it drift.

Concurrent file writes to the audit log. The append-only design assumes a single writer. Under load with multiple worker processes, interleaved writes will corrupt the hash chain. I serialize audit writes through a dedicated single-process logger with a queue in front of it. The latency hit is negligible compared to the integrity guarantee.

Stale LoRA adapters. When you update the fine-tune, old workers may still hold the previous adapter in memory. A rolling restart with strict version pinning in the model loader avoids the embarrassment of mixed-version detection results within a single batch.

Privacy regression in the prompt itself. I once added a few-shot example to the NER prompt that included a real (anonymized) sample. Months later, an audit revealed the example string had appeared in a model output. Few-shot examples in prompts are part of the data perimeter — treat them with the same care as production data.

When Not to Build This Yourself

It would be dishonest to pretend this pipeline is the right call for every team. If your data volume is genuinely small (under a few thousand documents per month), if your customers are not in regulated industries, and if your compliance posture allows cloud LLMs with appropriate data-processing addenda, a managed cloud masking service is faster to ship and easier to operate. The on-prem build pays off when one of three conditions holds: your customers contractually forbid cloud LLMs, your data residency requirements rule out cross-region transfers, or your audit cadence is tight enough that controlling the entire stack is materially cheaper than negotiating evidence packages from a vendor.

I am occasionally asked "Should I just buy a commercial PII detection product?" My honest answer: try the build first, even as a prototype, and measure the actual cost before deciding. Commercial products price aggressively against teams that have never tried it. Once you know your real numbers, the negotiation goes very differently.

Multilingual Edge Cases — What Breaks When Inputs Mix Scripts

Production data rarely arrives in a single language. Your customer base will send you Japanese-English bilingual emails, romanized Asian names with diacritics, and the occasional emoji-laden support ticket that confuses tokenizers in interesting ways. Plan for it from the start.

The most common pattern I see is a Japanese signature block on an otherwise English email. Layer 2 needs to handle code-switching gracefully, and Gemma 4 7B does this surprisingly well — but only if your prompt does not lock the model into a single language frame. The prompt I shared earlier deliberately uses neutral English instructions; an earlier version of mine asked "Extract Japanese named entities" and degraded sharply on bilingual inputs. The wording matters.

A subtler problem is romanized names. "Yamada" can refer to a person, a place, or — much less commonly — an organization. Without surrounding context, the model defaults to person, and that is usually right. But for a small set of customers whose communications mention historical sites or restaurants frequently, the default leaks. I solved this with a per-customer override list that tags certain literal strings as location regardless of the model's call. Domain knowledge always beats model output when they disagree.

Right-to-left scripts (Arabic, Hebrew) introduce another layer of fun: your regex line-anchor assumptions can quietly break, and your placeholder formatting can flip in unexpected ways during display. If your product touches markets with RTL scripts, test the mask placeholders in your downstream rendering layer with synthetic data before any real customer sees them.

Testing Strategy — The Test Suite I Actually Maintain

A masking pipeline without an honest test suite is a bug factory. Here is the layered approach I run.

Property-based tests for Layer 1 — using Hypothesis to generate strings of the form prefix + valid_email + suffix and asserting the email span is detected exactly. The same pattern applies to phones and Luhn-valid card numbers. Property tests catch regressions that hand-written cases miss because they explore the input space programmatically.

Snapshot tests for Layer 2 — a fixed set of ~200 representative inputs with manually labeled expected entities. Whenever I change the prompt or update the model, I run the suite and inspect any diff. A 5% regression rate is the threshold where I block the change and investigate. The suite lives in version control alongside the prompts so changes to either travel together.

Integration tests across the full pipeline — start with raw input, end with masked output, assert that no labeled PII string survives in the output. This is the ultimate behavioral test. It runs in CI on every change and takes about 90 seconds for the 200-input suite on a single GPU.

Adversarial tests — a small but important category. Inputs designed to fool the masker: PII embedded inside JSON fields, PII split across line breaks, PII with intentional typos. Each adversarial test in production traces back to a real incident. The suite grows by accretion, never deletion.

Audit log integrity tests — assert that breaking the chain by removing a middle entry is detected on the next read. This is the test that proves the audit guarantee actually holds. Without it, you are trusting the implementation; with it, you are verifying it.

Rolling Out the Pipeline Without Breaking Production

When you add masking to an existing system, the rollout strategy matters more than the masker itself. I run a four-phase rollout that has yet to surprise me.

Phase 1 is shadow mode. The masking pipeline runs on every input, but the masked output is logged and discarded. The original input continues to flow through the unchanged downstream system. This phase typically runs for two weeks and surfaces the false-positive and false-negative profile against production data without risking any user-visible change.

Phase 2 is opt-in for internal traffic. A small percentage of internal accounts (employee test accounts, QA bots) start receiving masked outputs. Bugs get caught here that shadow mode could not surface — display issues, downstream parsing assumptions, anything where the masked placeholder behaves differently than the original.

Phase 3 is percentage rollout to real customers. Starting at 1% and doubling weekly if metrics hold. The metrics I watch are masking error rate, downstream classification accuracy, and customer support ticket volume. A spike in any of the three pauses the rollout.

Phase 4 is default on. By the time we get here, we have weeks of data showing the masker behaves predictably, and the team has muscle memory for the dashboards. The cutover is anticlimactic, which is exactly how you want it.

Each phase has a written rollback procedure that anyone on the team can execute without paging me. A rollout where rollback requires the original author to be online is a rollout that will eventually break at the worst possible moment.

One Step You Can Take Today

If you take only one action from this article, do this: run Layer 1 (regex prescreen) against 100 representative samples from your own data. Write the false positives and the misses into a small table. The rules you actually need for your domain will surface immediately, and they will surprise you. Skip straight to the LLM and you skip the foundation that makes the whole pipeline trustworthy. The teams I have seen succeed with PII masking in production are the ones that owned the deterministic baseline first.