I Tested AI Search Citation Repeatability in 40 Engine Pairs

Short answer: I tested AI search citation repeatability with 10 prompts, two same-day runs per prompt, and four public AI Rank API engines per run. Citation counts changed in 15 of 40 engine pairs. If you are tracking AI visibility from one snapshot, you are probably over-reading the data.

The cleaner number is also useful: 30 of the 40 pairs had public model labels in both runs, and 5 of those 30 complete model-labeled pairs changed citation count. The other 10 pairs had one run where no normalized citation/model row was observed for that engine. I kept those rows in the measurement file, but I do not pretend they had a model label.

How I Tested

The run used 10 commercial software prompts and the public AI Rank engines claude, chatgpt, gemini, and perplexity. Each prompt was called twice on August 3, 2026. The domain parameter was held constant so every pair had the same monitored brand context.

I recorded only public response fields: HTTP status, elapsed time, public model label when present, citation count, and whether the second run matched the first. This article does not describe internal execution details, private data sources, or implementation strategy.

The Result

All 20 AI Rank calls returned HTTP 200. Across the 40 engine pairs, 25 pairs had the same citation count and 15 changed citation count. Median elapsed time was 44.3s for first runs and 38.6s for second runs.

The most important reading is not that one engine is "more stable" than another. This is a small same-day pilot, not a ranking of AI systems. The important reading is that repeatability exists as a measurement dimension. A brand that is cited in one run and absent in the next should not receive the same confidence score as a brand cited every time.

Why One Snapshot Is Weak

Traditional rank tracking already has volatility: location, personalization, query wording, and timing can change the result page. AI search adds another layer because the answer page combines a generated response, citations, and source selection. Google's own Search Central documentation frames AI features as part of Search experiences that can include links and source exploration, which means visibility is not just "rank 3" or "rank 9."

That is why single-run AI citation reports can be misleading. They are still useful as screenshots of one moment, but they are weak as proof of durable visibility.

How I Would Score Repeatability

I would separate four metrics instead of compressing everything into one visibility score:

In this pilot, I focused on count repeatability because it is the safest public measurement from the saved response. A later deeper study can add URL-level overlap, host-level overlap, and answer-text deltas if the full response is stored for every run.

Storage Schema

Store every run as an immutable observation. Do not overwrite yesterday's answer with today's answer. Do not collapse missing rows, zero citations, and request failure into the same value.

{
  "keyword": "best crm software for startups",
  "engine": "chatgpt",
  "public_model": "gpt-5.5",
  "run_index": 1,
  "measured_at": "2026-08-03T12:43:00Z",
  "http_status": 200,
  "citation_count": 7,
  "normalized_row_observed": true
}

For rows where the response did not expose a normalized citation/model row for one engine, set normalized_row_observed to false. That is more honest than filling the model label from a neighboring run.

Python Workflow

The script below runs a prompt twice, stores one row per public engine, and leaves room for a no-row state. Use it as a starting point for a weekly citation monitor.

import os
import time
import requests

BASE = "https://apiserpent.com"
API_KEY = os.environ["SERPENT_API_KEY"]
ENGINES = "claude,chatgpt,gemini,perplexity"

def run(keyword, run_index):
    response = requests.get(
        f"{BASE}/api/ai/rank",
        headers={"X-API-Key": API_KEY},
        params={"keyword": keyword, "domain": "example.com", "engines": ENGINES},
        timeout=120,
    )
    response.raise_for_status()
    payload = response.json()
    rows = []
    for engine, result in (payload.get("results") or {}).items():
        citations = result.get("citations") or result.get("citation_urls") or []
        rows.append({
            "keyword": keyword,
            "run_index": run_index,
            "engine": engine,
            "public_model": result.get("model"),
            "citation_count": len(citations),
            "normalized_row_observed": bool(result.get("model") or citations),
        })
    return rows

keyword = "best crm software for startups"
first = run(keyword, 1)
time.sleep(60)
second = run(keyword, 2)
for row in first + second:
    print(row)

What I Would Alert On

I would not alert on a single missing citation. That creates noisy dashboards. I would alert when a prompt loses citations across a minimum run window, when a competitor appears repeatedly where your domain does not, or when a high-value prompt has no normalized citation row across several scheduled checks.

For a small team, a simple rule works: collect three runs per prompt per week, then report stable, intermittent, and absent citation states. Stable is what earns confidence. Intermittent is what gets investigated. Absent is what becomes a content or distribution backlog item.

How to Read the Changed Counts

The changed rows in this pilot should not be read as "bad" rows. They are observations. Some changed from seven citations to ten, some from ten to nine, and some had no normalized citation row observed in one run. Those states have different meanings in a monitoring product.

A count increase can mean the answer became more heavily cited. A count decrease can mean the answer became shorter or cited fewer pages. A no-row state means your storage layer should preserve that the normalized citation list was not observed for that engine run. Treating all three as the same "visibility down" event would produce noisy analysis.

Suggested Run Cadence

For a lightweight monitor, I would start with weekly batches. Run every prompt three times in a short window, store each observation, and compute the stable citation rate at the end of the batch. That gives you enough signal to avoid overreacting to one transient row.

For high-value prompts, use a daily cadence and compare the same weekday week over week. AI visibility reports become more useful when they show durable trend lines: consistently cited, newly cited, intermittently cited, and no longer cited.

The dashboard should show sample size beside every metric. "Cited in 6 of 6 runs" and "cited in 1 of 1 run" are not equivalent, even if both look green in a summary card.

I would also split prompts into intent groups before charting them. Buying prompts, comparison prompts, troubleshooting prompts, and definition prompts often produce different answer shapes. A single repeatability score across all prompts can hide the fact that one group is stable while another group changes frequently.

For stakeholder reporting, keep the summary boring: number of prompt-engine pairs, number with the same citation count, number with a changed citation count, number with a no-row observation, and the exact measurement date. The useful story is not that AI search is random. The useful story is which prompts produce dependable evidence and which prompts need more samples before anyone acts on them.

Limitations

• This was a same-day pilot, not a seven-day or thirty-day volatility study.
• The sample used 10 software prompts and one monitored domain parameter.
• Some successful all-engine responses had no normalized citation/model row observed for one engine; those rows are disclosed instead of relabeled.
• The article reports citation-count repeatability, not URL-level citation overlap.

Recommendation

Use AI citation tracking as a repeated measurement, not a one-off report. Start with a stable prompt taxonomy, collect repeated observations through the AI Rank API, and score durable citation presence separately from one-run appearances.

For implementation detail, pair this with the Python AI citation tracker, AI search visibility metrics, and how AI search selects citations.

References

• Google Search Central: AI features and your website
• Google Search Central: optimizing for generative AI features
• Serpent AI Rank API

FAQ

Is this a 7-day AI citation volatility study?

No. This is a same-day repeatability pilot: 10 prompts, two runs per prompt, and four public AI Rank engines per run on August 3, 2026.

What changed in the pilot?

Across 40 engine pairs, 25 citation counts stayed the same and 15 changed. Among complete model-labeled pairs, 5 of 30 changed.

How should I handle a row with no normalized citation list?

Store it as an observed zero or no-row state with the timestamp. Do not replace it with a guessed model label or invented citations.

Can one AI citation snapshot prove brand visibility?

No. One snapshot is a sample. Use repeated runs and score stable citation presence separately from one-off appearances.

Which pillar should this guide link to?

The right pillar is /ai-rank-api because the workflow is AI visibility monitoring across public AI Rank engines.