How to Extract Google AI Overviews at Scale (2026 API Guide + JSON Schema)

Google AI Overviews may appear for a query, but their presence and sources vary. For SEO or AEO reporting, retain the returned overview fields and compare dated source lists; check Google's interface before calling a URL a displayed citation. If you would rather inspect the raw results page yourself, the guide to parsing Google SERP features covers the overview block alongside snippets and People Also Ask.

This guide walks through the exact JSON schema returned by modern SERP APIs, a Python recipe that runs in under thirty lines, the edge cases that broke our first three attempts, and a weekly diff job that emails you when the source list for a tracked keyword changes.

Why Extract AI Overviews?

Three reasons teams pull AIO data programmatically:

• Brand citation tracking. Is your domain inside the answer Google synthesises for your money keywords? If not, who is?
• AEO measurement. Citation rate inside AIO is the new "ranking position." You cannot improve what you do not measure — it sits at the core of the broader AI search visibility metrics teams now report on.
• Competitive intel. A domain you have never heard of suddenly cited in 14 of your top 50 keywords is signal.

You cannot answer any of these by clicking through SERPs by hand. You need an API call, parsed JSON, and a database.

How Google AIO Works (and What You Can Pull)

An AI Overview is a synthesised answer rendered above the organic results for queries Google decides will benefit from a summarised response. Three things to know:

1. An AIO can have a body and source links. The visible feature varies by query and market; check the Google interface when the exact displayed links matter.
2. The source list is what matters for AEO. The body changes wording often; the source list changes domains less often. Tracking the domains gives you a stable signal.
3. Not every query gets one. Overview presence can change with query wording, location, device and time. Check each returned response instead of assuming an informational or transactional query will have a particular feature.

The AIO JSON Schema

Here is an illustrative Serpent API response shape for an eligible paid Google Deep request with include_aio=true and full format. The example URLs are not evidence of what Google displayed for this query:

{
  "results": {
    "aiOverview": {
      "text": "Mesothelioma is a rare cancer affecting the mesothelium, the protective lining around organs. It is most commonly caused by exposure to asbestos and typically takes 20 to 50 years to develop after exposure...",
      "sources": [
        {
          "title": "What is Mesothelioma? - Mayo Clinic",
          "url": "https://www.mayoclinic.org/diseases-conditions/mesothelioma/symptoms-causes/syc-20375022"
        },
        {
          "title": "Mesothelioma - National Cancer Institute",
          "url": "https://www.cancer.gov/types/mesothelioma"
        }
      ]
    },
    "organic": [...]
  }
}

Key fields:

• results.aiOverview.text — the prose summary. Plain text, sometimes with inline citation markers like [1].
• results.aiOverview.sources[] — source entries with a title and url. Review them before calling them verified Google citations.

If results.aiOverview is null, record that no overview was available in this API response. It does not prove what appeared in Google's interface. Keep the organic rows and continue.

The 30-Line Python Recipe

Pip install requests if you need it. Replace API_KEY with your own.

import requests
import json
from urllib.parse import urlparse

API_KEY = "sk_live_your_key_here"
BASE_URL = "https://apiserpent.com/api/search"

def fetch_aio(keyword, country="us"):
    params = {
        "q": keyword,
        "country": country,
        "engine": "google",
        "include_aio": "true",   # required — AI Overview fields are opt-in
    }
    r = requests.get(BASE_URL, params=params, headers={"X-API-Key": API_KEY}, timeout=60)
    r.raise_for_status()
    data = r.json()
    aio = data.get("results", {}).get("aiOverview")
    if not aio:
        return {"keyword": keyword, "has_aio": False}
    source_urls = [s["url"] for s in aio.get("sources", [])]
    return {
        "keyword": keyword,
        "has_aio": True,
        "text": aio["text"],
        "sources": [urlparse(u).hostname or "" for u in source_urls],
        "source_urls": source_urls,
    }

if __name__ == "__main__":
    for kw in ["mesothelioma", "best ergonomic chair", "react vs vue"]:
        result = fetch_aio(kw)
        print(json.dumps(result, indent=2))

Run it. You will see three blocks of JSON, one per keyword, with the cited domains pulled out into a clean list. That is the entire pipeline. Everything else — storage, scheduling, alerting — is built on top of this 30-line core.

Scaling to 10,000 Queries a Day

One query per second is too slow for a real keyword list. Here is how to scale:

1. Concurrency, not parallelism

Use asyncio + httpx.AsyncClient with a semaphore below your live account's concurrency limit. Serpent's published Scale bracket at a $500–$999 spendable balance allows up to 100 concurrent calls and 750 per minute; the $1,000–$1,249 bracket allows up to 300 concurrent and 2,250 per minute; and the $1,250+ bracket allows up to 500 concurrent and 5,000 per minute. Those are up-to allocations, not guaranteed throughput. A semaphore of 10 is below either concurrency cap, but you still need to respect the per-minute, hourly and daily limits and measure actual response times.

import asyncio
import httpx

SEMAPHORE = asyncio.Semaphore(10)

async def fetch_one(client, keyword):
    async with SEMAPHORE:
        r = await client.get(BASE_URL, params={
            "q": keyword, "engine": "google", "include_aio": "true",
            "country": "us",
        }, headers={"X-API-Key": API_KEY}, timeout=60)
        return r.json()

async def fetch_many(keywords):
    async with httpx.AsyncClient() as client:
        tasks = [fetch_one(client, kw) for kw in keywords]
        return await asyncio.gather(*tasks, return_exceptions=True)

results = asyncio.run(fetch_many(my_keyword_list))

2. Batch by country

Run all US queries together, then GB, then DE. AIO results vary by country — do not mix them in the same response payload or you will get inconsistent data.

3. Deduplicate before you call

Hash (keyword, country) and skip ones already fetched today. AIO results are fairly stable within a 24-hour window for the same query.

4. Store the raw AIO text, not just the parsed fields

Future-you will want the original prose to re-parse with new logic. Keep both: a structured row in Postgres for analytics, and the raw JSON in S3 (or Cloud Storage) for replay.

Edge Cases You Will Hit

Six things that broke our first three pipelines:

1. Multi-section AIO. Sometimes Google renders the AIO as multiple expandable blocks. Modern SERP APIs concatenate them into a single text field, but you may want to keep section boundaries. Check for \n\n in results.aiOverview.text.
2. "Show more" differences. The API text and visible Google interface can differ. Save both observations when display fidelity matters.
3. Source list partial loads. On rare occasions Google ships an AIO without a source list. Treat sources == [] as a valid state, not an error.
4. Non-English AIOs. If you tracking country=de, the AIO text is in German. Run language detection before downstream NLP.
5. Different viewing contexts. Results can vary by account, region and time. Compare like-for-like observations before interpreting a change.
6. The AIO disappeared. Some queries had AIOs last week but not this week. Always check has_aio before processing — never assume the structure exists.

Tracking AIO Source Drift Week-Over-Week

Measure how the API source list changes, then verify meaningful differences against the Google interface. Here is the diff job:

import sqlite3
from datetime import date

def store_snapshot(keyword, sources):
    conn = sqlite3.connect("aio_tracker.db")
    conn.execute("""
        CREATE TABLE IF NOT EXISTS aio_snapshots (
            keyword TEXT, snap_date TEXT,
            sources TEXT, PRIMARY KEY (keyword, snap_date)
        )
    """)
    conn.execute("INSERT OR REPLACE INTO aio_snapshots VALUES (?, ?, ?)",
                 (keyword, date.today().isoformat(), ",".join(sources)))
    conn.commit()
    conn.close()

def diff_last_two(keyword):
    conn = sqlite3.connect("aio_tracker.db")
    rows = conn.execute("""
        SELECT snap_date, sources FROM aio_snapshots
        WHERE keyword = ? ORDER BY snap_date DESC LIMIT 2
    """, (keyword,)).fetchall()
    conn.close()
    if len(rows) < 2:
        return None
    new_set = set(rows[0][1].split(","))
    old_set = set(rows[1][1].split(","))
    return {
        "added": list(new_set - old_set),
        "removed": list(old_set - new_set),
        "stable": list(new_set & old_set),
    }

Run this on a schedule that fits your reporting needs. Any keyword where added or removed is non-empty merits review against the dated response and Google's interface; the API diff alone does not establish a displayed citation gain or loss.

Cost Math

How much does it cost to run an AIO tracker? This is a one-page cost illustration, not a promise that every provider returns equivalent Overview fields. Confirm each provider's field coverage, freshness and billing before treating the figures as interchangeable.

• 500 keywords × 1 query per week × 4 weeks = 2,000 queries/month
• At Serpent API Scale tier ($0.10 per 1,000 one-page Google Deep calls): $0.20/month in usage on eligible new credit after one qualifying $500 deposit
• At Serper.dev's Ultimate pack rate ($0.30 per 1,000 credits): $0.60 of nominal credit use, but that rate requires a $3,750 prepaid pack valid for six months; verify AI Overview field coverage separately
• At DataForSEO Standard ($0.60 per 1,000 10-result SERPs plus one base price per call for load_async_overview): $2.40 in modeled monthly usage for 2,000 one-SERP calls requesting that option
• At SerpApi Developer ($75/month for up to 5,000 searches): $75/month for this 2,000-search workload, not a prorated $30

At 10,000 keywords checked four times a month, 40,000 one-page Google Deep calls use $4 of eligible new Scale credit. The $500 qualifying deposit remains spendable but must be funded up front. Forty thousand calls exceed SerpApi's published Big Data allowance of 30,000 monthly searches, so request a suitable volume quote or model its documented early-renewal behavior instead of multiplying a smaller plan's effective rate.

FAQ

How do I extract Google AI Overviews via API?

On an eligible paid Google Deep call with include_aio=true and full format, Serpent may return results.aiOverview.text and results.aiOverview.sources. Store those API fields for review; check Google's interface separately before reporting a URL as a displayed citation.

Which queries trigger an AI Overview?

AI Overview presence varies by query, location, device and time. Do not assume a particular share of brand or informational searches has one. Check whether an overview was returned for each eligible request and confirm Google's interface before reporting a displayed citation.

How often do AI Overview source lists change?

There is no fixed refresh interval or universal change rate. Choose a schedule that fits your reporting needs, retain dated snapshots, and compare source URLs as well as answer text. Review changes against Google's interface before reporting them as displayed citation gains or losses.

Is scraping the AI Overview legal?

Reading public SERP data through a SERP API is consistent with public-data court precedent in major markets. The AIO block, like organic results, appears on a public results page. Use a SERP API provider rather than scraping directly if you want a clean compliance boundary.

Can I extract AI Mode answers the same way?

AI Mode is a separate Google surface from AI Overviews. It currently does not appear in the standard SERP HTML, so most SERP APIs do not yet return AI Mode responses. AIO extraction is the right starting point.

How do I detect when my domain stops being cited?

Run the diff job above. A removed URL means it left the API source list; review the answer and Google's interface before calling it a lost Google citation.