Plug Any SERP API into gpt-researcher (RETRIEVER=custom, Documented Properly)

gpt-researcher is the most-starred open-source deep-research agent — 28,522 stars as of July 21, 2026 — and it ships a RETRIEVER=custom mode that lets you plug in any search backend you want. The official docs give it about a paragraph. What's the exact request it makes? What JSON must you return? What happens to raw_content? Nobody says.

So we read the source, pinned every claim to repo tag v3.6.0 (commit 5d84d2f, 2026-07-14), built the integration both ways the code supports, and ran a recency-sensitive research query through four retriever configurations with the LLM and embeddings held constant. This is the reference we wished existed.

What a new user actually gets (measured)

The default config sets "RETRIEVER": "tavily" (config/variables/default.py). Run it without a Tavily key and nothing crashes — that's the problem. The retriever prints "Tavily API key not found, set to blank", initializes with an empty key, and every search gets a 401 that's caught and converted into an empty result list.

We ran the full pipeline that way. It "completed" in 11.1 seconds, retrieved zero sources, and wrote this 46-word report:

I could not gather any source material for "What were the most significant AI
model releases and announcements in June and July 2026?". No sources were
retrieved (searches may have returned nothing or been blocked), so I am not
able to produce a reliable, sourced report.

Two honest take-aways. One: recent builds at least tell you the report is empty instead of hallucinating one — credit where due. Two: the silent per-query failure means a half-broken retriever degrades quality without ever raising an error. Set RETRIEVER explicitly, always. The built-in keyless option is duckduckgo (via the ddgs package) — that's our baseline below. The option this post is about is custom.

RETRIEVER=custom, read from source

Retriever selection lives in gpt_researcher/actions/retriever.py: a match on the name returns the class, "custom" maps to CustomRetriever, comma-separated values run multiple retrievers per sub-query, and — worth knowing — an invalid name silently falls back to Tavily rather than erroring.

The whole custom retriever is ~70 lines in gpt_researcher/retrievers/custom/custom.py. The parts that define the contract:

# gpt_researcher/retrievers/custom/custom.py  (v3.6.0, commit 5d84d2f)
class CustomRetriever:
    def __init__(self, query: str, query_domains=None):
        self.endpoint = os.getenv('RETRIEVER_ENDPOINT')
        if not self.endpoint:
            raise ValueError("RETRIEVER_ENDPOINT environment variable not set")
        self.params = self._populate_params()   # every RETRIEVER_ARG_* env var
        self.query = query

    def search(self, max_results: int = 5):
        response = requests.get(
            self.endpoint,
            params={**self.params, "query": self.query},
            timeout=20,          # hardcoded — your endpoint gets 20 seconds
        )

Spelled out, for every sub-query the planner generates:

So with RETRIEVER_ARG_COUNTRY=us, the actual request per sub-query is:

GET {RETRIEVER_ENDPOINT}?country=us&query=top+AI+model+announcements+June+July+2026

Three more contract details straight from the code: max_results is accepted but ignored (your endpoint decides how many results return); any request exception or JSON parse failure is caught, printed, and turned into an empty list (the run continues without that sub-query); and a non-list JSON body is discarded with a printed warning. There is no auth header, no POST body, no pagination — the shim owns all of that.

The exact JSON shape — and the 100-character rule

The docstring in custom.py gives the expected response, and it's the one thing the docs do state:

[
  { "url": "http://example.com/page1", "raw_content": "Content of page 1" },
  { "url": "http://example.com/page2", "raw_content": "Content of page 2" }
]

What the docs don't state is what happens next, and it's the most consequential branch in the whole integration. In gpt_researcher/skills/researcher.py, _search_relevant_source_urls() splits your results in two:

# gpt_researcher/skills/researcher.py — _search_relevant_source_urls()
url = result.get("href") or result.get("url")
raw_content = result.get("raw_content")
if url and raw_content and len(raw_content) > 100:
    # raw_content signals the retriever already fetched the full page
    prefetched_content.append({"url": url, "raw_content": raw_content})
elif url:
    new_search_urls.append(url)   # gpt-researcher scrapes this itself

That len(raw_content) > 100 is the fork. It gives you two legitimate integration modes:

URL mode (raw_content: null) — you return links only; gpt-researcher's own scraper (default SCRAPER=bs, BeautifulSoup) fetches every page in full. Reports are grounded in complete page text. This is the mode that behaves like the built-in web retrievers.

Snippet mode (raw_content = title + snippet) — a SERP snippet is almost always longer than 100 characters, so gpt-researcher treats it as a fetched page and never scrapes the URL. Faster, no scraper failures, dramatically fewer input tokens — and the report is grounded in nothing but result-page text. This can happen to you by accident: naively map your SERP API's snippet field onto raw_content and you've silently turned off scraping for the entire run. We measured both modes below.

A 30-line shim for any SERP API

gpt-researcher won't call a SERP API directly — the param name (query vs q), auth, and response shape all differ. The adapter is a webhook-sized Express app. This is the complete file we ran, both modes included:

// serp-shim.js — any SERP API -> gpt-researcher CustomRetriever contract
const express = require('express');
const app = express();

const API_KEY = process.env.SERPENT_API_KEY; // never hardcode keys
const PORT = process.env.PORT || 5055;
// 'urls'     -> raw_content: null  => gpt-researcher scrapes pages (recommended)
// 'snippets' -> raw_content: title+snippet => no scraping (fast, thin context)
const MODE = process.env.SHIM_MODE || 'urls';

app.get('/search', async (req, res) => {
  const { query, ...extra } = req.query; // RETRIEVER_ARG_FOO=bar arrives as foo=bar
  try {
    const params = new URLSearchParams({ q: query, api_key: API_KEY, ...extra });
    const r = await fetch(`https://apiserpent.com/api/search/quick?${params}`);
    if (!r.ok) throw new Error(`SERP API ${r.status}`);
    const data = await r.json();
    const organic = (data.results && data.results.organic) || [];
    res.json(organic.map((o) =>
      MODE === 'snippets'
        ? { url: o.url, raw_content: `${o.title}. ${o.snippet || ''}` }
        : { url: o.url, raw_content: null }
    ));
  } catch (err) {
    console.error(`[shim] ${query}: ${err.message}`);
    res.json([]); // empty list = "no results"; never 500 the researcher
  }
});

app.listen(PORT, () => console.log(`SERP shim (${MODE} mode) on :${PORT}`));

Run it with SERPENT_API_KEY=YOUR_API_KEY node serp-shim.js (10 free calls, live rates here; the response shape is the same one you can inspect in the playground). The same 15 lines of mapping adapt any provider — only the URL, auth and the path to the organic array change. If you'd rather expose search to an IDE or chat client instead of a research agent, the same contract-adapting idea powers our MCP server tutorial.

gpt-researcher's side of the wiring is three lines of env:

RETRIEVER=custom
RETRIEVER_ENDPOINT=http://127.0.0.1:5055/search
RETRIEVER_ARG_COUNTRY=us   # forwarded to the shim as country=us

The LLM and embeddings we held constant

A retriever comparison is meaningless if the model changes between runs, so both comparison runs (and both extras) used the identical stack:

# LLM — any OpenAI-compatible endpoint works; native providers too
OPENAI_API_KEY=YOUR_LLM_KEY
OPENAI_BASE_URL=https://api.your-llm-provider.com   # OpenAI-compatible endpoints
FAST_LLM=openai:your-model
SMART_LLM=openai:your-model
STRATEGIC_LLM=openai:your-model

# Embeddings — local, open-source, zero rate limits
EMBEDDING=huggingface:sentence-transformers/all-MiniLM-L6-v2

Two hard-won notes. First, the openai: provider honors OPENAI_BASE_URL (llm_provider/generic/base.py), so any OpenAI-compatible chat endpoint drops in — we ran the final comparison through one, and the whole four-run session cost under a cent at typical flash-tier rates (39,000 measured LLM tokens total).

Second, embeddings are the hidden rate-limit trap. gpt-researcher embeds every scraped chunk for its relevance filter, which means dozens of embedding calls in a burst. Our first attempt used a hosted free-tier embedding API; it hit the per-minute cap mid-run and the pipeline "succeeded" into a near-empty report — the same silent-degradation pattern as the missing Tavily key. Local sentence-transformers embeddings (the huggingface: provider in memory/embeddings.py) removed the failure mode entirely and keep the comparison fair. The full decision matrix for pairing models with search backends is in our web-search-API comparison for agents.

The runner is the standard programmatic entry point:

import asyncio
from gpt_researcher import GPTResearcher

async def main():
    researcher = GPTResearcher(
        query="What were the most significant AI model releases and "
              "announcements in June and July 2026?",
        report_type="research_report",
    )
    await researcher.conduct_research()
    print(await researcher.write_report())

asyncio.run(main())

Same query, four runs: the numbers

The query is deliberately recency-sensitive — June and July 2026 events, asked on July 21, 2026 — because recency is where retrievers differ most. Same query, same research_report type, same LLM, same embeddings, four retriever configs. Measured on 2026-07-21:

What the numbers say — including the parts that don't flatter us:

The keyless baseline is genuinely usable. DuckDuckGo found real, current sources and produced the longest report. In this single run we saw none of the rate-limiting the ddgs library is known for at higher volumes. If it has a weakness here, it's citation drift: half its citations weren't June–July-2026-dated — among them an April 2026 roundup, a crypto-market piece, and a hackathon landing page.

The SERP retriever's edge is recency density and volume control. 12 of 14 citations dated to the exact window, and the sources skewed to dated release-log pages (monthly changelogs, release trackers) rather than evergreen roundups. It also retrieved 29 sources vs 19 — not because it's "better", but because the shim returns ~10 organic results per sub-query while the DuckDuckGo retriever is capped at MAX_SEARCH_RESULTS_PER_QUERY (default 5). Remember: your endpoint controls that dial, and every extra URL is an extra page scrape.

Snippet mode is a real trade, not a hack. Zero pages scraped, 49% fewer input tokens than URL mode, and still 15 citations with 12 dated — but every claim in that report is grounded in result-page text alone, and one of its cited domains was the kind of low-trust site that full-page scraping tends to filter out. Fine for cheap breadth-first drafts; wrong for anything you'll act on. The same trimming logic applies to agent tools generally — we measured it for the OpenAI Agents SDK and the Vercel AI SDK.

Wall-clock favors the baseline slightly. The custom runs added ~8 seconds — the shim's SERP round-trips (2–9 s each) plus 10 extra page scrapes. If latency matters more than coverage, return fewer results from the shim.

Honest scope limits: this is one query, one run per config, on one day — retriever output and LLM writing both vary run to run. Token counts are measured from provider usage metadata; the "dated" citation count is by URL/content inspection of each cited link. Both full reports and all metrics are archived and excerpted faithfully here.

Four gotchas the docs don't mention

1. The 20-second timeout eats whole sub-queries. timeout=20 is hardcoded in CustomRetriever.search(). In one pilot run our first shim call arrived cold, took just over 20 s, and that sub-query contributed zero sources — the run completed with a visibly thinner report and no error surfaced. Keep the SERP call on a quick endpoint (we used /api/search/quick, 2–9 s; a deep 100-result call would flirt with the limit) and warm the path before long sessions.

2. Import crash on Python ≤ 3.13 at v3.6.0. gpt_researcher/actions/query_processing.py uses Any and List annotations without importing them. Python 3.14 defers annotation evaluation (PEP 649) so it runs fine there; on 3.13 and older, import gpt_researcher raises NameError: name 'Any' is not defined. Fix is one line at the top of that file: from typing import Any, List. We hit this on Python 3.13.12 and patched exactly that.

3. Typos silently become Tavily. get_retrievers() substitutes the default retriever for any unrecognized name. Misspell RETRIEVER=custom and — with a Tavily key in the env — your runs quietly use a different search backend than you think. Without one, you get the empty-report failure from the top of this post. Grep your logs for your shim's port on the first run.

4. Your endpoint is the cost governor. Because max_results is ignored, result count, geo-targeting (RETRIEVER_ARG_COUNTRY), and freshness are all decided shim-side. That's a feature: one SERP call per sub-query costs a fraction of a cent (the grounding-cost math), and four sub-queries per report keeps search under a cent while the LLM dominates spend. For an agent that owns the whole research loop end to end — planner, retriever, writer — the from-scratch version of this architecture is in our deep-research agent build, and the broader retriever-selection question in SERP APIs for AI agents and the Tavily/Exa comparison. RAG pipelines that want the same live-data layer without the agent loop should start from the real-time RAG tutorial; if the goal is a citing answer engine rather than long-form reports, that build is in the Perplexity-alternative walkthrough.

FAQ

What JSON shape does gpt-researcher's custom retriever expect?

A JSON list of {"url": ..., "raw_content": ...} objects. At v3.6.0 a dict or any other top-level shape is discarded with a printed warning and that sub-query returns nothing. raw_content: null (or under 100 characters) means gpt-researcher scrapes the URL itself; longer means the text is used as-is and the page is never fetched.

What happens if I run gpt-researcher without a Tavily API key?

The default retriever initializes with a blank key, every Tavily call 401s, each sub-query silently returns empty, and the pipeline completes anyway. Our measured run: 11.1 seconds, zero sources, a 46-word non-report. Set RETRIEVER explicitly — duckduckgo for keyless, custom for your own endpoint.

Can I combine the custom retriever with other retrievers?

Yes — RETRIEVER accepts a comma-separated list, so RETRIEVER=custom,arxiv runs each sub-query through both and merges the URLs. Mind the sibling behavior: an invalid name doesn't error, it silently falls back to Tavily.

What timeout does the custom retriever use?

20 seconds, hardcoded — no env var changes it at v3.6.0. A slower response kills that sub-query's sources without failing the run. Use a fast SERP endpoint and keep the shim's own error handling returning [] rather than a 500.

Does max_results work with RETRIEVER=custom?

No — the researcher passes it (default MAX_SEARCH_RESULTS_PER_QUERY=5) but CustomRetriever.search() never uses it. Your endpoint decides the count. That's why our custom runs retrieved 29 sources against the baseline's 19 — and why you should trim shim output if scrape volume or latency matters.

Should I return SERP snippets as raw_content?

Only deliberately. Anything over 100 characters disables scraping for that URL, so the whole report grounds on snippet text. Measured: 49% fewer input tokens than URL mode, 15 citations — and zero pages actually read. Good for cheap drafts, bad for decisions.