How to Scrape Brave Search in 2026 (for Free!)

Brave Search matters more than its market share suggests. It runs its own crawler and its own index, so it returns results you cannot get from Google or Bing. After Microsoft retired the Bing Search API, that independence went from a curiosity to genuinely rare.

Then Brave moved its own Search API behind a paid, card-required plan. The free door that a lot of developers and AI builders relied on quietly closed, and "just scrape it" became the obvious fallback — except Brave sits behind Cloudflare.

This guide walks the whole path: why Brave's index is worth the trouble, what changed with the API, and a working headless scraper that gets past the Cloudflare wall and parses real results.

Why Brave's independent index matters now

Most "alternative" search engines are not really alternatives. Yahoo and DuckDuckGo both serve Microsoft's Bing index, so they echo the same results. Brave is different: it built its own crawler and an index of billions of pages, independent of both Google and Bing.

That independence became scarce. When Microsoft retired the Bing Search API in 2025, a whole ecosystem of tools and AI products lost their cheap, sanctioned window onto a non-Google index. Brave's API was one of the natural replacements — and for grounding AI answers on fresh, diverse results, a second opinion that is not just "Google again" is valuable.

So Brave is worth scraping for one concrete reason: result diversity you genuinely cannot get elsewhere. If you only need broad coverage and are happy with the Bing family, scraping Bing for free or our list of Bing Search API alternatives may be simpler.

What changed with the Brave Search API

For a while, Brave offered a free Search API tier: a modest monthly query allowance, capped request rate, and a card on file. It was a popular escape hatch for indie developers and AI side-projects.

In early 2026 Brave restructured that pricing and moved the old free allowance behind a paid plan. Running an independent crawler is expensive, and demand from AI products grounding their answers on live search spiked the load. The result: card required, metered per query, and no more genuinely free path through the official API.

Pricing and tiers move, so treat any specific number as a snapshot — the authoritative source is always brave.com/search/api. The structural change is what matters here: free went away, and scraping the public results page became the only zero-cost route.

API vs scraping: the honest trade-off

Before you reach for a scraper, be honest about the trade. The API, even paid, gives you clean JSON with no Cloudflare to fight and no parser to maintain. Scraping gives you zero cash cost but real engineering overhead. Here is the comparison.

For contrast, the official call is trivial — this is what you are giving up by scraping:

import requests

resp = requests.get(
    "https://api.search.brave.com/res/v1/web/search",
    headers={
        "X-Subscription-Token": "YOUR_BRAVE_TOKEN",
        "Accept": "application/json",
    },
    params={"q": "best running shoes 2026", "country": "us", "count": 20},
)

for item in resp.json().get("web", {}).get("results", []):
    print(item["title"], item["url"])

If you only need a few thousand queries a month and can put a card down, the paid API is genuinely the calmer option. Scraping is for when you want zero cash cost, or you are already running a multi-engine scraper and Brave is just one more target.

The Cloudflare problem

Here is the wall that stops most people. search.brave.com sits behind Cloudflare, and Cloudflare does not just check your headers — it fingerprints your TLS handshake (the JA3/JA4 signature) before a single byte of HTML is served.

That is why requests, httpx, and curl fail here even with perfect headers: their TLS signatures do not match a real Chrome, so you get a 403, a 1015 rate-limit, or a challenge page instead of results. No User-Agent string fixes a TLS mismatch.

The reliable answer is a real browser. A headless Chromium produces a genuine Chrome TLS fingerprint, runs the challenge JavaScript, and looks like an actual visitor — especially from a residential IP. The same bot-signal hygiene that keeps any scraper alive applies here; we cover it in beating headless Chrome detection.

Build the scraper with a headless browser

This Puppeteer scraper uses the stealth plugin to patch the obvious automation tells, navigates to Brave's results page, and waits for the real listings to appear rather than grabbing a half-loaded or challenge page.

const puppeteer = require('puppeteer-extra');
const Stealth = require('puppeteer-extra-plugin-stealth');
puppeteer.use(Stealth());

async function scrapeBrave(query, { offset = 0 } = {}) {
  const browser = await puppeteer.launch({
    headless: 'new',
    args: [
      '--no-sandbox',
      '--disable-blink-features=AutomationControlled',
      // '--proxy-server=http://gateway.your-proxy.com:7000',  // residential IP
    ],
  });
  const page = await browser.newPage();
  await page.setUserAgent(
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
    '(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
  );

  const url =
    `https://search.brave.com/search?q=${encodeURIComponent(query)}` +
    `&source=web&offset=${offset}`;
  await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 45000 });

  // Cloudflare may serve an interstitial first; wait for the real results
  await page
    .waitForSelector('#results .snippet', { timeout: 20000 })
    .catch(() => console.warn('No results selector — possible challenge page'));

  const results = await parseResults(page);
  await browser.close();
  return results;
}

The key line is waitForSelector('#results .snippet'). If Brave throws a Cloudflare interstitial, the results container never appears and the wait times out, which is your signal to back off and retry later rather than scrape a challenge page as if it were data.

Parsing results and paginating

Brave wraps each organic result in a .snippet element inside #results, with the destination link as the first real anchor and the title in a heading. As always, anchor on the structural container and walk inward, because the deep generated classes change often.

async function parseResults(page) {
  return page.$$eval('#results .snippet', (nodes) =>
    nodes
      .map((n, i) => {
        const a = n.querySelector('a[href^="http"]');
        const title = n.querySelector('.title') || n.querySelector('h3, h4');
        const desc = n.querySelector('.snippet-description, .desc');
        if (!a || !title) return null;
        return {
          position: i + 1,
          title: title.innerText.trim(),
          url: a.href,
          snippet: desc ? desc.innerText.trim() : null,
        };
      })
      .filter(Boolean)
  );
}

Brave paginates with the offset parameter rather than a result count — offset=0 is the first page, offset=1 the next, and so on. Loop it with a generous, randomized delay between pages, because hammering Cloudflare is the fastest way to earn a 1015:

const all = [];
for (let offset = 0; offset < 3; offset++) {
  const batch = await scrapeBrave('best running shoes 2026', { offset });
  if (!batch.length) break;            // challenge or end of results
  all.push(...batch);
  await new Promise((r) => setTimeout(r, 5000 + Math.random() * 5000));
}
console.log(all.length, 'results');

A note on Goggles re-ranking

One Brave feature worth knowing about is Goggles — user-defined rule sets that re-rank Brave's results to boost or bury sources. If you are scraping Brave for a specific lens (say, only independent blogs, or excluding content farms), a Goggle applied via the URL changes the ordering you receive.

For most scraping you can ignore them and take the default ranking. But if you need a consistent, opinionated view of the index across runs, Goggles are a legitimate, supported way to shape it without touching your parser.

Proxies, bandwidth, and staying unblocked

From one IP, you can pull a modest number of Brave queries before Cloudflare's rate counter trips. Past that, you rotate residential proxies so requests come from many real-looking addresses. Datacenter IPs are a poor fit here — Cloudflare distrusts known cloud ranges on sight.

Residential proxies are billed per gigabyte, so block images, fonts, and stylesheets at the request layer to keep that bill small. The full technique is in cutting scraping bandwidth by blocking resources. Any provider works the same way — Bright Data, Oxylabs, Decodo, IPRoyal, DataImpulse, and SOAX all expose a gateway and credentials you wire in with --proxy-server and page.authenticate().

The two free defenses that matter most are pacing and patience. A clean residential IP does not save a robotic request rhythm, which is the lesson of why proxies get banned. Slow down, randomize, and Brave stays open far longer.

The managed alternative

If the goal is reliable search data rather than the sport of beating Cloudflare, a managed SERP API removes the entire access problem for the major engines. Serpent returns clean JSON for Google, Bing, Yahoo, and DuckDuckGo — the proxies, rendering, and anti-bot are handled for you:

import requests

resp = requests.get(
    "https://api.apiserpent.com/api/search",
    headers={"X-API-Key": "sk_live_your_key"},
    params={"q": "best running shoes 2026", "engine": "bing", "country": "us"},
)

for r in resp.json()["results"]["organic"]:
    print(r["position"], r["title"], r["url"])

Brave's index is its own thing, so keep the scraper above for that diversity. But for broad, stable coverage without a proxy pool or a Cloudflare arms race, the API is the calmer path — compare your options in the best SERP API in 2026, or try a live query in the playground.

FAQ

Is Brave Search really an independent index?

Yes. Unlike Yahoo and DuckDuckGo, which serve Bing's results, Brave runs its own web crawler and index of billions of pages. After Microsoft retired the Bing Search API, Brave became one of the few major Western search indexes you can query that is not Google and not Bing, which is why it is worth scraping for result diversity.

Why did Brave's free search API tier disappear?

Brave restructured its Search API pricing and moved the old free allowance behind a paid, card-required plan. Running an independent crawler and index is expensive, and surging demand from AI products grounding answers on live search pushed Brave to monetize that access. Always check brave.com/search/api for the current tiers.

Can I get past Cloudflare when scraping Brave?

Brave's site sits behind Cloudflare, so plain HTTP clients get 403 or 1015 errors at the TLS layer. A real headless browser with a stealth setup and a residential IP clears the standard challenge for most queries; aggressive interactive challenges need real-browser tooling. Go slow and keep request volume modest per IP.

Is scraping Brave Search legal?

Scraping publicly visible search results is broadly treated as legal in the US after hiQ v. LinkedIn, but it can still breach Brave's terms of service, and the rules differ by country. Scrape at low volume, do not hammer the servers, and read the site's terms. This is general information, not legal advice.