Skip to content

docs(web,google-ads): screenshot + extract pages, advertiser search up to 3000 + fuzzy (SCR-240) #84

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
0xGhostCasper merged 1 commit into from
Oct 10, 2026
Merged

docs(web,google-ads): screenshot + extract pages, advertiser search up to 3000 + fuzzy (SCR-240) #84

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 87 additions & 5 deletions api-reference/endpoint/google-ads-transparency/search-advertisers.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,15 @@ take. This is the **entry point for brand-name lookups** — free-text search on
[`/ads/search`](/api-reference/endpoint/google-ads-transparency/search-ads) is
domain-based and will not find a brand by name.

**Credits:** 5
**Credits:** 5 — or 15 with `fuzzy=true`. An empty result is not billed.

<Note>
Google's own matching has **no typo tolerance**: advertiser names match by
word prefix (`ruff wear` finds *Ruff Wear, Inc.*, `ruffwear` does not) and
domains by substring (`uffwear` finds `ruffwear.com`). A single typo —
`rufwear`, `ruffwearp` — returns nothing. Set `fuzzy=true` to find misspelled
and look-alike advertisers.
</Note>

## Authorization

Expand All @@ -24,26 +32,56 @@ domain-based and will not find a brand by name.
</ParamField>

<ParamField query="region" type="string" default="US">
ISO 3166-1 alpha-2 region, or `anywhere`. Echoed on the response.
Echoed on the response only. Use `country` to filter advertisers.
</ParamField>

<ParamField query="num" type="integer" default="10">
Suggestions to return, `1`–`20`.
Advertisers to return, `1`–`3000`. 3000 is Google's own per-call ceiling;
one call returns every match up to it, so advertisers do not page.
</ParamField>

<ParamField query="num_domains" type="integer" default="10">
Domain rows to return, `0`–`100`. They follow the advertiser rows.
</ParamField>

<ParamField query="country" type="string">
ISO 3166-1 alpha-2. Only advertisers registered in this country. Domain rows
are not filtered.
</ParamField>

<ParamField query="cursor" type="string">
`next_page_token` from a previous response: the next page of domain rows.
Cannot be combined with `fuzzy`.
</ParamField>

<ParamField query="fuzzy" type="boolean" default="false">
Find misspelled and look-alike advertisers. Searches up to 7 variants of the
query (shorter prefixes, overlapping halves, swapped leading letters),
resolves the closest look-alike domains to the advertisers running ads on
them, drops weak matches, and ranks every row by `similarity`. Billed 15
credits (3 lookups), however many variants run.
</ParamField>

## Response

<ResponseField name="query" type="string">Echo of the requested text.</ResponseField>
<ResponseField name="region" type="string" />
<ResponseField name="country" type="string">The applied `country` filter, upper-cased, or null.</ResponseField>
<ResponseField name="fuzzy" type="boolean" />
<ResponseField name="variants" type="string[]">`fuzzy=true` only: the query variants that were searched.</ResponseField>
<ResponseField name="variants_failed" type="string[]">`fuzzy=true` only: variants whose upstream call failed. The rest are still returned.</ResponseField>
<ResponseField name="next_page_token" type="string">Pass as `cursor` for the next page of domain rows.</ResponseField>
<ResponseField name="advertisers" type="AdvertiserSuggestion[]">
<Expandable title="AdvertiserSuggestion">
<ResponseField name="advertiser_id" type="string">Pass to `/ads/search` and `/ads/advertiser`.</ResponseField>
<ResponseField name="name" type="string" />
<ResponseField name="region" type="string">The advertiser's own registered region, when disclosed.</ResponseField>
<ResponseField name="domain" type="string">Verified domain — a bare-domain suggestion has this but no name.</ResponseField>
<ResponseField name="verified" type="boolean" />
<ResponseField name="verified" type="boolean">Whether Google has verified the advertiser's identity.</ResponseField>
<ResponseField name="ads_count" type="integer" />
<ResponseField name="details_link" type="string">The advertiser's Transparency Center page.</ResponseField>
<ResponseField name="similarity" type="number">`fuzzy=true` only: 0–1 closeness to the query (1 = exact).</ResponseField>
<ResponseField name="matched_query" type="string">`fuzzy=true` only: the variant that surfaced this row, or the look-alike domain an advertiser was found behind.</ResponseField>
</Expandable>
</ResponseField>

Expand Down Expand Up @@ -111,9 +149,53 @@ advertisers = res.json()["advertisers"]
market you care about.
</Tip>

## Finding look-alike advertisers

`fuzzy=true` is built for brand protection. Impersonators rarely advertise
under the brand's name. They register under a personal or unrelated name and
point ads at a look-alike domain, so no name search reaches them. The fuzzy
mode resolves those domains to the advertiser behind them:

```bash
curl "https://scrapebadger.com/v1/google/ads/advertisers?query=rufwear&fuzzy=true" \
-H "X-API-Key: YOUR_API_KEY"
```

```json
{
"query": "rufwear",
"fuzzy": true,
"variants": ["rufwear", "ruf", "rufw", "fwear", "urfwear", "rfuwear", "ruwfear"],
"variants_failed": [],
"advertisers": [
{
"advertiser_id": "AR01181731973429723137",
"name": "Ruff Wear, Inc.",
"domain": "ruffwear.ca",
"similarity": 0.875,
"matched_query": "ruf"
},
{
"advertiser_id": "AR14220847488281083905",
"name": "葛茂朋",
"domain": "ruffwearcf.shop",
"similarity": 0.814,
"matched_query": "ruffwearcf.shop"
},
{ "advertiser_id": null, "domain": "ruffwear.com", "similarity": 0.875, "matched_query": "ruf" },
{ "advertiser_id": null, "domain": "ruffwearcf.shop", "similarity": 0.814, "matched_query": "fwear" }
]
}
```

The same query without `fuzzy` returns `"advertisers": []`. Pass any
`advertiser_id` to [`/ads/search`](/api-reference/endpoint/google-ads-transparency/search-ads)
to see the ads.

## Errors

| Status | Meaning |
| ------ | ------- |
| `400` | `query` shorter than 2 characters, or `num` out of range. |
| `400` | Unknown `country`, or `cursor` combined with `fuzzy`. |
| `422` | `query` shorter than 2 characters, or `num` / `num_domains` out of range. |
| `502` | Upstream Transparency Center failure — **not billed**. |
197 changes: 197 additions & 0 deletions api-reference/endpoint/web-scraping/extract.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
---
title: Extract Data
description: Scrape a URL and pull fields out of it with CSS/XPath selectors, AI, or both.
---

Fetches the page the same way as [`/v1/web/scrape`](/api-reference/endpoint/web-scraping/scrape), then extracts what you asked for:

- `extract_rules` run CSS or XPath selectors over the HTML and return `data`.
- `ai_extract_rules` and `ai_query` ask an LLM about the page and return `ai_extraction`.

Billing works like a scrape, plus the AI extraction credits when AI is requested and succeeds. Broken selectors are rejected with `422` before anything is fetched.

## Request Body

<ParamField body="url" type="string" required>
The page to extract from.
</ParamField>

<ParamField body="extract_rules" type="object">
Field name → selector. A selector starting with `/` or `(` is treated as XPath; anything else is CSS. CSS supports `::text` and `::attr(name)`.

For more control, pass an object instead of a string:
- `selector` (required): the CSS or XPath selector.
- `type`: `css` or `xpath`. Inferred when unset.
- `all`: return every match as a list instead of the first one.
- `output`: `text` (default) or `html` (the element's outer HTML).

```json
{
"title": "h1",
"price": "//span[@class='price']",
"links": { "selector": "a::attr(href)", "all": true }
}
```
</ParamField>

<ParamField body="ai_extract_rules" type="object">
Field name → plain-language description. The AI returns a JSON object with exactly these keys.

```json
{ "price": "the product price with currency", "in_stock": "true if the product can be bought" }
```
</ParamField>

<ParamField body="ai_query" type="string">
A freeform question about the page. When combined with `ai_extract_rules`, the answer is added under an `answer` key.
</ParamField>

<ParamField body="render_js" type="boolean" default={false}>
Render the page in a browser before extracting. Use this for pages that build their content with JavaScript.
</ParamField>

<ParamField body="wait_for" type="string">
CSS selector to wait for before extracting (browser render).
</ParamField>

<ParamField body="country" type="string">
ISO 3166-1 alpha-2 country code for the proxy exit.
</ParamField>

<ParamField body="proxy_tier" type="string" default="simple">
Proxy pool: `simple`, `premium` or `ultra`.
</ParamField>

At least one of `extract_rules`, `ai_extract_rules` or `ai_query` is required. Unknown fields are rejected with `422`.

## Response

<ResponseField name="success" type="boolean">
`true` when the page was fetched and extraction ran.
</ResponseField>

<ResponseField name="url" type="string">
The final URL after redirects.
</ResponseField>

<ResponseField name="status_code" type="integer">
HTTP status code of the fetched page.
</ResponseField>

<ResponseField name="data" type="object">
One key per `extract_rules` field: the first match as a string, a list with `all: true`, or `null` when nothing matched. `null` when no `extract_rules` were given.
</ResponseField>

<ResponseField name="ai_extraction" type="object">
The AI's JSON answer. `null` when no AI was requested.
</ResponseField>

<ResponseField name="ai_model" type="string">
The model that produced `ai_extraction`.
</ResponseField>

<ResponseField name="ai_error" type="string">
Why AI extraction failed, when it did. Selector results in `data` are still returned.
</ResponseField>

<ResponseField name="engine_used" type="string">
Engine that fetched the page.
</ResponseField>

<ResponseField name="credits_used" type="integer">
Credits charged for this request.
</ResponseField>

<ResponseField name="duration_ms" type="integer">
Total time in milliseconds.
</ResponseField>

## Examples

<CodeGroup>

```bash cURL
curl -X POST "https://scrapebadger.com/v1/web/extract" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://news.ycombinator.com",
"extract_rules": {
"top_story": ".titleline a",
"links": {"selector": ".titleline a::attr(href)", "all": true}
}
}'
```

```python Python
import requests

response = requests.post(
"https://scrapebadger.com/v1/web/extract",
headers={"x-api-key": "YOUR_API_KEY"},
json={
"url": "https://example.com/product/123",
"ai_extract_rules": {"price": "the product price", "name": "the product name"},
},
)
print(response.json()["ai_extraction"])
```

```javascript JavaScript
const res = await fetch("https://scrapebadger.com/v1/web/extract", {
method: "POST",
headers: { "x-api-key": "YOUR_API_KEY", "Content-Type": "application/json" },
body: JSON.stringify({
url: "https://example.com",
ai_query: "What is this page for, in one sentence?"
})
});
console.log((await res.json()).ai_extraction);
```

</CodeGroup>

## Error Responses

| Status | Description |
|--------|-------------|
| `402` | Insufficient credits |
| `422` | Invalid request or selector, or the target blocked the request (not billed) |
| `429` | Rate limit exceeded |
| `502` | The page could not be fetched, or AI-only extraction returned nothing (not billed) |

<ResponseExample>

```json 200 — Selectors
{
"success": true,
"url": "https://news.ycombinator.com/",
"status_code": 200,
"data": { "first_title": "Bitwarden Dual License Model" },
"ai_extraction": null,
"ai_model": null,
"ai_error": null,
"engine_used": "http",
"credits_used": 2,
"duration_ms": 1384
}
```

```json 200 — AI query
{
"success": true,
"url": "https://www.example.com/",
"status_code": 200,
"data": null,
"ai_extraction": {
"answer": "This page is a documentation example domain that should not be used for testing or monitoring."
},
"ai_model": "google/gemini-2.5-flash",
"ai_error": null,
"engine_used": "http",
"credits_used": 4,
"duration_ms": 1188
}
```

</ResponseExample>
16 changes: 14 additions & 2 deletions api-reference/endpoint/web-scraping/scrape.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,7 +245,19 @@ openapi: POST /v1/web/scrape
</ParamField>

<ParamField body="screenshot" type="boolean" default={false}>
Capture a full-page screenshot (PNG). Forces the browser engine. Returned as base64 in the `screenshot_url` response field.
Capture a PNG screenshot of the rendered page. Forces the browser engine. Returned in the `screenshot_url` response field as a `data:image/png;base64,…` URI. Captures the viewport unless `screenshot_full_page` is set. For a screenshot-only call, [`POST /v1/web/screenshot`](/api-reference/endpoint/web-scraping/screenshot) returns the bare base64 PNG.
</ParamField>

<ParamField body="screenshot_full_page" type="boolean" default={false}>
With `screenshot: true`, capture the whole scrollable page instead of the viewport.
</ParamField>

<ParamField body="window_width" type="integer">
Browser viewport width in pixels for the render (`320` – `3840`).
</ParamField>

<ParamField body="window_height" type="integer">
Browser viewport height in pixels for the render (`240` – `4320`).
</ParamField>

<ParamField body="video" type="boolean" default={false}>
Expand Down Expand Up @@ -390,7 +402,7 @@ checked against the estimate *including* the proxy surcharge.
</ResponseField>

<ResponseField name="screenshot_url" type="string">
Base64-encoded PNG screenshot of the page. Only present when `screenshot: true` was requested.
The PNG screenshot as a `data:image/png;base64,…` URI. Only present when `screenshot: true` was requested.
</ResponseField>

<ResponseField name="video_url" type="string">
Expand Down
Loading
Loading