From 5770739ea64bc72205a74b4a3c960ef8102d1143 Mon Sep 17 00:00:00 2001 From: kasparasizi1 <132673909+kasparasizi1@users.noreply.github.com> Date: Sat, 10 Oct 2026 18:56:45 +0300 Subject: [PATCH] docs(web,google-ads): /v1/web/screenshot + /extract pages, real screenshot fields, advertiser search up to 3000 + fuzzy (SCR-240) - New Web Scraping pages for POST /v1/web/screenshot and /v1/web/extract (they 404'd until scrapebadger#2369; now served from /scrape). - /scrape: screenshot is the viewport unless screenshot_full_page; new window_width/window_height; screenshot_url is a data: URI. - Google Ads Transparency search-advertisers + overview synced from the monorepo (scrapebadger#2370): num up to 3000, num_domains, country, cursor, fuzzy look-alike recall. Co-Authored-By: Claude Opus 5.5 --- .../search-advertisers.mdx | 92 +++++++- .../endpoint/web-scraping/extract.mdx | 197 ++++++++++++++++++ .../endpoint/web-scraping/scrape.mdx | 16 +- .../endpoint/web-scraping/screenshot.mdx | 138 ++++++++++++ docs.json | 4 +- google-ads-transparency/overview.mdx | 6 +- 6 files changed, 443 insertions(+), 10 deletions(-) create mode 100644 api-reference/endpoint/web-scraping/extract.mdx create mode 100644 api-reference/endpoint/web-scraping/screenshot.mdx diff --git a/api-reference/endpoint/google-ads-transparency/search-advertisers.mdx b/api-reference/endpoint/google-ads-transparency/search-advertisers.mdx index 6fd33ee..ec32f3c 100644 --- a/api-reference/endpoint/google-ads-transparency/search-advertisers.mdx +++ b/api-reference/endpoint/google-ads-transparency/search-advertisers.mdx @@ -9,7 +9,15 @@ take. This is the **entry point for brand-name lookups** — free-text search on [`/ads/search`](/api-reference/endpoint/google-ads-transparency/search-ads) is domain-based and will not find a brand by name. -**Credits:** 5 +**Credits:** 5 — or 15 with `fuzzy=true`. An empty result is not billed. + + + Google's own matching has **no typo tolerance**: advertiser names match by + word prefix (`ruff wear` finds *Ruff Wear, Inc.*, `ruffwear` does not) and + domains by substring (`uffwear` finds `ruffwear.com`). A single typo — + `rufwear`, `ruffwearp` — returns nothing. Set `fuzzy=true` to find misspelled + and look-alike advertisers. + ## Authorization @@ -24,26 +32,56 @@ domain-based and will not find a brand by name. - ISO 3166-1 alpha-2 region, or `anywhere`. Echoed on the response. + Echoed on the response only. Use `country` to filter advertisers. - Suggestions to return, `1`–`20`. + Advertisers to return, `1`–`3000`. 3000 is Google's own per-call ceiling; + one call returns every match up to it, so advertisers do not page. + + + + Domain rows to return, `0`–`100`. They follow the advertiser rows. + + + + ISO 3166-1 alpha-2. Only advertisers registered in this country. Domain rows + are not filtered. + + + + `next_page_token` from a previous response: the next page of domain rows. + Cannot be combined with `fuzzy`. + + + + Find misspelled and look-alike advertisers. Searches up to 7 variants of the + query (shorter prefixes, overlapping halves, swapped leading letters), + resolves the closest look-alike domains to the advertisers running ads on + them, drops weak matches, and ranks every row by `similarity`. Billed 15 + credits (3 lookups), however many variants run. ## Response Echo of the requested text. +The applied `country` filter, upper-cased, or null. + +`fuzzy=true` only: the query variants that were searched. +`fuzzy=true` only: variants whose upstream call failed. The rest are still returned. +Pass as `cursor` for the next page of domain rows. Pass to `/ads/search` and `/ads/advertiser`. The advertiser's own registered region, when disclosed. Verified domain — a bare-domain suggestion has this but no name. - + Whether Google has verified the advertiser's identity. The advertiser's Transparency Center page. + `fuzzy=true` only: 0–1 closeness to the query (1 = exact). + `fuzzy=true` only: the variant that surfaced this row, or the look-alike domain an advertiser was found behind. @@ -111,9 +149,53 @@ advertisers = res.json()["advertisers"] market you care about. +## Finding look-alike advertisers + +`fuzzy=true` is built for brand protection. Impersonators rarely advertise +under the brand's name. They register under a personal or unrelated name and +point ads at a look-alike domain, so no name search reaches them. The fuzzy +mode resolves those domains to the advertiser behind them: + +```bash +curl "https://scrapebadger.com/v1/google/ads/advertisers?query=rufwear&fuzzy=true" \ + -H "X-API-Key: YOUR_API_KEY" +``` + +```json +{ + "query": "rufwear", + "fuzzy": true, + "variants": ["rufwear", "ruf", "rufw", "fwear", "urfwear", "rfuwear", "ruwfear"], + "variants_failed": [], + "advertisers": [ + { + "advertiser_id": "AR01181731973429723137", + "name": "Ruff Wear, Inc.", + "domain": "ruffwear.ca", + "similarity": 0.875, + "matched_query": "ruf" + }, + { + "advertiser_id": "AR14220847488281083905", + "name": "葛茂朋", + "domain": "ruffwearcf.shop", + "similarity": 0.814, + "matched_query": "ruffwearcf.shop" + }, + { "advertiser_id": null, "domain": "ruffwear.com", "similarity": 0.875, "matched_query": "ruf" }, + { "advertiser_id": null, "domain": "ruffwearcf.shop", "similarity": 0.814, "matched_query": "fwear" } + ] +} +``` + +The same query without `fuzzy` returns `"advertisers": []`. Pass any +`advertiser_id` to [`/ads/search`](/api-reference/endpoint/google-ads-transparency/search-ads) +to see the ads. + ## Errors | Status | Meaning | | ------ | ------- | -| `400` | `query` shorter than 2 characters, or `num` out of range. | +| `400` | Unknown `country`, or `cursor` combined with `fuzzy`. | +| `422` | `query` shorter than 2 characters, or `num` / `num_domains` out of range. | | `502` | Upstream Transparency Center failure — **not billed**. | diff --git a/api-reference/endpoint/web-scraping/extract.mdx b/api-reference/endpoint/web-scraping/extract.mdx new file mode 100644 index 0000000..fc37965 --- /dev/null +++ b/api-reference/endpoint/web-scraping/extract.mdx @@ -0,0 +1,197 @@ +--- +title: Extract Data +description: Scrape a URL and pull fields out of it with CSS/XPath selectors, AI, or both. +--- + +Fetches the page the same way as [`/v1/web/scrape`](/api-reference/endpoint/web-scraping/scrape), then extracts what you asked for: + +- `extract_rules` run CSS or XPath selectors over the HTML and return `data`. +- `ai_extract_rules` and `ai_query` ask an LLM about the page and return `ai_extraction`. + +Billing works like a scrape, plus the AI extraction credits when AI is requested and succeeds. Broken selectors are rejected with `422` before anything is fetched. + +## Request Body + + + The page to extract from. + + + + Field name → selector. A selector starting with `/` or `(` is treated as XPath; anything else is CSS. CSS supports `::text` and `::attr(name)`. + + For more control, pass an object instead of a string: + - `selector` (required): the CSS or XPath selector. + - `type`: `css` or `xpath`. Inferred when unset. + - `all`: return every match as a list instead of the first one. + - `output`: `text` (default) or `html` (the element's outer HTML). + + ```json + { + "title": "h1", + "price": "//span[@class='price']", + "links": { "selector": "a::attr(href)", "all": true } + } + ``` + + + + Field name → plain-language description. The AI returns a JSON object with exactly these keys. + + ```json + { "price": "the product price with currency", "in_stock": "true if the product can be bought" } + ``` + + + + A freeform question about the page. When combined with `ai_extract_rules`, the answer is added under an `answer` key. + + + + Render the page in a browser before extracting. Use this for pages that build their content with JavaScript. + + + + CSS selector to wait for before extracting (browser render). + + + + ISO 3166-1 alpha-2 country code for the proxy exit. + + + + Proxy pool: `simple`, `premium` or `ultra`. + + +At least one of `extract_rules`, `ai_extract_rules` or `ai_query` is required. Unknown fields are rejected with `422`. + +## Response + + + `true` when the page was fetched and extraction ran. + + + + The final URL after redirects. + + + + HTTP status code of the fetched page. + + + + One key per `extract_rules` field: the first match as a string, a list with `all: true`, or `null` when nothing matched. `null` when no `extract_rules` were given. + + + + The AI's JSON answer. `null` when no AI was requested. + + + + The model that produced `ai_extraction`. + + + + Why AI extraction failed, when it did. Selector results in `data` are still returned. + + + + Engine that fetched the page. + + + + Credits charged for this request. + + + + Total time in milliseconds. + + +## Examples + + + +```bash cURL +curl -X POST "https://scrapebadger.com/v1/web/extract" \ + -H "x-api-key: YOUR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "url": "https://news.ycombinator.com", + "extract_rules": { + "top_story": ".titleline a", + "links": {"selector": ".titleline a::attr(href)", "all": true} + } + }' +``` + +```python Python +import requests + +response = requests.post( + "https://scrapebadger.com/v1/web/extract", + headers={"x-api-key": "YOUR_API_KEY"}, + json={ + "url": "https://example.com/product/123", + "ai_extract_rules": {"price": "the product price", "name": "the product name"}, + }, +) +print(response.json()["ai_extraction"]) +``` + +```javascript JavaScript +const res = await fetch("https://scrapebadger.com/v1/web/extract", { + method: "POST", + headers: { "x-api-key": "YOUR_API_KEY", "Content-Type": "application/json" }, + body: JSON.stringify({ + url: "https://example.com", + ai_query: "What is this page for, in one sentence?" + }) +}); +console.log((await res.json()).ai_extraction); +``` + + + +## Error Responses + +| Status | Description | +|--------|-------------| +| `402` | Insufficient credits | +| `422` | Invalid request or selector, or the target blocked the request (not billed) | +| `429` | Rate limit exceeded | +| `502` | The page could not be fetched, or AI-only extraction returned nothing (not billed) | + + + +```json 200 — Selectors +{ + "success": true, + "url": "https://news.ycombinator.com/", + "status_code": 200, + "data": { "first_title": "Bitwarden Dual License Model" }, + "ai_extraction": null, + "ai_model": null, + "ai_error": null, + "engine_used": "http", + "credits_used": 2, + "duration_ms": 1384 +} +``` + +```json 200 — AI query +{ + "success": true, + "url": "https://www.example.com/", + "status_code": 200, + "data": null, + "ai_extraction": { + "answer": "This page is a documentation example domain that should not be used for testing or monitoring." + }, + "ai_model": "google/gemini-2.5-flash", + "ai_error": null, + "engine_used": "http", + "credits_used": 4, + "duration_ms": 1188 +} +``` + + diff --git a/api-reference/endpoint/web-scraping/scrape.mdx b/api-reference/endpoint/web-scraping/scrape.mdx index a2c57c9..2e8ba94 100644 --- a/api-reference/endpoint/web-scraping/scrape.mdx +++ b/api-reference/endpoint/web-scraping/scrape.mdx @@ -245,7 +245,19 @@ openapi: POST /v1/web/scrape - Capture a full-page screenshot (PNG). Forces the browser engine. Returned as base64 in the `screenshot_url` response field. + Capture a PNG screenshot of the rendered page. Forces the browser engine. Returned in the `screenshot_url` response field as a `data:image/png;base64,…` URI. Captures the viewport unless `screenshot_full_page` is set. For a screenshot-only call, [`POST /v1/web/screenshot`](/api-reference/endpoint/web-scraping/screenshot) returns the bare base64 PNG. + + + + With `screenshot: true`, capture the whole scrollable page instead of the viewport. + + + + Browser viewport width in pixels for the render (`320` – `3840`). + + + + Browser viewport height in pixels for the render (`240` – `4320`). @@ -390,7 +402,7 @@ checked against the estimate *including* the proxy surcharge. - Base64-encoded PNG screenshot of the page. Only present when `screenshot: true` was requested. + The PNG screenshot as a `data:image/png;base64,…` URI. Only present when `screenshot: true` was requested. diff --git a/api-reference/endpoint/web-scraping/screenshot.mdx b/api-reference/endpoint/web-scraping/screenshot.mdx new file mode 100644 index 0000000..4e2c78a --- /dev/null +++ b/api-reference/endpoint/web-scraping/screenshot.mdx @@ -0,0 +1,138 @@ +--- +title: Screenshot +description: Render a URL in a real browser and get a PNG screenshot back. +--- + +Loads the page in the browser engine and returns a PNG of the viewport, or of the whole scrollable page with `full_page`. It is billed like a browser scrape plus the proxy tier, the same as `POST /v1/web/scrape` with `screenshot: true`. If the page loads but no screenshot comes back, the call returns `502` and costs **0 credits**. + +## Request Body + + + The page to capture. Must be a valid HTTP or HTTPS URL. + + + + Capture the whole scrollable page instead of only the viewport. + + + + Viewport width in pixels. Range: `320` – `3840`. + + + + Viewport height in pixels. Range: `240` – `4320`. + + + + CSS selector to wait for before capturing, e.g. `"#main"`. + + + + ISO 3166-1 alpha-2 country code for the proxy exit, e.g. `"US"`. + + + + Proxy pool: `simple`, `premium` or `ultra`. Same pricing as on [`/v1/web/scrape`](/api-reference/endpoint/web-scraping/scrape). + + +Unknown fields are rejected with `422`. + +## Response + + + `true` when a screenshot was captured. + + + + The final URL after redirects. + + + + HTTP status code of the captured page. + + + + Always `image/png`. + + + + The PNG image, base64-encoded (no `data:` prefix). + + + + Browser engine that rendered the page. + + + + Credits charged for this request. + + + + Total time in milliseconds. + + +## Examples + + + +```bash cURL +curl -X POST "https://scrapebadger.com/v1/web/screenshot" \ + -H "x-api-key: YOUR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"url": "https://example.com", "full_page": true, "width": 1280}' \ + | jq -r .screenshot | base64 --decode > page.png +``` + +```python Python +import base64 +import requests + +response = requests.post( + "https://scrapebadger.com/v1/web/screenshot", + headers={"x-api-key": "YOUR_API_KEY"}, + json={"url": "https://example.com", "full_page": True, "width": 1280}, +) +response.raise_for_status() +with open("page.png", "wb") as f: + f.write(base64.b64decode(response.json()["screenshot"])) +``` + +```javascript JavaScript +import { writeFile } from "node:fs/promises"; + +const res = await fetch("https://scrapebadger.com/v1/web/screenshot", { + method: "POST", + headers: { "x-api-key": "YOUR_API_KEY", "Content-Type": "application/json" }, + body: JSON.stringify({ url: "https://example.com", full_page: true, width: 1280 }) +}); +const { screenshot } = await res.json(); +await writeFile("page.png", Buffer.from(screenshot, "base64")); +``` + + + +## Error Responses + +| Status | Description | +|--------|-------------| +| `402` | Insufficient credits | +| `422` | Invalid request, or the target blocked the request (not billed) | +| `429` | Rate limit exceeded, or the browser farm is at capacity (not billed) | +| `502` | The page loaded but no screenshot was captured (not billed) | + + + +```json 200 +{ + "success": true, + "url": "https://www.example.com/", + "status_code": 200, + "content_type": "image/png", + "screenshot": "iVBORw0KGgoAAAANSUhEUgAAAyAAAAO/CAIAAA...", + "engine_used": "cloakbrowser", + "credits_used": 6, + "duration_ms": 4210 +} +``` + + diff --git a/docs.json b/docs.json index 2f52a30..f1cf870 100644 --- a/docs.json +++ b/docs.json @@ -74,7 +74,9 @@ "group": "API Reference", "pages": [ "api-reference/endpoint/web-scraping/scrape", - "api-reference/endpoint/web-scraping/detect" + "api-reference/endpoint/web-scraping/detect", + "api-reference/endpoint/web-scraping/screenshot", + "api-reference/endpoint/web-scraping/extract" ] } ] diff --git a/google-ads-transparency/overview.mdx b/google-ads-transparency/overview.mdx index ef200d5..eb78d02 100644 --- a/google-ads-transparency/overview.mdx +++ b/google-ads-transparency/overview.mdx @@ -40,7 +40,9 @@ region. For a brand name, go through [`/ads/advertisers`](/api-reference/endpoint/google-ads-transparency/search-advertisers) - first and use the `advertiser_id` it returns. + first and use the `advertiser_id` it returns. Its name match has no typo + tolerance either: add `fuzzy=true` to find misspelled and look-alike + advertisers, including those behind look-alike domains. ```bash @@ -85,7 +87,7 @@ set. Keep paging. | -------- | ---- | ------- | | Search creatives | `GET /v1/google/ads/search` | 10 | | Creative detail | `GET /v1/google/ads/creative` | 5 | -| Search advertisers | `GET /v1/google/ads/advertisers` | 5 | +| Search advertisers | `GET /v1/google/ads/advertisers` | 5 (15 with `fuzzy=true`) | | Advertiser detail | `GET /v1/google/ads/advertiser` | 5 | ## Quickstart