🕷️ Universal Stealth Web Scraper

A small, asynchronous web scraping template built on Playwright and BeautifulSoup. It renders JavaScript-heavy pages in a real browser, applies stealth patches to reduce the automation fingerprint, retries transient failures, and writes the result to JSON or CSV.

Use it as a starting point for price monitoring, lead generation, market research, or any data pipeline that needs a rendered page rather than raw HTML.

---

🚀 Features

• 🥷 Stealth patches — playwright-stealth masks the most obvious automation tells.
• ⚡ Async — built on asyncio and Playwright's async API.
• 🔁 Retries with exponential backoff — transient timeouts and dropped connections are retried with jitter, so parallel runs don't retry in lockstep.
• 🌐 Proxy support — route traffic through a residential or datacenter proxy.
• ⏳ Waits for data, not just the document — wait_for_selector guards against pages that return 200 before the content renders.
• 💾 JSON and CSV output — either or both.
• ⚙️ CLI — everything configurable from the terminal, no code edits needed.
• 🧼 Whitespace normalised — text indented across several lines comes back as one clean string, so identical values do not look different.

---

🛠️ Tech Stack

• Python 3.10+
• Playwright (browser automation)
• playwright-stealth 2.x
• BeautifulSoup4 (HTML parsing)

---

📦 Installation

git clone https://github.com/Whyslab/universal-web-scraper-template.git
cd universal-web-scraper-template
pip install -r requirements.txt
playwright install chromium

The last step downloads the browser Playwright drives (~115 MB). It is required — skipping it produces an "Executable doesn't exist" error on the first run.

---

▶️ Usage

python scraper.py --url "https://quotes.toscrape.com" --selector "span.text"

That writes results.json and results.csv next to the script.

Options

Flag	Default	Description
--url	required	Page to scrape
--selector	required	CSS selector to extract, e.g. h2.title
--output	results	Output file name, without extension
--format	both	json, csv, or both
--proxy	—	Proxy as http://user:pass@ip:port
--retries	3	Attempts before giving up

Through a proxy

python scraper.py \
  --url "https://example.com/catalog" \
  --selector ".product-price" \
  --proxy "http://user:pass@1.2.3.4:8080" \
  --format json

---

🧪 Tests

pip install -r requirements-dev.txt
pytest -q

14 tests, no browser needed. extract_items() is deliberately split out from the Playwright path so parsing can be tested directly, and scrape_data() takes an optional fetcher so the retry loop can be driven without a network. They cover selector extraction, whitespace normalisation, JSON/CSV/both output, the empty-result path, CSV columns following the first row, and that the retry loop stops at the limit instead of looping forever.

CI runs ruff check, ruff format --check and the suite on Python 3.10 and 3.12.

---

🧩 Extending it

scrape_data() returns a list of dicts, one per matched element:

[{"content": "…"}, {"content": "…"}]

For structured records, edit extract_items() to pull several fields off each card instead of one string:

return [
    {
        "title": item.select_one("h2").get_text(strip=True),
        "price": item.select_one(".price").get_text(strip=True),
    }
    for item in soup.select(selector)
]

save_results() derives CSV columns from the first row's keys, so new fields appear in the output automatically.

---

⚠️ Notes

• Stealth patches reduce the automation fingerprint; they are not a guarantee against a determined anti-bot system.
• Check the target site's robots.txt and terms of service, and keep request rates reasonable.

---

📄 License

MIT — see LICENSE.