Requests first, browser last

Most scraping projects start by launching a browser, because the first page someone tries renders its content with JavaScript and a browser is the thing that obviously works.

Then it runs across ten thousand pages and the bill arrives — in memory, in wall-clock time, and in the operational misery of a box with forty orphaned Chrome processes on it.

The discipline worth building early is an escalation ladder: use the cheapest method that actually returns the data, and move up only when forced.

The ladder

1. The site's own API. Before anything else, open the network tab and watch what the page requests. A site that renders client-side is, by definition, fetching its data from somewhere — very often a clean JSON endpoint you can call directly.

This is the best outcome available. It is faster than parsing HTML, it returns structured data with no extraction step, and it is dramatically more stable, because a JSON response shape changes far less often than a page's markup. A surprising share of scraping jobs collapse into a handful of HTTP calls once someone spends twenty minutes looking.

2. A plain HTTP request and an HTML parser. Milliseconds, a few hundred kilobytes, no process to supervise. If the data is in the response body, this is where the job should stay.

3. A real browser. Only when the content genuinely does not exist until scripts have run.

The distance between steps two and three is not incremental. A browser is a hundred-odd megabytes of resident memory, a second or more of startup, and a process that can hang. At any volume it is the difference between a job that runs on a small VPS and one that needs a fleet.

Let Scrapy own the crawl

For anything past a handful of pages, the crawl itself is a solved problem and worth not re-solving. Scheduling, deduplication, retry with backoff, concurrency limits, per-domain throttling, and a pipeline to push extracted items through — Scrapy has all of it, tested against far more sites than any of us will write.

The common mistake is writing a for loop over URLs with requests and then gradually reimplementing a worse version of that list, one production incident at a time.

Keep the browser out of the crawler where you can. When a specific page type needs one, isolate that: the crawler discovers and schedules, and only the handful of routes that need rendering pay for it.

Running browsers on a VPS without losing the box

When Selenium is genuinely required, treat browsers as hostile tenants rather than as library calls.

One browser per worker, with a hard cap on workers. Memory is the binding constraint, not CPU. Size the cap against available RAM with headroom, and let work queue rather than launching an unbounded number of browsers.

Recycle the process on a schedule. Long-lived browser sessions leak — memory, temp files, stale profile directories. Restarting after a fixed number of pages is cruder than finding the leak and considerably more reliable.

Set every timeout, then a supervisor timeout above them. Page load, script, implicit wait, and then an outer limit that kills a worker which has sailed past all of them. A hung browser holds its memory forever.

Reap orphans explicitly. A crashed worker leaves its browser behind. Something must clean up processes whose parent is gone, or the box degrades over days in a way that looks like a mystery.

Rate limits are an engineering constraint, not an obstacle

Fetch politely: a sane concurrency limit per domain, real delays between requests, exponential backoff on 429 and 5xx, an honest user agent, and robots.txt respected.

This is partly about not being a problem for somebody else's server. It is also, pragmatically, the thing that keeps a scraper working. A job that hammers a host gets blocked, and then you are maintaining an adversarial system instead of a data pipeline — which is a much worse job and one that breaks every time the other side changes something.

Check the terms of service, and be deliberate about personal data. "Technically reachable" and "appropriate to collect and store" are different questions, and the second one is the one that matters when someone asks later.

Cache aggressively while developing. There is no reason to re-fetch a page fifty times while iterating on a selector.

Extraction breaks silently, so make it loud

The page will change. A class name gets renamed in a redesign nobody told you about, and your selector starts returning nothing.

The dangerous property is that this usually does not raise an error. It returns an empty list, the pipeline writes zero rows, the job exits successfully, and the dashboard shows a flat line that could just as easily be a quiet week.

Two habits fix it:

Validate every extracted item against a schema. Required fields, expected types, plausible ranges. An item missing its price is a failure, not a null.

Alert on yield, not just on errors. If a run that normally produces nine thousand items produces two hundred, something is broken regardless of the exit code. Track the rate and page when it moves.

Keep the raw response

Store the raw HTML or JSON alongside the extracted record, at least for a rolling window.

When you discover the extraction was subtly wrong for three weeks — a currency misparsed, a field taken from the wrong column — you can re-extract from what you already have. Without it, the only remedy is to crawl the whole thing again, against a site that has since moved on, for data that no longer exists in the form you needed.

Storage is cheap. Re-crawling is not, and sometimes it is impossible.

The summary

Ask whether the site has an API. Use plain HTTP where the data is in the response. Let Scrapy run the crawl. Reach for a browser only for the pages that truly need one, and supervise it like the liability it is. Validate what you extract, alert on yield, and keep the raw payload so today's bug is tomorrow's reprocessing job rather than tomorrow's re-crawl.