Proxies for Web Scraping

Most scraping projects fail for the same reason: they treat every target the same way. A sitemap and a marketplace product page have nothing in common — one is a static file, the other runs ASN checks, request-rate scoring and TLS fingerprinting. Paying for residential traffic on the first is waste; using a datacenter IP on the second is a guaranteed block. The work is in the classification, not in the proxy list.

Classify the target before you choose a pool

The practical consequence: route per host. A single crawl usually touches a protected HTML page, an unprotected JSON API and an image CDN, and only the first needs a residential exit. Splitting traffic this way commonly cuts residential gigabytes by a third without touching success rate.

Rotate per request, and control the two things the target can see

You cannot control how a site scores IP reputation. You can control requests per IP and request shape. Rotating per request handles the first; realistic headers, a real user agent, cookie persistence and paced intervals handle the second.

# Scrapy: rotation at the middleware level + a humanly possible rate
# settings.py
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY = 1.0
RANDOMIZE_DOWNLOAD_DELAY = True
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
RETRY_TIMES = 2
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": None,
    "myproject.middlewares.RotatingProxy": 543,
}

# middlewares.py — new session id per request
import uuid

class RotatingProxy:
    def __init__(self, pwd):
        self.pwd = pwd

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.get("PROXY_PASSWORD"))

    def process_request(self, request, spider):
        sid = uuid.uuid4().hex[:12]
        user = "ACCOUNT-country-us-sid-%s" % sid
        request.meta["proxy"] = "http://%s:%s@GATEWAY_HOST:PORT" % (user, self.pwd)

Do not render what you can read

A headless browser is 10 to 50 times the bandwidth of a plain HTTP request, and gigabytes are what you are buying. Check three things before you reach for Playwright: does the page ship data in JSON-LD or a __NEXT_DATA__ block, is there a public or internal JSON endpoint the page calls, and does the HTML already contain what you need without JavaScript. In practice, roughly two thirds of product and listing pages answer yes to one of those.

# price and stock often sit in structured data, no browser needed
import json, re, requests

r = requests.get(url, proxies=PROXY, timeout=30)
m = re.search(r'', r.text, re.S)
if m:
    data = json.loads(m.group(1))
    print(data.get("offers", {}).get("price"))

What a page actually costs

Work in billable gigabytes, not in requests. Take a 400 KB HTML page, fetched once, at a 95% success rate:

Two things follow. First, retries are not free — every blocked request is paid for and produces nothing, which is why success rate dominates the price comparison. Second, the cheapest optimisation is usually fewer bytes per page, not a cheaper proxy: dropping a headless browser in favour of the embedded JSON often cuts cost by 80% before you negotiate anything.

A crawl loop that survives real targets

1. Fetch the cheap way first. Plain HTTP, no browser, per-request rotation.
2. Detect the block, not just the status. A 200 page that says "unusual traffic" is a block. Match on known markers, page length and the presence of the field you came for.
3. Escalate once. If the field is missing, retry with a new session; if it is still missing, try the country-specific exit; only then consider rendering.
4. Record the outcome per host. Success rate, median latency, bytes per usable row. This table tells you which targets deserve residential traffic and which do not.
5. Back off on 429. Cut concurrency per domain in half and raise the delay; do not simply add more IPs.

Concurrency, in numbers you can act on

Treat these as starting points to tune against your own 403 rate, not as limits. The single biggest mistake is raising concurrency to compensate for a low success rate — that makes the block worse, not better.

Frequently asked questions