Proxies for Price Monitoring

Price data is the most time-sensitive thing a retailer collects, and the easiest to get wrong in a way nobody notices for a month. A price scraped from the wrong city is not a small error; it is a pricing decision made on a number a customer never saw. This page covers the three things that actually decide data quality: the location you ask from, the cadence you run at, and whether you are reading the price the page displays or a number left over in the markup.

Location is part of the price

Marketplaces and large retailers vary availability, delivery estimates, tax display and sometimes the price itself by location. Collecting a single national feed and calling it "the price" produces a number that matches no customer's screen. Target the cities your customers actually buy from, and keep the location in the record.

# one sticky session per city: same shopper, same cart context
import requests

CITIES = [("ca", "losangeles"), ("ny", "newyork"), ("tx", "houston"), ("il", "chicago")]
PWD = "PASSWORD"

for state, city in CITIES:
    user = "ACCOUNT-country-us-state-%s-city-%s-sid-%s01-stime-30" % (state, city, city[:6])
    p = "http://%s:%s@GATEWAY_HOST:PORT" % (user, PWD)
    r = requests.get(product_url, proxies={"http": p, "https": p}, timeout=30)
    print(city, r.status_code, extract_price(r.text))  # helper defined below

Read the data, not the paint

Most product pages carry the price twice: once in visible markup, once in structured data or an embedded state object. The visible one is often a pre-discount list price, or a placeholder replaced by JavaScript after load. Extracting from the structured block is cheaper and more stable than rendering.

# JSON-LD first, then the embedded state object, then a fallback selector
import json, re

def extract_price(html):
    m = re.search(r'application/ld\+json">(.*?)', html, re.S)
    if m:
        d = json.loads(m.group(1))
        offers = d.get("offers") or {}
        if isinstance(offers, list):
            offers = offers[0]
        return offers.get("price"), offers.get("priceCurrency")
    m = re.search(r'"price"\s*:\s*"?([0-9.]+)', html)
    return (m.group(1), None) if m else (None, None)

Cadence: match it to how prices move

Every extra pass costs traffic whether or not the price changed. The useful trick is to diff before you re-crawl: store a hash of the product block, and only re-parse when it changes. On a catalogue with stable prices this typically cuts parsed volume by an order of magnitude even though you still fetch the page.

What a catalogue costs

Assume 100,000 SKUs, one pass per day, 400 KB per page, 95% success rate:

At $1/GB that is roughly $42 a day, or $1,260 a month, for a full daily pass. Two levers move that number far more than any unit-price negotiation: read JSON instead of rendering (often halves the bytes), and crawl only the SKUs that matter at hourly frequency while the long tail runs daily.

A monitoring loop that produces decisions, not just rows

1. Define the record. SKU, retailer, country, city, price, currency, promotion text, timestamp, HTTP status, extraction source. Without the source field you cannot debug a bad number a week later.
2. Validate before you store. A price that moved 70% in one hour is usually a parse error or a currency switch, not a repricing. Re-fetch once with a new session and compare.
3. Alert on the second confirmation. Two consecutive passes showing the same change is worth a notification; the first is worth a retry.
4. Keep an evidence copy. For MAP and channel disputes, store a screenshot hash or the raw HTML fragment alongside the price — a claim needs proof attached to the moment.
5. Review the target list monthly. Sites add protection; a host that ran fine on datacenter last quarter may need residential today.

Where teams get it wrong

• One national view. Treating a US-wide crawl as "the price" hides the city-level differences that actually drive conversion.
• Rendering every page. Paying 20x the bandwidth to read a number that was already in the HTML.
• Hourly everything. Most prices do not move hourly; the traffic does.
• No timestamp discipline. Two prices without timestamps cannot be compared, which makes the whole dataset unusable for a pricing decision.

Frequently asked questions