Python · Scrapy

Crawling with Scrapy over a ProxyGuys proxy

Scrapy has proxy support built in. Its HttpProxyMiddleware, switched on by default, picks up a proxy either from the standard proxy environment variables or from the proxy key in a request's meta, and turns any login in that URL into the right header. You only have to supply the address. Scrapy's own documentation is at docs.scrapy.org.

Get the HTTP proxy URL

Make sure the proxy is running: a newly bought one stays Pending until you press Start proxy on My Proxies. Then, in the Connection column, use the copy button on the "HTTP proxy" box. It copies a complete URL of the form http://USERNAME:PASSWORD@HOST:HTTP_PORT, which is exactly what Scrapy wants. If your password contains # or %, write those as %23 and %25; Scrapy decodes the login before sending it.

A spider to test with

Save this as books_spider.py. Its first request goes to an IP echo service and writes the address to the log, so every run tells you which IP it crawled from. It then walks the catalogue of books.toscrape.com, a site built for scraping practice.

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    # The first request only reports which IP the crawl is using.
    start_urls = ["https://api.ipify.org?format=json"]

    def parse(self, response):
        self.logger.info("Crawling from %s", response.json()["ip"])
        yield response.follow("https://books.toscrape.com/", callback=self.parse_listing)

    def parse_listing(self, response):
        for book in response.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css("p.price_color::text").get(),
            }
        next_href = response.css("li.next a::attr(href)").get()
        if next_href:
            yield response.follow(next_href, callback=self.parse_listing)

Option 1: environment variables, no code changes

The quickest route. Export the proxy URL and run the spider; the built-in middleware applies it to every request.

pip install scrapy

# macOS / Linux: the URL the "HTTP proxy" copy button gives you
export https_proxy="http://USERNAME:PASSWORD@HOST:HTTP_PORT"
export http_proxy="$https_proxy"
scrapy runspider books_spider.py -O books.json

# Windows PowerShell
$env:https_proxy = "http://USERNAME:PASSWORD@HOST:HTTP_PORT"
$env:http_proxy = $env:https_proxy
scrapy runspider books_spider.py -O books.json

Look for the "Crawling from" line in the output. It should show an IP that isn't your own. A lookup site may file that IP under another city or state than the one you picked; their databases differ, which is normal for mobile networks.

Option 2: a downloader middleware in your project

Inside a project made with scrapy startproject myproject, a small middleware keeps the proxy in code while the login stays in environment variables (PROXY_USER, PROXY_PASS, PROXY_HOST, PROXY_HTTP_PORT). It fills in meta["proxy"] only when a request hasn't set one, so a single request can still go elsewhere.

# myproject/middlewares.py
import os
from urllib.parse import quote


class ProxyGuysMiddleware:
    """Routes every request through the ProxyGuys HTTP port unless it names its own proxy."""

    def __init__(self):
        user = quote(os.environ["PROXY_USER"], safe="")
        password = quote(os.environ["PROXY_PASS"], safe="")
        self.proxy = (
            f"http://{user}:{password}@"
            f"{os.environ['PROXY_HOST']}:{os.environ['PROXY_HTTP_PORT']}"
        )

    def process_request(self, request, spider=None):  # newer Scrapy omits spider
        request.meta.setdefault("proxy", self.proxy)
# myproject/settings.py
DOWNLOADER_MIDDLEWARES = {
    # below 750, so it runs before Scrapy's own HttpProxyMiddleware
    "myproject.middlewares.ProxyGuysMiddleware": 350,
}

The number sets the order. Scrapy's HttpProxyMiddleware sits at 750, so 350 lets this one set the proxy first; the built-in one then adds the Proxy-Authorization header.

A fresh IP for each crawl

Scrapy runs on an event loop, so a blocking API call inside a spider would stall it. It's simpler to change the IP between crawls. This runner uses only the standard library: it asks the ProxyGuys API for a new IP (same city and carrier), then starts the crawl. You need an API key from Create key on the API page (it's shown once) in PROXYGUYS_API_KEY, and the proxy's API id, which looks like pp123456, in PROXYGUYS_PROXY_ID.

# run_crawls.py: a new IP before each crawl, then the crawl itself
import json
import os
import subprocess
import sys
import time
import urllib.error
import urllib.request

API = "https://proxyguys.com/api/v1"
# Skip any proxy variables you exported for Scrapy: call the API directly.
direct = urllib.request.build_opener(urllib.request.ProxyHandler({}))


def rotate(proxy_id, attempts=4):
    req = urllib.request.Request(
        f"{API}/proxies/{proxy_id}/rotate",
        method="POST",
        headers={"Authorization": "Bearer " + os.environ["PROXYGUYS_API_KEY"]},
    )
    for _ in range(attempts):
        try:
            # The API answers once the new IP is in place, usually 30 to 90 seconds.
            with direct.open(req, timeout=150) as resp:
                return json.load(resp).get("new_ip")
        except urllib.error.HTTPError as err:
            body = json.load(err)
            if err.code == 409 and body.get("code") == "cooldown":
                time.sleep(float(body.get("retry_after", 60)))
                continue
            raise SystemExit(f"rotate failed ({err.code}): {body.get('error')}")
    raise SystemExit("proxy still cooling down")


for run in range(1, 4):
    print(f"run {run}: new IP", rotate(os.environ["PROXYGUYS_PROXY_ID"]))
    subprocess.run(
        [sys.executable, "-m", "scrapy", "runspider", "books_spider.py", "-O", f"books-{run}.json"],
        check=True,
    )

A new IP can be requested once every 60 seconds. When a call comes too soon, the API answers 409 with "code": "cooldown" and retry_after in seconds, and the runner sleeps for that long before trying again. The reply's new_ip may be null; the "Crawling from" log line shows the address either way. Timed rotation, which needs no code at all, is covered in Changing and rotating IPs.

Troubleshooting

  • A 407 status, or a tunnel error that mentions 407. On https:// pages a refused login surfaces as a failed CONNECT tunnel rather than a response. Copy the password again with Copy password and check any # or % in it is encoded.
  • Timeouts or dropped connections on every request. Check that the URL uses the HTTP port. The ⋮ → Details line holds the SOCKS5 port, which Scrapy's standard download handler can't use.
  • The log shows your own IP. The proxy wasn't applied: the variables were exported in another terminal, HTTPPROXY_ENABLED is set to False, or the middleware path in DOWNLOADER_MIDDLEWARES doesn't match your project name.
  • Crawling from a whitelisted IP. With the whitelist protocol set to HTTP, use http:// plus the address from Copy endpoint, with no login. See the IP whitelist guide.
  • A site starts refusing you mid-crawl. See Blocked or flagged IP.