Python · Beautiful Soup

Beautiful Soup never downloads anything. It reads HTML you have already fetched and lets you pick pieces out of it with CSS selectors or tree searches. So the proxy belongs to whatever fetches the page, and in most scripts that is requests. This guide fetches pages through a ProxyGuys proxy, hands them to Beautiful Soup, follows the "next page" links, and can move to a new IP every few pages. The library's manual is at crummy.com.

Before you start

You need Python 3 and a proxy that has been started; a new proxy waits as Pending until you press Start proxy on My Proxies. Install the two libraries, plus the SOCKS extra if you prefer the SOCKS5 port:

pip install requests beautifulsoup4
# if PROXY_URL will be a socks5h:// address
pip install "requests[socks]"

The examples use Python's built-in html.parser, so nothing else is required. lxml is an optional alternative parser if you already use it.

One environment variable for the proxy

On My Proxies, the Connection column has a copy button on the "HTTP proxy" box and another on the "SOCKS5 proxy" box. Each copies a complete URL with your login inside: http://USERNAME:PASSWORD@HOST:HTTP_PORT or socks5h://USERNAME:PASSWORD@HOST:SOCKS5_PORT. The h in socks5h makes the proxy resolve hostnames, so DNS goes through it too. Paste one into PROXY_URL:

# paste the URL from the "HTTP proxy" (or "SOCKS5 proxy") copy button
export PROXY_URL="http://USERNAME:PASSWORD@HOST:HTTP_PORT"
# export PROXY_URL="socks5h://USERNAME:PASSWORD@HOST:SOCKS5_PORT"

# optional: rotate between batches (from ⋮ → Copy change-IP link)
export PROXY_CHANGE_IP_URL="PASTE_YOUR_CHANGE_IP_LINK"

In PowerShell, use $env:PROXY_URL = "..." instead of export. If your password contains # or %, write them as %23 and %25 inside the URL, or set a new password without them (New password under Credentials).

Fetch one page and pull out a quote

quotes.toscrape.com is a practice site made for scraping. Each quote sits in a div.quote holding a span.text and a small.author. You can find selectors like these on any site by right-clicking an element and choosing Inspect in Chrome or Firefox.

import os

import requests
from bs4 import BeautifulSoup

proxy = os.environ["PROXY_URL"]
r = requests.get(
    "https://quotes.toscrape.com/",
    proxies={"http": proxy, "https": proxy},
    timeout=30,
)
r.raise_for_status()

soup = BeautifulSoup(r.text, "html.parser")
first = soup.select_one("div.quote")
print(first.select_one("small.author").get_text(strip=True))
print(first.select_one("span.text").get_text(strip=True))

Every page, with a new IP every few pages

The full script follows the li.next > a link until there is none, saves the results to quotes.json, and prints the IP it starts from. If PROXY_CHANGE_IP_URL is set, it changes the IP after every four pages.

import json
import os
import time
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

PROXY_URL = os.environ["PROXY_URL"]
CHANGE_IP_URL = os.environ.get("PROXY_CHANGE_IP_URL")  # optional
PAGES_PER_IP = 4
PROXIES = {"http": PROXY_URL, "https": PROXY_URL}

session = requests.Session()
session.headers["User-Agent"] = "Mozilla/5.0 (compatible; quote-collector/1.0)"
last_switch = 0.0


def get_soup(url):
    r = session.get(url, proxies=PROXIES, timeout=30)
    r.raise_for_status()
    return BeautifulSoup(r.text, "html.parser")


def exit_ip():
    r = session.get("https://api.ipify.org?format=json", proxies=PROXIES, timeout=30)
    return r.json()["ip"]


def switch_ip(old_ip):
    global last_switch
    time.sleep(max(0.0, 60 - (time.monotonic() - last_switch)))  # one change per 60 s
    last_switch = time.monotonic()
    # the reply can take as long as the change itself; a refusal is printed too
    print(requests.get(CHANGE_IP_URL, timeout=150).text.strip())
    for _ in range(12):  # watch for the new IP for up to two minutes
        time.sleep(10)
        session.close()  # so the check doesn't reuse a connection from before the change
        try:
            ip = exit_ip()
        except (requests.RequestException, ValueError, KeyError):
            continue
        if ip != old_ip:
            return ip
    return old_ip


ip = exit_ip()
print("starting from", ip)
quotes, url, pages = [], "https://quotes.toscrape.com/", 0

while url:
    soup = get_soup(url)
    for card in soup.select("div.quote"):
        quotes.append(
            {
                "text": card.select_one("span.text").get_text(strip=True),
                "author": card.select_one("small.author").get_text(strip=True),
                "tags": [a.get_text(strip=True) for a in card.select("a.tag")],
            }
        )
    nxt = soup.select_one("li.next > a")
    url = urljoin(url, nxt["href"]) if nxt else None
    pages += 1
    if url and CHANGE_IP_URL and pages % PAGES_PER_IP == 0:
        ip = switch_ip(ip)
        print("now on", ip)
    time.sleep(1)  # go easy on the site

with open("quotes.json", "w", encoding="utf-8") as f:
    json.dump(quotes, f, ensure_ascii=False, indent=2)
print(f"saved {len(quotes)} quotes from {pages} pages")

About the rotation part. The change-IP link (⋮ → Copy change-IP link) changes the IP when opened and needs no login, so treat it like a password; ⋮ → New change-IP link replaces it if it leaks. Its answer is a line of plain text, and the new IP usually takes 30 to 90 seconds to show up, so switch_ip checks the echo service every 10 seconds. Closing the session first makes sure the check runs on a new connection. The link works once every 60 seconds, which is why the function waits out the rest of that minute before using it again.

For a call that only returns once the new IP is live, use the API's rotate endpoint with a Bearer key, as in the requests guide. To rotate on a timer with no code, see Changing and rotating IPs.

Check the result

The "starting from" line should show an IP that isn't yours, and quotes.json should fill with entries. A lookup site might place the IP in another city or state than the one you selected. Lookup databases differ, and for mobile IPs that's normal.

When something goes wrong

• AttributeError: 'NoneType' object has no attribute 'get_text'. A selector found nothing, so the page you received isn't the one you expected. Print r.text[:500] to see what came back. If it is a block or challenge page, read Blocked or flagged IP.
• ProxyError with 407. The login in PROXY_URL is wrong or a # or % in the password isn't encoded. Copy the URL again from the Connection column.
• Missing dependencies for SOCKS support. You used a socks5h:// URL without the extra: pip install "requests[socks]".
• A timeout or dropped connection on the first request. The scheme and port don't match: http:// takes the HTTP port, socks5h:// the SOCKS5 port. The ⋮ → Details line shows the SOCKS5 port.
• Running from a whitelisted IP. Set PROXY_URL to the whitelist endpoint from Copy endpoint, with the scheme that matches the whitelist protocol and no login. The IP whitelist guide has the steps.