Proxies for Web Scraping: Setting Up a Mobile Proxy in Python

Updated

Scrapers that stop working usually fail in one of two ways: the target site stops trusting the address the traffic comes from, or the script keeps sending from one address long after the site has noticed it. A mobile proxy helps with the first problem, and changing the IP on demand helps with the second. This guide connects a ProxyGuys proxy to a Python scraper step by step, then covers picking a US city and the errors you are most likely to run into.

Why scrapers reach for mobile IPs

With a ProxyGuys proxy, requests leave through a phone on a US carrier network (AT&T, T-Mobile or Verizon) over 4G and 5G. Carriers commonly put many subscribers behind a shared public address, so a site that bans a mobile IP outright risks locking out its own visitors. That is the main reason mobile addresses get more patience than addresses owned by hosting companies. It is not a free pass, though: a script that pulls pages faster than any person would still gets challenged, so pacing and IP changes matter as much as the type of IP.

What you need

  • Python 3 with requests (pip install requests). For SOCKS5, add the extra: pip install "requests[socks]".
  • A proxy that reads Running in My Proxies. A new one arrives as Pending · starts on activation; press Start proxy to begin its time.
  • For IP changes from code, an API key: open API in the sidebar and press Create key. The key is shown once, so save it straight away.

Step 1: keep the proxy URL out of your code

In My Proxies, the copy button on the proxy's HTTP Connection box gives you a complete URL in the form http://USERNAME:PASSWORD@HOST:HTTP_PORT. Store it, and the API key, as environment variables so neither ends up in a repository:

export PROXY_URL="http://USERNAME:PASSWORD@HOST:HTTP_PORT"
export PROXYGUYS_API_KEY="YOUR_API_KEY"

Proxy passwords may contain ! # $ % & *. Some of those characters mean something special inside a URL, so a password that contains them should be percent-encoded (Python's urllib.parse.quote does this), or the URL gets misread. To sidestep that, New password on the proxy lets you set one made only of letters and digits (6 to 100 characters).

Step 2: send a first request

import os
import requests

proxy = os.environ["PROXY_URL"]
session = requests.Session()
session.trust_env = False  # so an HTTP_PROXY / HTTPS_PROXY set on this machine can't take over
session.proxies.update({"http": proxy, "https": proxy})

r = session.get("https://api.ipify.org?format=json", timeout=30)
print(r.json())  # the mobile IP your scraper now uses

Reusing one Session for every request keeps connections open between calls instead of setting up a new one each time. If the printed IP is your own, the proxy settings were not applied; if you get an error, see the troubleshooting list further down.

Step 3 (optional): use SOCKS5 instead

The SOCKS5 Connection box copies socks5h://USERNAME:PASSWORD@HOST:SOCKS5_PORT. The final h matters: it tells requests to let the proxy resolve hostnames, so DNS lookups do not leave from your own network. The API's /credentials call returns the URL with plain socks5://, so change it to socks5h:// before using it. Nothing else in the script changes.

Step 4: take a fresh IP between batches

Changing the IP at natural break points, such as after each batch of pages, spreads your traffic over several addresses without cutting requests off halfway. The API call below waits until the new IP is live, so the next batch starts clean. One change is allowed per 60 seconds; a call made too early is answered with 409, code: "cooldown" and a retry_after value in seconds, which the loop honours.

import os
import time
import requests

API = "https://proxyguys.com/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['PROXYGUYS_API_KEY']}"}
PROXY_ID = "pp123456"  # the API id from GET /proxies, not the username

def fresh_ip(tries=3):
    for _ in range(tries):
        r = requests.post(f"{API}/proxies/{PROXY_ID}/rotate", headers=HEADERS, timeout=150)
        if r.status_code == 409 and r.json().get("code") == "cooldown":
            time.sleep(int(r.json().get("retry_after", 60)))
            continue
        r.raise_for_status()  # any other refusal stops here; its "error" field says why
        return r.json().get("new_ip")  # can be None
    raise RuntimeError("still cooling down")

for batch in batches:        # your own list of URL lists
    for url in batch:
        page = session.get(url, timeout=30)
        handle(page)         # your parsing code
    fresh_ip()

There are two other ways to change the IP. The change-IP link (proxy ⋮ menu, Copy change-IP link) works without a key, which also means it belongs only in places you trust. Automatic rotation lives in the Rotation column: IP rotation set to Every 5 min, Every 10 min, Every 30 min or Hourly keeps the city and carrier. A timer does not know where your script is, so a change can land in the middle of a request; changing between batches avoids that.

Step 5: choose the city

Prices, search results and listings often depend on where the visitor appears to be. Every plan includes every location, covering cities across the US. In My Proxies, click the proxy's location (shown as City, ST), type in Search city or state…, narrow by carrier with the chips if needed, and press Connect. From code, GET /locations?available_only=1 returns cities with their location_id, and POST /proxies/{id}/relocate with {"location_id": "…"} moves the proxy. Leave about three minutes between moves.

Troubleshooting

  • 407 Proxy Authentication Required: the username or password was refused. Copy the URL again from the Connection box, check the password encoding, and update the environment variable if you changed the password.
  • Timeouts or refused connections: check that the status reads Running. Pending means Start proxy has not been pressed; a Renew button means the proxy's time has run out. If anyone on the account used Change gateway IP, every proxy moved to a new host IP, so copy the host again.
  • 401 from the API: the key is missing, mistyped or revoked.
  • 409 from the API: on a rotate with code: "cooldown", or on a move with too_soon, you asked too soon; wait the number of seconds in retry_after. Any other 409 is a different refusal, so read the error field.
  • An IP checker shows another city: lookup sites use their own databases, so a mobile IP can appear in a different city or state than the one you picked. That is normal for mobile networks.
  • The IP changed by itself: carriers reassign addresses, and if a phone drops, the proxy moves to another phone at the same location. Write the scraper so that a new exit IP partway through a run is not treated as an error.

Frequently asked questions

Should a scraper use the HTTP port or the SOCKS5 port?
Either works with Python requests. HTTP needs no extra package. SOCKS5 needs requests[socks], and with a socks5h:// URL the proxy also resolves hostnames for you. Both ports are shown on your proxy in My Proxies.
How often can I change the IP?
On demand, once every 60 seconds, with the Change IP button, the change-IP link or the API. A change usually takes 30 to 90 seconds. Automatic rotation can run every 5, 10 or 30 minutes or hourly.
Is there a data cap on scraping traffic?
No. ProxyGuys plans come with unlimited bandwidth and no data caps.
Can a server use the proxy without a password in the script?
Yes. Open IP whitelist from the proxy's ⋮ menu, add the server's IPv4 address (up to 15 addresses), pick HTTP or SOCKS5, press Enable IP auth, and connect to the whitelist port shown there with no username or password. A few servers don't offer the whitelist, and the window says so.
How much does a proxy cost?
A week is $25 and a month is $80. Every plan includes every location.

Related articles