Proxies for Web Scraping: Setting Up a Mobile Proxy in Python
Updated
Scrapers that stop working usually fail in one of two ways: the target site stops trusting the address the traffic comes from, or the script keeps sending from one address long after the site has noticed it. A mobile proxy helps with the first problem, and changing the IP on demand helps with the second. This guide connects a ProxyGuys proxy to a Python scraper step by step, then covers picking a US city and the errors you are most likely to run into.
Why scrapers reach for mobile IPs
With a ProxyGuys proxy, requests leave through a phone on a US carrier network (AT&T, T-Mobile or Verizon) over 4G and 5G. Carriers commonly put many subscribers behind a shared public address, so a site that bans a mobile IP outright risks locking out its own visitors. That is the main reason mobile addresses get more patience than addresses owned by hosting companies. It is not a free pass, though: a script that pulls pages faster than any person would still gets challenged, so pacing and IP changes matter as much as the type of IP.
What you need
- Python 3 with
requests(pip install requests). For SOCKS5, add the extra:pip install "requests[socks]". - A proxy that reads Running in My Proxies. A new one arrives as Pending · starts on activation; press Start proxy to begin its time.
- For IP changes from code, an API key: open API in the sidebar and press Create key. The key is shown once, so save it straight away.
Step 1: keep the proxy URL out of your code
In My Proxies, the copy button on the proxy's HTTP Connection box gives you a complete URL in the form http://USERNAME:PASSWORD@HOST:HTTP_PORT. Store it, and the API key, as environment variables so neither ends up in a repository:
export PROXY_URL="http://USERNAME:PASSWORD@HOST:HTTP_PORT"
export PROXYGUYS_API_KEY="YOUR_API_KEY"
Proxy passwords may contain ! # $ % & *. Some of those characters mean something special inside a URL, so a password that contains them should be percent-encoded (Python's urllib.parse.quote does this), or the URL gets misread. To sidestep that, New password on the proxy lets you set one made only of letters and digits (6 to 100 characters).
Step 2: send a first request
import os
import requests
proxy = os.environ["PROXY_URL"]
session = requests.Session()
session.trust_env = False # so an HTTP_PROXY / HTTPS_PROXY set on this machine can't take over
session.proxies.update({"http": proxy, "https": proxy})
r = session.get("https://api.ipify.org?format=json", timeout=30)
print(r.json()) # the mobile IP your scraper now uses
Reusing one Session for every request keeps connections open between calls instead of setting up a new one each time. If the printed IP is your own, the proxy settings were not applied; if you get an error, see the troubleshooting list further down.
Step 3 (optional): use SOCKS5 instead
The SOCKS5 Connection box copies socks5h://USERNAME:PASSWORD@HOST:SOCKS5_PORT. The final h matters: it tells requests to let the proxy resolve hostnames, so DNS lookups do not leave from your own network. The API's /credentials call returns the URL with plain socks5://, so change it to socks5h:// before using it. Nothing else in the script changes.
Step 4: take a fresh IP between batches
Changing the IP at natural break points, such as after each batch of pages, spreads your traffic over several addresses without cutting requests off halfway. The API call below waits until the new IP is live, so the next batch starts clean. One change is allowed per 60 seconds; a call made too early is answered with 409, code: "cooldown" and a retry_after value in seconds, which the loop honours.
import os
import time
import requests
API = "https://proxyguys.com/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['PROXYGUYS_API_KEY']}"}
PROXY_ID = "pp123456" # the API id from GET /proxies, not the username
def fresh_ip(tries=3):
for _ in range(tries):
r = requests.post(f"{API}/proxies/{PROXY_ID}/rotate", headers=HEADERS, timeout=150)
if r.status_code == 409 and r.json().get("code") == "cooldown":
time.sleep(int(r.json().get("retry_after", 60)))
continue
r.raise_for_status() # any other refusal stops here; its "error" field says why
return r.json().get("new_ip") # can be None
raise RuntimeError("still cooling down")
for batch in batches: # your own list of URL lists
for url in batch:
page = session.get(url, timeout=30)
handle(page) # your parsing code
fresh_ip()
There are two other ways to change the IP. The change-IP link (proxy ⋮ menu, Copy change-IP link) works without a key, which also means it belongs only in places you trust. Automatic rotation lives in the Rotation column: IP rotation set to Every 5 min, Every 10 min, Every 30 min or Hourly keeps the city and carrier. A timer does not know where your script is, so a change can land in the middle of a request; changing between batches avoids that.
Step 5: choose the city
Prices, search results and listings often depend on where the visitor appears to be. Every plan includes every location, covering cities across the US. In My Proxies, click the proxy's location (shown as City, ST), type in Search city or state…, narrow by carrier with the chips if needed, and press Connect. From code, GET /locations?available_only=1 returns cities with their location_id, and POST /proxies/{id}/relocate with {"location_id": "…"} moves the proxy. Leave about three minutes between moves.
Troubleshooting
- 407 Proxy Authentication Required: the username or password was refused. Copy the URL again from the Connection box, check the password encoding, and update the environment variable if you changed the password.
- Timeouts or refused connections: check that the status reads Running. Pending means Start proxy has not been pressed; a Renew button means the proxy's time has run out. If anyone on the account used Change gateway IP, every proxy moved to a new host IP, so copy the host again.
- 401 from the API: the key is missing, mistyped or revoked.
- 409 from the API: on a rotate with
code: "cooldown", or on a move withtoo_soon, you asked too soon; wait the number of seconds inretry_after. Any other 409 is a different refusal, so read theerrorfield. - An IP checker shows another city: lookup sites use their own databases, so a mobile IP can appear in a different city or state than the one you picked. That is normal for mobile networks.
- The IP changed by itself: carriers reassign addresses, and if a phone drops, the proxy moves to another phone at the same location. Write the scraper so that a new exit IP partway through a run is not treated as an error.