Rotating Proxies with Python Requests: A Copy-Paste Starter Kit

If you're scraping, monitoring prices, or checking rank data at any real volume, a single IP will get rate-limited or blocked fast. Rotating proxies across requests is the standard fix, but most tutorials either hand-wave the retry logic or skip authentication entirely. This is a copy-paste starter kit: four increasingly useful patterns for rotating proxies with Python's requests library, from "just make it work" to "make it survive failures." Prerequisites pip install requests You'll need a pool of proxy addresses. Any provider that gives you host:port (or user:pass@host:port for authenticated proxies) works with the patterns below — for these examples we're using the format :, which is what you'd swap in from a provider like Squid Proxies. 1. The absolute minimum: one proxy, one request import requests proxy = "http://:" proxies = { "http": proxy, "https": proxy, } response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10) print(response.json()) That's it, requests routes both HTTP and HTTPS traffic through the same proxy dict. If your proxy requires auth, embed the credentials directly in the URL: proxy = "http://username:password@:" 2. Rotating across a pool One proxy isn't rotation, it's just a detour. Here's a pool with round-robin cycling using itertools.cycle: import requests from itertools import cycle PROXIES = [ "http://user:pass@proxy1.squidproxies.com:8000", "http://user:pass@proxy2.squidproxies.com:8000", "http://user:pass@proxy3.squidproxies.com:8000", ] def get_proxy_pool(proxies): return cycle(proxies) pool = get_proxy_pool(PROXIES) for _ in range(5): proxy = next(pool) proxies = {"http": proxy, "https": proxy} resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10) print(proxy, "->", resp.json()) cycle() loops the list indefinitely, so next(pool) always gives you the next proxy in sequence: proxy1, proxy2, proxy3, proxy1, proxy2, ... If you'd rather pick randomly than in sequence (useful for avoiding predictable request-timing patterns), swap in random.choice(PROXIES) instead. 3. Rotation with retries and backoff The real reason to rotate is that individual proxies fail, they get temporarily blocked, time out, or hit a rate limit on the target site. A rotation pattern without retry logic just fails louder. Here's a version that retries through the pool on failure, with exponential backoff between attempts: import time import random import requests def make_request(url, proxy, timeout=10): proxies = {"http": proxy, "https": proxy} try: resp = requests.get(url, proxies=proxies, timeout=timeout) resp.raise_for_status() return resp except requests.exceptions.RequestException as e: print(f"Request failed via {proxy}: {e}") return None def fetch_with_rotation(url, proxies, max_retries=3): tried = [] for attempt in range(max_retries): proxy = random.choice(proxies) tried.append(proxy) resp = make_request(url, proxy) if resp is not None: return resp time.sleep(2 ** attempt) # 1s, 2s, 4s... raise RuntimeError(f"All {max_retries} attempts failed. Tried: {tried}") response = fetch_with_rotation("https://httpbin.org/ip", PROXIES) print(response.json()) A few things worth calling out: raise_for_status() turns HTTP error codes (403, 429, 503) into exceptions, so a "successful" request that actually got blocked doesn't silently pass through as a 200. Exponential backoff (2 ** attempt) gives a rate-limited proxy a moment to cool down instead of hammering it again immediately. tried in the error message makes debugging which proxies are consistently failing much faster than a bare "all retries failed." 4. A reusable session class If you're making many requests in the same script, wrapping this in a class keeps connection pooling intact (via requests.Session) while still rotating the proxy per call: import random import requests class RotatingProxySession: def __init__(self, proxies): self.proxies = proxies self.session = requests.Session() def get(self, url, **kwargs): proxy = random.choice(self.proxies) self.session.proxies = {"http": proxy, "https": proxy} return self.session.get(url, **kwargs) def post(self, url, **kwargs): proxy = random.choice(self.proxies) self.session.proxies = {"http": proxy, "https": proxy} return self.session.post(url, **kwargs) client = RotatingProxySession(PROXIES) for i in range(5): resp = client.get("https://httpbin.org/ip") print(f"Request {i}: {resp.json()}") This is the pattern I'd actually reach for in a real scraping job, it's a small enough abstraction that you can drop in retry logic from section 3, custom headers, or per-domain proxy assignment without rewriting the calling code. Wrapping up Rotation solves the "one IP gets blocked" problem; retries with backoff solve the "one proxy in the pool is temporarily bad" problem. You want both, not one or the other. If you're pulling proxies from a provider dashboard rather than hardcoding a list, most APIs return the pool as JSON, you can drop that straight into the PROXIES list in any of the examples above. Next up in this series: the same patterns in async Python with aiohttp, for when synchronous requests become the bottleneck.