Proxies in research
Researchers collect web data for studies of prices, online discourse, misinformation, platform behaviour, algorithmic personalisation, accessibility, labour markets and much more. Two questions come up again and again: do I need proxies, and how do I use them in a way that will satisfy an ethics board, a journal's reviewers and my own standards?
The answer to the first is often "less than you think". The answer to the second is about transparency and restraint. This guide covers both, with Geonode as an example provider. Geonode's pay-as-you-go residential billing with non-expiring bandwidth suits research budgets, which are typically small and project-based.
Do you need proxies at all?
| Situation | Proxies needed? |
|---|---|
| Collecting from sites that publish data openly, at low rates | Usually not |
| Using official APIs or data access programmes | No; use the API |
| A site rate-limits your institution's IP | Possibly datacenter proxies to spread load politely |
| Studying how content varies by country or city | Yes, residential with location targeting |
| A site blocks university or cloud IP ranges | Possibly residential, if collection is otherwise appropriate |
| Studying personalisation or algorithmic differences | Yes, with careful experimental design |
Start without proxies. Add them only when the research design requires location variation, or when polite collection is still being blocked and the collection is otherwise justified.
Research designs that need location
Some of the most interesting web research is comparative: how do search results, prices, news recommendations, ad delivery or content moderation differ across countries? For these, location is the independent variable, and residential exits targeted by country or city are the instrument. Design carefully:
- Control everything else: fresh browser profiles, the same device type, the same time window, the same language settings where appropriate.
- Replicate: multiple sessions per location to separate real differences from noise.
- Verify location: record the exit IP's location and what the site reports.
- Randomise order of locations to avoid time-of-day bias.
Proxy tip: Before scaling, run a short pilot on your real targets with Geonode and record success rate, latency and GB per result. Our our complete Geonode guide explains which Geonode product to start with.
Ethics and approvals
- Ethics review: many institutions require review for web data collection, especially when it involves people's content or personal data.
- Personal data: collect the minimum, pseudonymise early, secure storage, defined retention. Data protection law applies to research too, though some laws provide research exemptions with conditions.
- Platform terms: some platforms offer researcher access programmes; use them where available.
- Harm to sites: polite rates, off-peak collection, caching, and never degrading a service.
- Harm to peers: residential traffic flows through real people's connections. Use it proportionately and choose providers with clear consent practices. Geonode states its residential IPs come from opt-in partners.
Documenting proxy use in your methods
Reviewers increasingly ask how data was collected. Record and report:
- Provider and product type (residential, datacenter)
- Targeting level (country, city) and how location was verified
- Session settings (rotating or sticky, duration)
- Request rates and collection dates
- How blocks and failures were detected and handled
- Limitations, such as geolocation inaccuracy
This makes your study reproducible and defensible.
Budget-friendly setup
- Start with no proxy, then datacenter, then residential only for the parts that need it.
- Fetch HTML only; avoid images and media unless they're your data.
- Buy a small non-expiring residential balance for the project.
- Ask providers about academic or nonprofit discounts.
See cheap proxies.
Budget tip: For jobs where a fixed IP list is enough, Webshare is one of the cheapest options: shared datacenter proxies are sold by the IP, and static residential (ISP) proxies cost well under a dollar per IP per month at the time of writing. More in Webshare for academic research.
A worked example: comparing search results across countries
A media researcher wants to study how search results for health-related queries differ across five countries.
- Query set: 40 queries, translated by native speakers for each country.
- Instrument: a headless browser with fresh profiles, routed through Geonode residential exits targeted to each country, with the search engine's language and region settings set to match.
- Protocol: each query run three times per country per day, at randomised times, for two weeks.
- Data: result URLs, positions, SERP features and the location the search engine reported.
- Ethics and limits: low request rates, no personal data collected, the study's method and proxy settings documented in the paper's appendix.
The design separates genuine country differences from day-to-day noise, and the documentation lets reviewers judge whether geolocation inaccuracy could affect conclusions.
Residential tip: If you want to try this on an ethically sourced residential pool, Rayobyte offers a free trial with no credit card, then pay-as-you-go traffic from $3.50 per GB that drops to $0.70 per GB at 1 TB. See how Rayobyte sources residential IPs.
Questions reviewers often ask
- How did you verify that exits were in the intended locations?
- How did you prevent personalisation from earlier sessions?
- How did you handle blocked or failed requests, and how many were there?
- Could the proxy provider's network characteristics have biased results?
- Was collection consistent with the platforms' terms and your ethics approval?
Answering these in your methods section, with numbers where possible, makes proxy-based research far more convincing. See ethical web scraping.
Many studies need residential proxies only for regional comparisons. To compare Geonode with other providers on cost and ethics, see our guide to the best proxies in 2026.
Choosing between providers for this? Our Webshare vs Geonode for Academic Research comparison shows where each one is stronger, with costs for typical workloads.
Considering Rayobyte instead of, or alongside, Geonode for this? Rayobyte vs Geonode compares the two, and How Rayobyte Sources Its Residential IPs, and Why It Matters covers the Rayobyte setup.
FAQ
Will reviewers object to proxy use?
Not if it's justified by the design and documented clearly. Undisclosed or unjustified evasion of blocks is what raises concerns.
Can I use proxies to get around a platform's research restrictions?
That's risky ethically and legally. Use official research access where it exists, and get advice from your institution.
How do I verify my exits are where I think?
Log exit IPs, look them up in more than one geolocation database, and check what the target site displays.
What's a typical research budget for proxies?
Often small: a few GB of residential traffic for a comparative study, plus cheap datacenter traffic for bulk collection.
Sources and further reading
Independent references that back up the points above, from standards bodies, governments, universities and non-profits. They open in a new tab.
- Bandwidth (computing) (Wikipedia): What bandwidth means, the unit most residential proxies are priced by.
- Regulation (EU) 2016/679 (GDPR) (EUR-Lex, European Union): The full text of the EU's General Data Protection Regulation.
- Introduction to robots.txt (Google Search Central): Google's own guide to how it reads robots.txt.
- Proxy Auto-Configuration (PAC) file (MDN Web Docs (Mozilla)): The reference for writing PAC files.
- Berkman Klein Center for Internet & Society (Harvard University): Harvard's centre for research on the internet and society.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.