The landscape in 2026
AI models are trained and evaluated on large amounts of web data, and AI teams of every size need web collection: foundation model builders, companies fine-tuning models on domain content, and teams building evaluation sets. At the same time, many websites have tightened their rules on automated traffic, and robots.txt files, terms of service and dedicated AI opt-out signals now commonly state preferences about AI crawling. Legal and regulatory attention on training data has grown.
So the job is two-sided: collect what you need efficiently, and do it in a way that respects publishers' choices and can be documented. Geonode supports both sides of the technical work: datacenter and residential proxies for your own crawlers, and Crawl, Map and Scraper APIs that return clean HTML or Markdown.
Start with permission and opt-outs
- Respect robots.txt, including rules aimed at AI crawlers. If a site disallows your crawler, don't collect from it.
- Check terms of service for restrictions on automated collection and AI use.
- Honour other opt-out signals that publishers use.
- Prefer licensed and open datasets where they meet your needs. Many publishers now license content for AI use.
- Identify your crawler honestly where appropriate, with a user agent and contact information.
A proxy is not a way around a publisher's "no". Using proxies to evade blocks on sites that have opted out creates legal and reputational risk.
Datacenter first, residential where justified
For sites that allow crawling, datacenter traffic is usually enough and far cheaper per GB. Rotating datacenter traffic spreads load across IPs to stay within per-IP limits. Residential traffic makes sense for permitted sources that still challenge datacenter ranges, or where you need location-specific content (regional language variants, local information). Route domain by domain.
| Source situation | Approach |
|---|---|
| Allows crawling, lenient | Rotating datacenter, polite rate |
| Allows crawling, challenges datacenter IPs | Rotating residential, polite rate |
| Disallows your crawler or AI use | Don't collect |
| Offers licensing or an API | Use it |
Crawl architecture
- Seed and discover: sitemaps, link graphs and Geonode's Map API (which discovers a site's URLs) help you find pages without crawling blindly.
- Queue by domain with per-domain rate limits and concurrency caps.
- Fetch HTML with HTTP clients; use rendering only where content requires JavaScript.
- Extract main content: boilerplate (navigation, footers, ads) wastes storage and training tokens. Markdown extraction keeps structure while dropping clutter. Geonode's Scraper and Crawl APIs can return Markdown directly.
- Deduplicate at URL and content level, including near-duplicates.
- Record provenance for every document: URL, fetch time, robots.txt status at fetch time, licence or terms notes.
Proxy tip: You can try every step in this guide with a small balance: Geonode bills residential bandwidth per GB with no expiry, so testing costs very little. See our complete Geonode guide for the full product overview.
Quality and safety filtering
- Filter out pages with personal data you don't intend to collect.
- Remove spam, auto-generated pages and low-quality content.
- Detect and drop block pages, error pages and consent walls.
- Track language and region for balance.
Costs
AI collection is volume-driven, so bandwidth is the main cost. HTML-only fetching, boilerplate stripping before storage, and routing permitted lenient sites through datacenter traffic keep it manageable. For very high, steady residential volumes, compare per-GB pricing with Geonode unlimited residential plans. For smaller targeted collections, API per-request pricing can be simpler.
Documentation
Keep a data card or similar record for each dataset: sources, collection dates, filtering steps, opt-outs honoured, licences, known limitations and personal data handling. Regulators, partners and your own future team will ask.
A responsible collection checklist
Before collection starts:
- Purpose of the dataset written down, with the minimum data needed
- Source list reviewed for robots.txt, AI-specific opt-outs and terms
- Licensed or open alternatives considered
- Personal data handling plan: filtering, retention, access
- Crawler identity and contact details decided
During collection:
- Per-domain rate limits and concurrency caps enforced
- Datacenter traffic by default, residential only for permitted sources that need it
- Opt-out status recorded at fetch time for each document
- Block pages, consent walls and errors filtered out
- Monitoring of success rates and bandwidth per domain
After collection:
- Deduplication and quality filtering applied
- Provenance stored with every document
- Data card written: sources, dates, filters, limitations
- Process for removing content on request
Evaluation sets are different
Not all AI data work is about scale. Evaluation sets are small, carefully chosen collections used to test models, and they benefit from precise, location-aware collection: for example, checking how a model answers questions about local services in different countries against real local web content. For those, Geonode's country and city targeting and the Scraper API's Markdown output are often more useful than a large crawl. See Geonode for AI agents.
FAQ
Is it legal to collect web data for AI training?
It depends on jurisdiction, source terms, copyright, personal data and how you use the data. It's an evolving area; get legal advice for significant projects.
Should I use residential proxies for AI crawling?
Only for permitted sources that block datacenter traffic, or for location-specific content. Datacenter traffic is enough for most permitted crawling.
How do I reduce storage and token waste?
Extract main content, convert to Markdown, and deduplicate aggressively.
What if a site blocks my crawler?
Treat it as a signal. Check its robots.txt and terms, and don't try to evade the block.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.