What real estate data collection involves
Real estate data comes from listing portals, agency websites, rental platforms, auction sites and public records. Teams collect it to track asking prices, rents, time on market, inventory and price cuts, or to feed valuation models. The work is repetitive: the same searches and listings checked daily or weekly.
Product choice per source
| Source | Typical defence | Webshare product |
|---|---|---|
| Agency websites, small portals | Light | Datacenter list |
| Public records and government data | Light, but rate-limited | Datacenter list, slow pace |
| Major national portals | Moderate to strong | Rotating residential |
| Regional portals that block foreign IPs | Geo-blocking | Datacenter or residential in that country |
Start with a Webshare datacenter list and move sources that block it to rotating residential traffic.
Local targeting
Most portals serve everyone in a country the same way, but some restrict or alter content for foreign visitors. Use IPs in the portal's country. City-level IPs are rarely needed for listing data, because location is in the search itself; save city targeting for sites that personalise by visitor location. See Webshare geo-targeting.
Collection pattern
- Search pages: run saved searches by area and property type, with a session per search so pagination works. See Webshare sticky sessions.
- Listing pages: fetch each listing once, then re-check on a schedule.
- Change tracking: store price, status and key fields; record changes with dates.
- Delisting detection: mark listings that disappear from searches, then confirm.
Data quality
- Deduplicate the same property listed by several agents or on several portals, using address normalisation and coordinates.
- Handle currencies and units (square feet vs square metres).
- Validate that extracted prices exist, so block pages aren't stored as data.
Budget tip: You can test most of this guide for free: Webshare gives every new account 10 datacenter proxies and 1 GB of bandwidth a month without a credit card. Move to a paid plan once the pilot works. See our complete Webshare guide.
Compliance
Listing data can include personal information (agent names, phone numbers, sometimes owner details). Collect only what you need, follow data protection rules, and check each portal's terms, many of which restrict automated access or reuse. Public records may have their own reuse rules. See ethical web scraping.
Comparing providers
For a Geonode-based approach to the same job, see Geonode for real estate data, and our head-to-head Webshare vs Geonode for real estate. Geonode offers non-expiring residential bandwidth, which suits weekly or monthly collection runs.
Example collection schedule
| Data | Source | Frequency | Webshare product |
|---|---|---|---|
| New listings | Portal search pages | Daily | Rotating residential, session per search |
| Price changes | Listing pages | Daily for active listings | Rotating residential |
| Agency inventory | Agency sites | Weekly | Datacenter list |
| Sold prices | Public records | Monthly | Datacenter list, slow pace |
| Rental listings | Rental platforms | Daily | Residential, test datacenter |
Estimating volume
A city with 20,000 active listings, checked daily at about 400 KB per HTML page, is roughly 240 GB a month if every listing is fetched every day. Fetching search pages daily and listing pages only when the search shows a change can cut that by 80% or more.
Building useful metrics
- Median asking price by area and property type.
- Price cuts: share of listings reduced, and average cut.
- Days on market from first seen to delisted.
- Inventory: active listings over time.
- Rent to price ratios where both are available.
These are far more useful to analysts than raw listing dumps.
Proxy tip: If your residential usage goes up and down, compare Geonode: its per-GB residential bandwidth doesn't expire, so you only pay for what you use. See Webshare vs Geonode.
Geography and mapping
Store coordinates where listings provide them, and geocode addresses otherwise. Consistent geography lets you aggregate by neighbourhood, school district or custom areas, and deduplicate the same property across portals.
A real-world example
A property investment firm tracks rental listings in five cities to spot rising rents early. Two national portals require residential IPs; the firm runs saved searches daily through Webshare residential sessions, one per city and property type, and fetches listing pages only when a listing is new or its price changes. Agency websites and public records go through a datacenter list weekly. The firm's dashboard shows median rents and days on market per neighbourhood, updated every morning.
If you're still shortlisting, compare this setup with the options in our best proxies guide, and use our cheap proxies guide to estimate what a sensible budget looks like.
Key takeaways
- Agency sites and public records usually accept datacenter traffic; major portals often need residential.
- Use sessions per search area so pagination works.
- Fetch listing pages only when searches show changes to save bandwidth.
- Deduplicate properties across portals and normalise units and currencies.
- Handle agent and owner details as personal data and check portal terms.
FAQ
Do I need residential proxies for real estate portals?
For major portals, often. For agency sites and smaller portals, datacenter IPs usually work.
How often should I re-check listings?
Daily for active markets and price tracking; weekly is enough for trend analysis.
Can I collect agent contact details?
Be careful. Contact details are personal data in many jurisdictions. Collect them only with a lawful basis.
Can I track sold prices with proxies?
Where sold prices are published on public records or portals, yes; check each source's reuse terms.
How do I deduplicate the same property on several portals?
Normalise addresses, compare coordinates, and match on key attributes like size and bedrooms.
Should I collect listing photos?
Only if your product needs them. Photos dominate bandwidth and storage.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.