What job market data is used for
Job postings are one of the best real-time signals of the economy and of individual companies. Teams collect them to:
- Track hiring trends by role, skill, location and company.
- Benchmark salaries where ranges are published.
- Feed job aggregators and search engines.
- Spot competitor expansion and new products from hiring patterns.
Sources and how they behave
| Source | Typical defence | Webshare product |
|---|---|---|
| Company career pages and ATS-hosted job pages | Light | Datacenter list |
| Government and public job portals | Light, rate-limited | Datacenter list, slow pace |
| Large job boards | Moderate to strong | Rotating residential |
| Regional job boards | Sometimes geo-restricted | IPs in the board's country |
Many companies host jobs on applicant tracking systems with consistent page structures, which makes career-page collection efficient and usually possible on Webshare's cheap datacenter lists. Large boards are better handled with rotating residential traffic.
Collection pattern
- Discover job URLs from search pages, sitemaps or structured feeds.
- Fetch each posting; many include JSON-LD job data, which is easier to parse than HTML.
- Re-check active postings periodically and record when they close.
- Deduplicate the same job posted on several boards, using title, company, location and description similarity.
Data quality
- Normalise titles, locations and salary formats (hourly, monthly, yearly; currencies).
- Mark remote, hybrid and on-site consistently.
- Keep posting and closing dates for trend analysis.
- Validate pages so block pages aren't stored as jobs.
Compliance
Job postings are public, but job boards' terms often restrict automated collection and reuse. Recruiter names and contact details are personal data. Collect what you need, respect rate limits and check the rules for each source. See ethical web scraping.
Budget tip: Before scaling, run a short pilot with Webshare on your real targets and record success rate, latency and bandwidth per result. Our our complete Webshare guide explains which Webshare product fits which job.
Comparing providers
For a Geonode-based approach, see Geonode for job market data. Geonode offers non-expiring residential bandwidth that suits periodic collection runs.
Discovering career pages at scale
Many companies use a handful of applicant tracking systems with predictable URL patterns and public job feeds. Building a list of companies and their career page URLs, then fetching them through a datacenter list, is often the most efficient way to collect job data. Large boards add coverage for companies without their own pages.
Normalisation reference
| Field | Normalise to |
|---|---|
| Title | Standard role taxonomy plus seniority |
| Location | City, region, country, plus remote or hybrid flag |
| Salary | Annual amount in one currency, with original values kept |
| Employment type | Full-time, part-time, contract, internship |
| Dates | Posted, first seen, last seen, closed |
Trend metrics
- New postings per week by role and location.
- Median posting duration.
- Share of postings with salary ranges.
- Skills mentioned, by frequency.
Proxy tip: For a second opinion on hard targets, run the same test through Geonode residential proxies, which support sticky sessions up to 24 hours and ASN targeting. We compare the two in Webshare vs Geonode.
Respecting candidates and recruiters
Job data is about hiring, not people. Avoid collecting candidate information, and keep recruiter contact details out of datasets unless you have a clear lawful reason.
A real-world example
An HR analytics firm publishes a monthly hiring index for software roles. It collects postings from 4,000 company career pages hosted on common applicant tracking systems through a Webshare datacenter list, plus two large job boards through residential sessions. Postings are deduplicated across sources, titles are mapped to a standard taxonomy, and the index reports new postings, salary disclosure rates and remote share by city.
Not sure this is the right proxy type for the job? The best proxies in 2026 guide compares residential, ISP, datacenter and mobile options, and our cheap proxies guide covers budget picks that still work.
Key takeaways
- Company career pages and ATS-hosted jobs usually work on datacenter lists; large job boards often need residential.
- JSON-LD job data is easier to parse than HTML.
- Deduplicate jobs across boards and normalise titles, locations and salaries.
- Record posting and closing dates for trend analysis.
- Check board terms and avoid collecting personal data you don't need.
FAQ
Can I collect job postings with datacenter proxies?
From company career pages and many ATS platforms, usually yes. Large job boards often need residential IPs.
How do I avoid duplicate job postings?
Match on company, normalised title, location and description similarity, and keep a canonical record.
How often should I collect job data?
Daily for active monitoring; weekly is often enough for trend analysis.
Do job boards allow scraping?
Many restrict it in their terms. Check each source and consider official feeds or partnerships.
How do I detect closed postings?
Record the last date a posting was seen and confirm removal on the next check.
Can I collect salary data from postings?
Where postings publish salary ranges, yes; normalise them to a common unit and currency.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.