The short answer
A honeypot is bait. Sites add links hidden with CSS, or form fields that are invisible to people, and wait. A human never clicks them; a careless crawler that follows every link or fills every field does. The site then flags or bans that IP and session.
How to avoid them
Only follow links a real visitor could see, respect nofollow and robots.txt, do not fill hidden fields, and crawl the pages you actually need rather than everything.
Also known as
Bot trap, spider trap, hidden link trap.
Learn more
- How to avoid IP bans when scraping
- Ethical web scraping
- Web scraping proxies
- Headless browser proxies
- OWASP automated threats
- RFC 9309, robots.txt
- Web scraping overview
- MITRE ATT&CK, the defender's view of proxies
Affiliate disclosure: Team of Monkeys is an affiliate of Geonode, Webshare and Rayobyte. Links and banners marked as sponsored, and links to Geonode, Webshare and Rayobyte in the text, use our referral links, and we may earn a commission if you buy through them, at no extra cost to you. Our explanations and comparisons are written to be useful whichever provider you choose.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.