Cloudflare Accuses Perplexity AI of Using Stealth Crawlers to Evade Website Blocks
The post Cloudflare Accuses Perplexity AI of Using Stealth Crawlers to Evade Website Blocks appeared on BitcoinEthereumNews.com.
In brief Cloudflare accused Perplexity AI of using “stealth crawlers” to evade bans, rotating IP addresses and mimicking regular browsers to access blocked websites. Cloudflare delisted Perplexity from its verified bots program and deployed new technical defenses to catch and block deceptive scraping. Perplexity denies the claims, calling Cloudflare’s evidence a “sales pitch” and disputing that any banned content was accessed. Perplexity’s crawlers kept accessing content from tens of thousands of websites even after those sites explicitly blocked them, according to internet infrastructure provider Cloudflare. Cloudflare said Monday it had delisted Perplexity from its verified bot program and implemented blocks against what it characterized as deceptive scraping practices. San Francisco-based Perplexity was founded in 2022 by Aravind Srinivas (CEO, former OpenAI researcher), Denis Yarats (former Facebook AI), Johnny Ho, and Andy Konwinski (co‑founders of Databricks). The company has received funding from investors including Elad Gil, Nat Friedman (former GitHub CEO), and Nvidia, among others, and was valued at $18 billion after raising $100 million last month. The recent conflict erupted after Cloudflare customers complained that Perplexity was still scraping their sites despite implementing both robots.txt directives and specific firewall rules to block the AI company’s declared crawlers. Cloudflare engineers Gabriel Corral, Vaibhav Singhal, Brian Mitchell, and Reid Tatoris confirmed in tests that “Perplexity’s crawlers were in fact being blocked on the specific pages in question.” To test Perplexity’s behavior, Cloudflare created multiple newly purchased domains with restrictive robots.txt files that prohibited all automated access. “We conducted an experiment by querying Perplexity AI with questions about these domains, and discovered Perplexity was still providing detailed information regarding the exact content hosted on each of these restricted domains.” What happened next surprised them. Rather than respecting the blocks, Perplexity appeared to switch tactics. “We observed that Perplexity uses not only their…
Filed under: News - @ August 5, 2025 1:18 am