Identifying AI Web Scrapers Using Canary Tokens

From pre-training to query-time augmentation, web-scraped data helps to improve the quality and contextual relevancy of content generated by AI chatbots such as ChatGPT and Claude. However, large-scale web scraping to feed such chatbots can affect site stability and raise legal, privacy, or ethics concerns. If website owners wish to limit AI-chatbot-related web scraping on their site, due to these or other concerns, they may turn to scraper access control mechanisms like the Robots Exclusion Protocol. To be most effective, such mechanisms require site owners to first identify the scrapers that they wish to restrict (e.g., via User-Agent strings). Existing mechanisms to identify AI-chatbot-related scrapers rely on voluntary disclosure by companies, one-off research experiments, or crowd-sourced reports -none of which are reliable or scalable.

This paper proposes a novel technique for accurately and automatically inferring AI-chatbot-related scrapers. We host dynamic websites that serve unique canary tokens to each visiting scraper, then prompt AI chatbots for information about our sites. If a chatbot consistently generates outputs containing tokens unique to a scraper, it provides evidence of exposure to that scraper. Via experiments across 18 production chatbots, we demonstrate that our approach can reliably identify which scrapers feed which chatbot, including several that are not publicly known or disclosed by the companies. Our approach provides a promising avenue for unprivileged third parties to infer which scrapers serve data to which chatbots, enabling better control over and transparency into unwanted scraping.