Anubis: A Proof‑of‑Work‑Based Defense Against AI‑Driven Web Scraping

In the current era of AI‑powered web crawling, many sites face relentless traffic from automated agents that strain server resources and compromise user experience. To mitigate this threat, the Anubis framework has been deployed by site administrators as a pragmatic, PoW‑based gatekeeper. At its core, Anubis follows the principles of the Hashcash protocol—a PoW system originally devised to curb email spam. When a client requests a page, Anubis generates a cryptographic puzzle that requires a modest amount of computational effort before the request can be processed. Legitimate users, whose browsers can solve the puzzle quickly, experience no perceptible delay. In contrast, mass scrapers attempting to fetch thousands of pages concurrently must perform the same calculations, multiplying their load and thereby increasing the cost per request. This PoW approach is intentionally non‑intrusive; the additional computational overhead is negligible for a single user but scales linearly with the number of concurrent scrapers, making large‑scale scraping economically untenable. Importantly, Anubis is designed as a temporary placeholder. Its primary goal is to buy time for developers to implement more granular anti‑bot techniques, such as fingerprinting based on subtle differences in font rendering and other headless browser artifacts. By reducing the frequency with which legitimate users encounter the PoW challenge, Anubis preserves a smooth user experience while tightening security. A practical consideration for sites utilizing Anubis is the requirement for modern JavaScript features. Clients that block or polyfill these features—most notably browser‑enhancing extensions like JShelter—will fail the PoW verification. Site operators should therefore advise users prone to encountering the challenge to disable such extensions for the protected domain. Additionally, because Anubis relies on JavaScript‑based rendering for its PoW logic, sites should ensure that critical content is still accessible via graceful degradation paths for older browsers or environments where JavaScript execution is restricted. In summary, Anubis presents a balanced blend of immediate, low‑cost deterrence coupled with a strategic roadmap toward more sophisticated bot detection. By employing a proven PoW model and complementing it with advanced fingerprinting, organizations can protect their resources from AI‑driven scraping while maintaining optimal service quality for genuine users.