Photo source: GeekWire
On Monday, Cloudflare announced plans to introduce a marketplace within the next year, which will allow website owners to charge AI model providers for access to scrape their content. This initiative is part of a broader strategy by CEO Matthew Prince to give publishers more control over how and when AI bots utilise their sites.
“If you don’t compensate creators one way or another, then they stop creating, and that’s the bit which has to get solved,” Prince stated in an interview.
As a preliminary step, Cloudflare launched free tools called AI Audit, providing website owners with analytics on AI model crawling activity. This feature allows users to block unwanted AI bots with a single click, giving them the option to restrict all web scrapers or permit certain ones based on agreements or perceived benefits.
Cloudflare’s initiative addresses a big challenge for smaller publishers—survival in an era where users may prefer platforms like ChatGPT over individual websites. Currently, many small sites are scraped for content without compensation, jeopardising their financial stability.
Earlier this summer, Perplexity faced accusations of scraping content from sites that requested not to be crawled. In response, Cloudflare introduced a feature allowing customers to block all AI bots instantly.
“That was out of frustration we were hearing, where people were feeling like their content was being stolen,” Prince noted.
Some website owners reported that excessive scraping resembled a DDoS attack, impacting server performance and increasing cloud costs. Cloudflare’s new tools will allow selective blocking of specific bots while permitting others.
Even large publishers with licensing deals with OpenAI often lack insight into how much their content is being scraped. The upcoming marketplace aims to empower smaller publishers to negotiate directly with AI providers for content access.
While details about the marketplace are still under wraps, Prince indicated that website owners could charge based on scraping frequency or request credit instead of monetary compensation. He believes this shift is essential for creating a sustainable ecosystem where content creators are fairly compensated.