General
Opting out: How to stop AI companies from using your online content to train their models
A US company created a button for website owners to block AI crawlers. Here’s a look at how to block AI from websites and social media.
We have ad block and now there’s an artificial intelligence (AI) block.
US cybersecurity company Cloudflare has created a button for website customers to block their data from being used by AI crawlers: Internet bots that roam the web to collect training data.
“We helped people protect against the scraping of their websites by bots (…) so I really think AI is the new iteration of content owners wanting to control how their content is used,” John Graham-Cumming, the company’s chief technical officer, told Euronews Next in an interview.
When a connection comes to a website hosted by Cloudflare, they are able to see who is requesting to see the website, including any AI crawlers that identify themselves. The blocker will respond by showing them an error.
Some AI bots pretend to be human users when accessing the website, so Cloudflare built a machine learning model that scores how likely a website request is coming from a human or robot user, Graham-Cumming said.
The CTO couldn’t say which clients are using the new button but said it’s been “very popular,” with a wide variety of small and large companies.
Blocking AI crawlers in general is becoming more popular, according to one study from the Data Provenance Initiative, a group of independent AI researchers.
Their recent analysis of over 14,000 web domains found that five per cent of all data assembled into the Internet’s public databases of C4, RefinedWeb, and Dolma is now restricted. But researchers note this number goes up to 25 per cent when looking at the highest quality sources.
Ways of blocking AI crawlers
There are ways to manually block AI crawlers from accessing your content.
Raptive, a US company advocating for creators, wrote in a
