Photo of DeepakNess DeepakNess

I'm NOT Blocking AI Crawlers, For Now

During the middle of last year, Cloudflare introduced a new button called "Block AI Bots", designed to block all AI crawlers. Itā€™s available even on their free-tier plan, and you can enable it with just one click.

Cloudflare Block AI Button

I love it! But I'm not using it.

While I think itā€™s a great feature, Iā€™ve decided not to enable it on my websitesā€”and I have good reasons. AI assistants like ChatGPT are already driving a significant amount of traffic to my projects. For example, the website shown in the Google Analytics screenshot below receives around 450 visits per month from ChatGPT alone, and Iā€™m happy with that.

The future of AI and content

Looking ahead, I believe avoiding new technologies will only hurt us rather than help us. AI is one such technology that we shouldnā€™t resistā€”we should embrace and adapt to it.

However, if you create content that takes a lot of effort and you want to keep it exclusive to your audience, then blocking AI bots might be the right choice for you.

Thereā€™s also an ethical concern that if AI companies are "stealing" copyrighted content without permission, thatā€™s a serious issue. We need a proper system where authors receive due credit if AI models use their work.

For now, Iā€™m allowing AI crawlers because they benefit me. But if I find out in the future that theyā€™re using my content without proper attribution, I wonā€™t hesitate to block them.

Comment via email