Cloudflare /crawl endpoint is just 'business'
Cloudflare has multiple tools like scrape shield, bot protection, and even CAPTCHA that made it extremely difficult for bots to scrape the sites using these. And now Cloudflare has launched a new /crawl endpoint that helps you crawl the entire website with just a single API call. The endpoint is offered as a new tool in their existing browser rendering service.
And it's a business, somewhat questionable, but still a business tactic commonly used. They, first, created the demand by letting people use their service to protect their websites against scrapers, and now they themselves started offering another service that makes scraping easy.
Webmentions