Scraping using Pi agent + Cursor SDK + Codex /goal
I think, I have found the perfect web scraping setup for myself. Currently, I am using the Cursor SDK via pi-cursor-sdk and Codex's /goal via pi-codex-goal inside the Pi agent using Cursor's Composer 2.5 model and doing great so far.
As you can see in the above screenshot, the scraper is running for over 2 days, continuously. It stopped a few times due to the model downtime but then /goal resume continued right from where it stopped.
Basically, I am using the Composer 2.5 model from Cursor for this. The model writes the scraping script in Python and then keeps the process running, if it fails for any reason, the model makes necessary changes in the script and keep it running. For the last few days, as you see in the screenshot below, you can see the sudden spike in my usage. I have used more than 1.5 billion tokens as I'm running multiple scrapers at the same time, continuously.
I'm on the Pro+ plan ($60/mo) on Cursor and I think it's the best value for money AI subscription right now.
Webmentions