Thankyou for the detailed instructions.
I'll step through a couple things here.
Firstly to find the blog, the scraper must crawl all links on the homepage(searching for blog isn't always accurate enough). We'll call this an average of 5 requests.
Say an average blog has 300 posts. Counting pagination, that is 330 requests.
Totaling everything, we come to 14M requests for all websites.
This will take time, unless you have further constraints that would prevent this amount.
I'll be happy to talk with you about a final solution.
My current bid is to build the scraper only.
Thanks
- Custom scripts, solutions, frameworks
- Over 50 bots completed(from freelancer and other freelancing sites)
- Hosting may be available
- HTTP(S) 1.0, 1.1