I am scraping a large data set from the web. To make things go faster, I need to recruit a few people willing to run the script on their computer.
The task is INCREDIBLY easy. It will require you to have Ruby and PHP installed, but after that all you have to do it run the scripts for various inputs and then.... wait. The downloads are throttled at 1 per second, and i'm looking to downloading about 500000 entries. So it will take patience. But it's really easy. I need distribute this task because if I don't distribute this load it will take me much longer downloading from a single point.
1) run the script for the requested input ranges (I will provide these). If the script fails because the connection breaks, you will have to restart the script from the file where the connection last worked (easy).
2) You will also break up the downloaded html and jpg files into ranges of 10000, so the directories are traversable without windows system explorer choking on the high file count.
3) Finally you will zip the files up and send to me via ftp.
You have ~2 weeks to complete the task for 500000 entries.