Đã hoàn thành

161398 Run my html scrape for me

I am scraping a large data set from the web. To make things go faster, I need to recruit a few people willing to run the script on their computer.

The task is INCREDIBLY easy. It will require you to have Ruby and PHP installed, but after that all you have to do it run the scripts for various inputs and then.... wait. The downloads are throttled at 1 per second, and i'm looking to downloading about 500000 entries. So it will take patience. But it's really easy. I need distribute this task because if I don't distribute this load it will take me much longer downloading from a single point.

1) run the script for the requested input ranges (I will provide these). If the script fails because the connection breaks, you will have to restart the script from the file where the connection last worked (easy).

2) You will also break up the downloaded html and jpg files into ranges of 10000, so the directories are traversable without windows system explorer choking on the high file count.

3) Finally you will zip the files up and send to me via ftp.

You have ~2 weeks to complete the task for 500000 entries.

Kĩ năng: Bất kì công việc gì, Python

Xem nhiều hơn: web scraping python 3, things for people to do, html web scripts, go ruby, web scraping ruby, scraping python, scrape web, scrape python, scrape html, run, python task, python scraping, python ftp, my entries, php load html, web script python, run faster, python html script, python scrape web, python send data, html load php, 10000 downloads, run windows, file explorer python, python php scrape

Về Bên Thuê:
( 5 nhận xét ) Cambridge, United States

ID dự án: #1907587

Được trao cho:


I can do it.

$20 USD trong 14 ngày
(0 Đánh Giá)