Hi we need the following project to get under way as soon as possible.
There is a database online that we need for our own analysis.
The public database is free to everyone but the site operators disallow frequent pings on their server so that the DB can not be just downloaded. There may be some IP logging also and rules made by the site operators.
There are 2.2 million records that we want to get but we are in no hurry. If the records were pulled at one every 60 seconds it would take 3 months to do. This is fine for our purpose.
I will need some indicator of what level the pull of the data is at. Just a simple XML or file with %complete on it so I can route it through our management screen while the pull is working away. We will need to monitor if we have been blocked and so on.
The candidate will provide steps on their strategy to complete this task in a concise prfessional manner and possibly present their previous work in this area.
The database to store the data is undecided and we actually often prefer a simple text file to work with in statistics packages.
Please contact me via PMB for the site to be pulled after your previous work and strategy are told.