I am looking to develop a web spider which can visit multiple sites every two days automatically via cron and extract data formating it into a web template. Once data is in a template which the design of will be provided we will need the ability to run a search query to sort and filter results which will need to be able to be emailed with custom header and footer.
There will be a need not to send all data via email so we will need to create a filter to disallow certain fields prior to sending an email to clients etc.
The spider must be built so it is hard to track or disallow with a robot txt. Insight on this is welcomed.
The results will need to be displayed on a web platform which will require a log in and the ability to add users over time. Also when sending results to clients we will need the ability to add a customized message and email signature.
Looking to develop this spider asap, please submit your bids. Only experienced developers are welcome to bid on this project we need a firm development timeline and completion date with no overuns. So please ak questions if need be.