I am looking to develop a web spider which can visit multiple real estate sites every two days automatically via cron and extract data formating it into a website. Once data is in a the site which the design of will be provided we will need the ability to run a search query to sort and filter results which will need to be able to be emailed with custom header and footer.
There will be a need not to send all data via email so we will need to create a filter to disallow certain fields prior to sending an email to clients etc.
The spider must be built so it is hard to detect or disallow with a robot txt. Insight on this is welcome.
The results will need to be displayed on a web platform which will require a log in and the ability to add users over time. Also when sending results to clients we will need the ability to add a customized message and email signature.
Looking to develop this spider asap, please submit your bids. Only experienced developers are welcome to bid on this project we need a firm development time line and completion date. So please ask questions if need be.