I want to crawl following sites every day and collect realestate data and save it in our database. Every day complete refresh of each data set from each site.
The solution should be scalable so that more websites can be added at later date. Also should be able to remove website from crawling list any time without a need to any code change.
Set of data from all the website will be same.
[url removed, login to view] [url removed, login to view]
[url removed, login to view]
We do not want a trail error on this project. So if you have done similar projects, please share it. I will be disqualifying copy paste companies upfront.
Just dont say I can do it. Does not help. Tell me how in a brief summary.