I am looking for 1 or 2 people to develop two web crawlers. One will be crawling of the web and does not require any data extraction. I would like this crawler must be able to cycle through all sites once a week. The second will be crawling each page on specific domains and will be required to extract data and search for changes. I want this to run every day.
I'm really looking for partners and do not want to work with a company. If your a college or high school student and need a summer project than this is perfect for you, it will become huge. Also, you must not be offended if you are working with a college or high school student. In your PMB message to me, please confirm that you are OK with this.
This is a search engine project, but I am not looking to beat Google. Google is the leader, and will probably stay the leader for its type of search. However, this concept is a completely new and revolutionary way to search. Whether a user choose to use the search engine we develop or Google will depend on the content they are seeking.
Please use the PMB for more details.