A simple web crawler
- Collects the links from the websites
- Save the requested information in a mySQL database
- It must be possible to run mutiple crawlers
- The program must use threads for speed!
- Respect the robots exclusion standard
- Crawling depth
- Number of threads
- User agent of the crawler
- List of domains that will be crawled
Information to collect:
- Website Title:
- Meta Description:
- Meta Keywords:
15 freelancer đang chào giá trung bình $206 cho công việc này
I have been creating such bots from a long time now. I assure you quality work and 100% satisfaction guaranteed. I've read your requirements and easily write such a crawler in java. Please check your PMB.
"SCRAPER AND CRAWLER HERE". We are new to freelancer.com so don't have reviews here. But you can check our reviews in elance, odesk and scriptlance. Please check your PMB for details.