I need a spider/crawler that crawls sites, and counts links to sites.
What it should do is crawl an entire site and index every external link on the website- it should not index internal links.
This spider is to count links- because links are a very important part of SEO. I will give the spider 1 link to go to and then it should keep following to different sites from there. It is a forum with alot of links to different websites. So if you can build mulyiple spiders off of the origional one that would be so much better. It should not index the same link twice. Please post to PMB to discuss the rest of the project. Please also give the the base cost of this and let me know your detailed plan on this project. I want you to post first that you are interested and post all your questions, and the second post should have a detailed project report on it analyzing how you will do this and how it will work etc.