I have installed phpdig, a php based web crawler and search engine on my server. There are a couple of sites I'd like to index, but I only want to index certain pages on them.
Site #1 - I only want to index pages that have the URL structure [url removed, login to view] All these pages can be accessed through links on the site.
Site #2 - The pages I want to index from this site are all structured as such: [url removed, login to view] They have a script that shows all the most recent pages, so the crawler can probably use that to index the pages.
If you think you know how to get phpdig to do this, I'd like you to show me how it's done, and show me exactly which code was modified.
This is part of a bigger project so it may lead to more work. If you have web crawler experience, that is a plus.