I have installed phpdig, a php based web crawler and search engine on my server. There are a couple of sites I'd like to index, but I only want to index certain pages on them.
Site #1 - I only want to index pages that have the URL structure [url removed, login to view]
Site #2 - The pages I want to index from this site are behind a query wall, so you have to perform a query to access the pages. The pages are all structures [url removed, login to view]
If you think you know how to get phpdig to do this, I'd like you to show me how it's done, and show me exactly which code was modified.
This is part of a bigger project so it may lead to more work. If you have web crawler experience, that is a plus.