I need a custom web crawler/scraper for a WordPress information site.
How I want it to work is thus:
- It needs to be a WordPress plugin.
- There will be about 60 different topics. Each topic will have 10 or so webpages which need to be scraped and made into WordPress posts. E.g. one topic may be â??universityâ?? and have a list of ten pages under the university category.
- The data on these pages will change every few days so it will need to scrape to check for any new updates regularly and make posts of these.
- Each webpage that needs to be scraped is literally that â?? a URL to a single webpage. No RSS feeds.
- Each webpage will have URLs on them which will link to more detail. It is these URLs which need to be converted into WordPress posts. One post for each URL link. The post created will contain the information in the URL link and also a copy of the link too so people can follow it for more information.
- The new post will then need to be tagged according to which topic it came under. E.g. if this website was under the â??universityâ?? topic list, the posts will be tagged with â??universityâ??.
- The plugin should have a way of alerting a site administrator to any new posts that have been created so that they can be checked.
- It should be possible and straightforward for the site admin to add new websites to scrape and new topics relatively easily.
- It should allow another WordPress plugin to work where posts will expire and be taken offline after a certain date.
All the crawled data is freely available to the public and there are no copyright concerns. The posts created from the scraped data will link back to the site the data came from.
It needs to abide by the usual Wordpress guidelines for plugins such as being API based with a core plugin, extensions as needed and use hooks, filters etc. The usual setup.
Expert data scraper here and have great WordPress experience too. Detailed proposal in the PMB. Thanks, Irfan - The Administrator removed this message for containing contact details which breaches our Terms of Service
Hello
It is our pleasure to work with you on your project. We are willing to provide you quality services,please look further for the expertise on our work.
Kind Regards