I am looking for a web scraper for 4 wordress websites.
How it works:
There is a list of about 200 one word phrases each time it runs:
The scraper searches each website (one after the other) and sees if there is a result.
If there is a result, it will include a text with a hyperlink.
Test if hyperlink is working.
Scrape the hyperlink and store it.
If hyperlink does not work, move to the next website.
If a working hyperlink was found, store the hyperlink into a list.
If no results or no working hyperlinks found, move to "Not found" list of phrases.
Display all stored hyperlinks.
Display all Not found phrases.
The sites may have Cloudflare enabled, the scraper must use a Cloudflare bypass ONLY IF CLOUDFLARE IS ENABLED!.
Can be in Python or any other stable language.
Any replies without "I have read and fully understood what needs to be done" will be ignored.