I have a web application that is built using Machine Learning and scrapes and collects information from different websites with captchas etc. and cross checks with other information that is found online.
It works well.
However, at the moment it takes too long. At present one search using the application can take 10 days to complete returnin 200 results for example.
I'm looking for someone to reduce this time, perhaps using cross-threading, perhaps using load-balancing, but overall making the system much more efficient - most searches should only tkae a few hours, rather than 10 days.
The candidate should be a programmer familiar with python, scrapy, flask as well as various server technologies.