I have a python code to scrape a website and analyze it. There is an option to top up the incremental changes if needed. Right now it uses a single process to fetch the data. I have used selenium and bs4 for it. I need an expert to optimize it, structure the code properly and use multithreading / parallel processing to fetch the data in lesser time. The website throws error if there are multiple requests in a very less time, so there is a need to bypass that as well. Currently, the bottleneck is that we can send a single request every second to that website. Website is [login to view URL]
After the optimization, we need you to host it on heroku or any optimal cloud platform and program it to run at times according to certain data points that are to be fetched from another part of the website. Some part of the code has to be parameterized so that we can change some parameters from the heroku terminal directly. The fetched data has to be saved on a set folder in Microsoft Onedrive.
6 freelancer chào giá trung bình₹8833 cho công việc này
I am a Web developer and web scrapping expert. I had 4 year experienced in this field. I can do your project in cheap price with high quality. Please response me as soon as possible.