I need a web crawler program to get the information for a public online ecommerce platform.
- select category or sub-categories of ecommerce platform (continuous update, allow add or delete)
- input brands to be ignored (continuous update, allow add or delete)
- run the program 7x24
- go through each listing one by one (by program in backend)
- record down the sales quantity (to be found out by program, as online platform will not report the data)
The most time consuming method (manual) is to click individual listing every day, check the inventory available for sale (by buying maximum quantity in shopping cart if no inventory data is available), and compare with previous day.
- record all sales quantity for each listing (if quantity is different)
- output and update to an excel file (any database e.g. MYSQL format is OK, but still need to convert back to excel file as output file) 1 by 1, and save the complete record of each category (after complete the category) in Google cloud.
- product name (with http link back to product listing), quantity sold yesterday, quantity sold last 7 days, quantity sold last 30 days, total no. of visit
- unit price, no. of sellers, brand name, seller status, image http link
- sort by highest quantity sold, brand name, unit price.
The program shall re-start automatically if there is the server is down, loss of internet connection etc.
Getting the inventory of a listing is the most time consuming.
How to update and store the data is most challenging as the listing position in web site is changing every second. The output excel file has to store 31 days data.
Online ecommerce platform does not have any support nor API.
The program shall be able to run in Windows 10 (English). It is recommended the browser to use internet explorer as the computer has other program running with Google Chrome. if developer can ensure no conflict, it is OK to use Google chrome (e.g. incognito mode).
Interested parties please send a detail quotation - time and cost. Debugging is the most time consuming.
Ecommerce platform may blocking enormous machine visit the site. Developer shall handle how to deal with such IP blocking etc.
Deliverable shall be an executable program, which we can download from Google cloud. No teamviewer etc. installation.
After completion of the program (and debug), developer shall run the program by 7 days (7x24) and send out the excel output file for checking. If positive, developer shall upload the executable file to Google cloud for us to install the program in PC to test for another 7 days. Next, developer shall install and set up the program in Windows VPS for continuous running.
30 days warranty shall be counted after running in VPS.
Automatic or chat bot response will not be considered.
Interested party please contact for the name of public ecommerce site.
Applicant shall prove he can do the program by providing a sample excel file extracted from captioned web site. Manual data capture for 5 items is sufficient.
Upon completion of the project, applicant will be awarded for 2nd project with different ecommerce site (same requirement).
5 freelancer đang chào giá trung bình $175 cho công việc này
Hi, sir i have read your project detail and find it very interesting , i have experience with web scraping and crawling . We can discuss details in the chat .