Đã hoàn thành

Need an expert developer to develop a file scraper to scrape firmware files and their info from various vendors websites

A python based CLI script that can download all product’s firmware (including all versions) from web pages for a given list of predefined vendors and store the information (meta data) in SQLite [login to view URL] mandatory metadata fields include ( Manufacturer, Model, Version, Type, Name, Release Date(if available), Download link, ( calculated Sha2 hash of the file)i.e. ( Cisco, Video Surveillance 6030 IP Camera, 2.7.0, IP Camera, [login to view URL], 21/08/2015, "link" ) There is a non-mandatory binary field which indicates if the device is discontinued or not depending on the fact that vendor mention that on the website or not. The firmware files itself will be stored in the file system and will be referenced by index ID in SQLite.

The arguments to the script should be a list of comma separated vendor names or the location of a text file containing the vendor name.

There are no GUI components in the server where the script will run hence headless mode for browser should be used by the script

Solution Scope

1. Script will be written per vendor. This is required because each vendor website will have its own implementation of the firmware download page. However, efforts will be put to identify and implement reusable components, if any.

2. The script will only download new firmware that have been added by the vendor. Hence first execution of script will download all the firmware available but the subsequent runs will only download new ones which will get added. This will be achieved by analysing data available in SQLite and skipping the files that are already been downloaded and processed.

3. Each vendor, that will be provided, will be analysed manually to identify the following, which will be required to develop the script:

a. URL for the firmware download page

b. Credential Requirements (Simple Signups, Specific Signups, No Signups)

c. Any Captcha on the page

d. Any honeypot traps

4. If there are credential required to download the firmware and the credentials are simple ones where a simple sign up is required, the signup will be done manually as part of the manual analysis using a gmail account dedicated for this work.

5. Script will try to imitate human like behaviour (to a limit) while scraping the web page as well as uses Tor, so that if the vendor site has scraper/crawler detection logic implemented, it can be skipped. This will be achieved by adding random delays, random view time, avoiding honeypot traps through manual analysis

Solution Brief

A Python Selenium and SQLite based solution will be developed which will have the following features/components:

1. File Management Module: Responsible for storing and managing the downloaded files and meta data. Firmware and installer files will be stored on the filesystem which will have a structured folder hierarchy. Meta data of the files will be stored in SQLite. Meta Data will refer to the stored files through paths on the file system and file index/name.

2. Vendor Scrappers: Python Selenium based scrapper will be written for each of the vendor, responsible for downloading the files and grabbing the meta data from the vendor’s site. This will make use of the file management module to store the file and meta data to SQLite.

3. Configuration File: All the configurations for the framework (including vendor specific like credentials, url etc) will be stored in a json file which can be easily modified manually.

4. Execution Script: The configuration file can be setup to represent the polling interval for each of the vendor scraper and when the execution script is run it will go and schedule each of the vendor scripts individually according the polling interval defined in the config.

Deliverable:

1) Python Source Code including the comments in the code explaining each function & its details. We should be able to give any required input as an argument and execute it as one line command in the Linux terminal.

2) Dependencies

3) Manual to install, configure and use the scraper

Kĩ năng: Python, Selenium, Web Scraping, SQLite, Linux

Xem nhiều hơn: freelance website, freelance jobs from home, freelancer company, feelancer, freelance online, freelancer login india, freelancer jobs, online freelance jobs, aspnet word file scraper, need php developer team, need web developer, need php developer delhi, need game developer, need upload large audio file, company asia need php developer offsite, need someone excel attached file, need lisp developer, need dotnetnuke developer, need full time sharepoint expert developer, i need a developer who can help me develop my app

Về Bên Thuê:
( 2 nhận xét ) Brussels, Belgium

ID dự án: #23139167

Được trao cho:

pandread1

Hi, Im a Python Developer from Athens Greece, experienced in Web Scraping and all the related libraries. Similar projects have been completed in the past and I am able to provide evidence in chat. Interested in your Thêm

€14 EUR / giờ
(15 Đánh Giá)
5.1

16 freelancer đang chào giá trung bình €16/giờ cho công việc này

zekovicm

Hi there,I am Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this project ! I can start immediately and finish it within the Thêm

€20 EUR / giờ
(123 Nhận xét)
7.4
chirgeo

Hi. sure we can create such a script. Are all this data available for all vendors? Do you provide links to all vendors or we need to get them> Please provide some vendors and links and I will make some tests.

€20 EUR / giờ
(121 Nhận xét)
7.4
adeelpirzada

Hi, Allow Me to Introduce Myself i have done scrapping almost on Half of Worldwide web including eCommerce giants (Amazon, eBay, craigslist) News Feed, Social media websites, API's. I develop my own scrapers an Thêm

€12 EUR / giờ
(36 Nhận xét)
6.5
tangramua

Dear Sir,   Our team has a huge experience in Python, Linux, Web Scraping, SQLite, Selenium as a result we can successfully complete this project. Having the required skills, we will be glad to help you.   We have 20 y Thêm

€15 EUR / giờ
(54 Nhận xét)
6.7
schoudhary1553

Hello, I can help you with your project - Need an expert developer to develop a file scraper to scrape firmware files and their info from various vendors websites I have gone through your job posting and become very Thêm

€18 EUR / giờ
(45 Nhận xét)
6.4
Seeniea

Hello Dear Interesting job. Please send me message for discussing. Thanks Best Regards

€25 EUR / giờ
(6 Nhận xét)
5.5
abhilashtv

Hi, ➲ 10+ years of full-time experience in Python / Django with 50,000+ Upwork hours billed and 50+ successful Python projects ➲ Upwork Top 10 Certification for Python and Django ➲ Guaranteed Results Policy: Pay only i Thêm

€13 EUR / giờ
(15 Nhận xét)
5.6
mithusamsukha

Yes, I have read your read job description. As I will help you in develop a python based CLI script that can download all product’s firmware including all versions from web pages. As I am a professional python expert Thêm

€12 EUR / giờ
(3 Nhận xét)
3.5
Exiver19

**Ensure Affordable Price and Quality Work** Dear Client, Response: 1) Python Source Code including the comments in the code explaining each function & its details. We should be able to give any required input as an ar Thêm

€13 EUR / giờ
(1 Nhận xét)
2.2
jzf2050

Dear Client, I have read your project info carefully, and I considered that I am qualified for this project. I have more than 10 years working experiences of software related, and skilled in Python programming, famil Thêm

€14 EUR / giờ
(2 Nhận xét)
1.6
rnglabaza

Hi, I'd be happy to help you with this. I have experience in python, scrapping and selenium. I'm also experienced in networking and information security, so i know possible security features, workarounds and ability Thêm

€15 EUR / giờ
(1 Nhận xét)
1.0
Webllisto003

Hello, I have carefully checked your requirements. I can help you with this project. I am a full-stack developer working at a large-scale. I am a specialist developer having much experience and development work and I Thêm

€15 EUR / giờ
(0 Nhận xét)
0.0
Karishma576

Hello, I have thoroughly read the given following features/components of script that you want to develop and I can develop a file scraper to scrape firmware files and their info from various vendors websites as per gi Thêm

€15 EUR / giờ
(0 Nhận xét)
0.0
umarsrao786

I have a lot of experience in creating and deploying python/selenium scripts (that run on a headless browser) on local machines or aws servers.

€30 EUR / giờ
(0 Nhận xét)
0.0
Letsoptimize

Hi, I have more than 4 years of experience with scraping data from sites using python. I have built many automated scraper which use proxy servers to bypass web blocks. Also i can implement multiprocessing to scrape da Thêm

€12 EUR / giờ
(0 Nhận xét)
0.0