Đã hoàn thành

Sync a local file store by screen scraping a website

We are a data integration company based in Melbourne Australia and we are interested in running our first Freelancer task.

The developer should be familiar with making Http requests from C#.Net

1. Overview

A C#.Net application is required to run on MS Azure. It can run as a service or some other style of application, and it will call a set of URLs and screen-scrape the html responses in order to sync the data files on the website with a local store. Note that the client has full legal access to those folders and files, and that this is simply automating a manual process. The application will be called by a scheduled task every n hours, and the objective is to keep the local file/folder store in sync with the remote store.

2. Authorisation

The application should navigate to the URL, which will be provided in the detailed specifications. Normally when you manually navigate to that page you would be prompted for a username/password. Instead, these credentials will be read from a config file and passed via the http request header (or similar). You will be assigned these credentials as part of the detailed specification.

3. Screen scraping

The URL above points to a tree structure. Each folder in that tree should be opened recursively. The tree nodes will contain either further tree nodes or a list of files. This tree structure of folders and files should be replicated on the local machine, however only csv files need to be downloaded. If new folders or csv files are encountered, or existing ones are updated, these should be downloaded. You do not need to worry about deletions.

4. Technical constraints

As mentioned, the service should run on MS Azure. It can be run as a scheduled task, meaning that you do not need to build timers/scheduling into the system (this will be done by the scheduler). The application should be written in C# .Net.

An SQL database will be available if required to store the tracking of files and folders, or this could be done by simply using a JSON file or XML file stored on the system.

To be completed in the next couple of weeks.

Kĩ năng: .NET, ASP.NET, Azure, Lập trình C#, Microsoft SQL Server

Xem nhiều hơn: screen scraping opensource mls, paid screen scraping, screen scraping data mining, getting green screen video website, screen scraping visual basic, flash stream local file rtmp, extract table html file store, aspnet file store, perl screen scraping, j2me local file blackberry store, curl screen scraping, screen scraping job, screen scraping flights, screen scraping bot, copy content website screen scraping, website screen scraping webcrawler, read text file store local variable, finding word static website screen scraping, sync local copy website

Về Bên Thuê:
( 1 Nhận xét ) Brighton, Australia

ID dự án: #19888227

Được trao cho:


Hi, I can help you to create an C#.net application that will sync files on the website. I've read the project description carefully that you want run this application in MS Azure and I can come to know that you want to Thêm

$1176 AUD trong 10 ngày
(49 Đánh Giá)

43 freelancer đang chào giá trung bình $1361 cho công việc này


I am an expert with httpwebrequest , CUrl and selenium browser automation you can see many jobs of similar kind listed on my profile. I can easily get this done. Thank you

$750 AUD trong 5 ngày
(170 Nhận xét)

Hello Luke, We have gone through your requirements. Please find below details: Our understanding: We need to develop a solution to sync data files between remote and local store. Remote store provides files i Thêm

$1500 AUD trong 20 ngày
(51 Nhận xét)

Hi This is Wang from China. I checked the project description in detail and strongly feel confident. The project description is in great details. I have rich experience in C# programming both for service or schedul Thêm

$1764 AUD trong 20 ngày
(91 Nhận xét)

Hi, I have been invited by your project RECRUITER "Emily M." and I have experience auto login from code and do automation. Yes, I can create application console/web application which can be schedule on azure as p Thêm

$1500 AUD trong 15 ngày
(127 Nhận xét)

Hi, Our developer is familiar with c# , .net so we can help you with this project as per your requirement. We have also experience with azure so we can easily help you with this project. Here is the technology Thêm

$1411 AUD trong 8 ngày
(26 Nhận xét)

*** Expertise in Microsoft Technologies( ASP.NET,C#, MVC, MSSQL, having 10+ years experience)*** Hello Dear, I have read your requirements for Sync of local file store by screen scraping a website I have 10 ye Thêm

$1250 AUD trong 20 ngày
(16 Nhận xét)

Hi, I have 10 years experience in C#, ASP.NET/ MVC with bootstrap, Jquery/Ajax and for services using WCF/Rest WCF/Rest Web API and SQL Server for the back-end. Involve in designing databases, Indexing Strategie Thêm

$1411 AUD trong 20 ngày
(50 Nhận xét)

Greeting! My Name is Noor All Safaet and expert in Web Development. I would love to have the opportunity to discuss with you about your project. I have 5 years experience as a developer. Technical Summary 1. .NET, Thêm

$1588 AUD trong 20 ngày
(141 Nhận xét)

Hello - My name is Arvin. I’ve read your brief description and we can build the scraper you are looking for. We have extensive experience building highly complex scrapers. We have built scrapers which are fully autom Thêm

$2000 AUD trong 30 ngày
(5 Nhận xét)

Hi, I have prior experience of working over several scrapping application and automation applications using [login to view URL] Is it mandatory to use httpwebrequest as selenium will eas Thêm

$1400 AUD trong 9 ngày
(65 Nhận xét)

Hi, there i have done scrapping almost on Half of Worldwide web including eCommerce giants (Amazon, eBay, craigslist) News Feed, Social media websites, API's. I develop my own tools based on client requirements with Thêm

$1125 AUD trong 10 ngày
(30 Nhận xét)

Hello, Dear How are you? My name is Sundong. I have read your project description and am ready for discussing with you about project. I have experienced in developing .NET, ASP.NET, Azure, C# Programming, Microsoft Thêm

$1250 AUD trong 20 ngày
(13 Nhận xét)

Hi there! I'm an experienced .net software developer (9+ years). I already have developed multiple webscraping software with similar behavior!I believe that I have the required skills for this project

$1176 AUD trong 14 ngày
(17 Nhận xét)

Hi there, I am interested in the project. I have reviewed the requirement and I am sure I can deliver you the exact solution you are looking for. The solution can be developed as a azure webjob to do the task. Thank y Thêm

$1125 AUD trong 10 ngày
(67 Nhận xét)

Hi There: Hope this note finds you in great spirits. We are team of IT professionals mentored by very experienced leadership. We offer solutions on WEB and Mobile technologies. Please visit our company site [login to view URL] Thêm

$2000 AUD trong 30 ngày
(2 Nhận xét)

Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl Thêm

$1500 AUD trong 14 ngày
(11 Nhận xét)

Hi, I have gone through your requirements and I can definitely help you with the required development of the application in the Azure. I have more than 10 years of experience in .NET Development with hands on experie Thêm

$1200 AUD trong 14 ngày
(22 Nhận xét)

Hi, We have 8+ years of experience in c# and .net tech stack, We have screen scraped or automated more than 500 websites till now. We can handle ip blocking, geo tracking etc. We have also scrapped a approx 2000+ Thêm

$1323 AUD trong 20 ngày
(16 Nhận xét)

Hello, Luke- I'm interested in your project. I have many years of experience with automating data-oriented processes, including many business-critical processes. I have broad and deep knowledge of web-scraping methods, Thêm

$900 AUD trong 5 ngày
(12 Nhận xét)

HI Luke, I'm Isaac from Cloud Devs, We can write a quick console app to do this. Its ideal to have a SQL db to store the changes rather than a xml file. BTW html file attachment is missing the images. We Thêm

$4444 AUD trong 20 ngày
(2 Nhận xét)