Đã Đóng

Expired Domain Web Crawler Needed

I am looking for a software application that can crawl a list of websites for expired domains (any domains with a status of 404). The crawler should crawl a list of inputted names via a source text file. The source text file would look like this, 1 domain name per line:

[login to view URL]

[login to view URL]

[login to view URL]

[login to view URL]

Once software grabs the source domains, it should fully crawl each site and report to a csv file any expired domains it finds. The format for the csv should be:

domain where found, URL where found, expired domain

An example line would be:

[login to view URL],[login to view URL],[login to view URL]

The crawler should use just one thread, and for the user agent, should just be chrome if that is possible.

I plan to do a phase 2 later where the crawler will check an API to see if the expired domains are available for purchase, and also to add threads, proxies, and other features. This will all be phase 2 and 3.

Right now, I just want phase 1 completed where it takes an input of "[login to view URL]" and then for domains it finds in 404 status, it should output "[login to view URL]"

This needs to be desktop based running on Windows 10 Pro. It can run in Powershell or Python, and other languages are fine as well. For continued development, it would be beneficial to have it in Python, but any language you prefer is fine.

Please include these with your bid:

1. Project Price in Full to do phase 1

2. Time to complete, since I am really needing this soon

3. What language you will be using

4. Any relevant experience you have with crawlers

Kĩ năng: Web Scraping

Xem nhiều hơn: java web crawler image capture, web bloggers needed, web crawler cpu usage, write web crawler, expired domain checker, web crawler names, web sites needed, web designers needed, web crawler system requirement, web crawler demo gathering information function, java spring web crawler, web page sql database input, good sites web projects needed, simple input file php script web crawler, php web crawler input, expired domain crawler

Về Bên Thuê:
( 65 nhận xét ) Loveland, United States

ID dự án: #20031872

29 freelancer đang chào giá trung bình $169 cho công việc này

e3d

Hi, I can make this crawler for you, but I actually already providing my clients with expired domains with backlinks found on sites of their choosing. also for the architecture that you've outlined you'll need lots of Thêm

$250 USD trong 3 ngày
(238 Nhận xét)
8.7
zekovicm

Hi there,I am Web Scraping expert from Bosnia & Herzegovina,Europe. I have created similar crawler in the past for checking expired domains . I can do it again for you ! Check out my profile and former clients feedba Thêm

$220 USD trong 5 ngày
(91 Nhận xét)
7.1
devi222

Hi Sir, I have created same crawler 2 years ago for some employer and a lot of crawlers for other types, I can complete in 2 days. Im a Python Programmer. I have created many Bots to scrape the websites like Google, Al Thêm

$150 USD trong 2 ngày
(119 Nhận xét)
6.8
Bluesky122

Hello. I have read your project description carefully and bid here. I am a senior python developer and have expertise in webscraping. I can complete your project in a day. I will use [login to view URL] for GUI. I can pro Thêm

$200 USD trong 1 ngày
(29 Nhận xét)
5.8
yongjin818

Dear As an expert in web scraping, I developed 500+ spiders using python, scrapy, php,nodejs,c# and proxies. And I scraped data and download images, files for CEOs and output result as csv, json, xml(yellow, ebay, wi Thêm

$140 USD trong 7 ngày
(60 Nhận xét)
6.0
schoudhary1553

Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl Thêm

$250 USD trong 4 ngày
(14 Nhận xét)
5.8
brianconey

Hello How are you I have read your job description in detail . I am Xu It is a piece of cake i have done it before Please contact me and Let's discuss more thanks for your posting . hope to hear from you .

$140 USD trong 7 ngày
(17 Nhận xét)
5.5
mtanveer2012

Hi, 1. Project Price in Full to do phase 1 Ans: 150 USD. 2. Time to complete, since I am really needing this soon Ans: 1-2 Days maximum. 3. What language you will be using Ans: Python. 4. Any relevant experience you ha Thêm

$150 USD trong 1 ngày
(38 Nhận xét)
5.3
smsaurabhv

‌Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHO Thêm

$111 USD trong 3 ngày
(51 Nhận xét)
5.0
alexwmsoft

Dear Client Thank you for taking for your precious time to looking for my bid. I have fully read the job description and it is very clear to me. I am a senior web scraper who is good at python. I am using pytho Thêm

$155 USD trong 3 ngày
(4 Nhận xét)
4.5
sky930320

Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spide Thêm

$250 USD trong 7 ngày
(8 Nhận xét)
4.1
shiyusun180409

I checked your description. As I am a software and scrap expert , I have a wealth of experience and skills in these projects. I prefer python language and i will complete it within 1~2 days. You will get good results. Thêm

$150 USD trong 2 ngày
(13 Nhận xét)
4.2
shaktistanwar

Hi I am a technical evangelist, developer and can do this task for you. I have more than 15 years of experience in software development and am a microsoft certified professional.I have strong work experience in Nopcom Thêm

$250 USD trong 3 ngày
(1 Nhận xét)
3.4
PiushGoutam2018

Hi, I am interested in taking up your project. I have done 10+ projects in python software development,website development,desktop automation,website automation and web scraping,.I deliver fast,clean and quality work Thêm

$35 USD trong 1 ngày
(17 Nhận xét)
3.4
NonikNone

Hello how are you? My name is Ugur Catak. I am a software developer. About your project: Since we don't need to download entire page, we can request for the status code only to speed the bulk query cycle. 1. Project Thêm

$110 USD trong 2 ngày
(2 Nhận xét)
2.4
Xiong402Jin

1. Project Price in Full to do phase 1 $200 2. Time to complete, since I am really needing this soon two days 3. What language you will be using python 4. Any relevant experience you have with crawl Thêm

$200 USD trong 3 ngày
(3 Nhận xét)
2.3
drohanyadav

Hi, I will be using python language to do this project. I work fast, and I will probably finish it in few hours, but there are always some small complications, so it will be take me maximum of 2 days. I have been web s Thêm

$60 USD trong 2 ngày
(3 Nhận xét)
2.0
dhirajbaheti

Hi, My name is Dhiraj and I am a professional web scrapper and data mining expert in python with more than 6 years of experience. I have expertise in scraping complex/dynamic websites using requests, BeautifulSou Thêm

$150 USD trong 1 ngày
(3 Nhận xét)
2.0
frelanceralam201

Hi 1. 150$ 2. one day 3. NodeJs 4. I'm a senior developer and have good experiences on web crawling and scraping development. Although I've just started using this website, I have a lot of experience with Selenium Webd Thêm

$150 USD trong 1 ngày
(1 Nhận xét)
1.5
BestService222

Hi I've carefully read the description of your project. I am very interested in your project As I am a professional Crawler Developer, I am sure I can help you I'm ready to start to work on your project right now. Thêm

$140 USD trong 3 ngày
(1 Nhận xét)
0.8