- We need you to build us a crawling tool from scratch which follow these requirements:
--- Requirements ---
- The tool should be used Scapy Python and MongoDB.
- It should dockerized into docker-compose separated into two containers: web and db.
- It should be easier to extend if we gonna crawl the new site.
Such as we will have folderA will crawl site A, folderB will crawl site B...etc
- It should be easier for configure the database and the site we gonna crawling.
- When the structure of codebase is done, we will review it and you need to send us a documentation to show:
+ How can we setup crawler for new site ?
+ Which file/folder need to be created
+ Which command need to be executed to crawl all the sites which were configured in the codebase.
- We will send you an example site, so you will do an example for us when the structure of codebase is done.
- I will send you invitation to github repository then you will work on it.
---- Note ---
- Please read it carefully before you ask us or bid with the right price.
- Only release payment if the codebase is good enough and easy for us to crawling the site.
- Please mark prefix 'DDPythonMongo' when you bid. Otherwise, I'll skip the bid due to automatically bid or bot.
14 freelancer đang chào giá trung bình $204 cho công việc này
DDPythonMongo I have rich experience in Crawling system and Python, MongoDB. I fully understood your deeds. I have some questions to ask you. Can I have a chat with you? Hope you reply. Thank you.
Currently working at a large project where my responsibilities are crawling, maintaining and updating big websites like google news, yahoo news, bing news etc.