Data Processing Software Architecture
$1500-3000 CAD
Thanh toán khi bàn giao
Purpose
This project is a proof of concept of a modern data processing engine that will bring to ingest, explore, prepare, manage, and serve data for immediate BI and machine learning needs. Currently, it is restricted to personal usage only.
Scope
The purpose of the modern data processing engine is to bring modern data warehousing capabilities with reasonable pricing ease.
System Features
• At the moment, the system is powered by a spark engine and other components.
• Functional Requirements: The system connects to the following data sources.
MS SQL Server
AWS Redshift
Azure synapse
web APIs
Flat Files
Snowflake
• The system has query engine capacity using SQL which allows running federated queries on the above-mentioned data sources except for web API.
• The system has integrated data processing functionality that allows users to use Apache Spark to process data from the above-mentioned data sources.
• The system has directed acyclic graph capability using Air flow.
• The system provides a search facility that will allow full text searching of all data in the memory of the engine that the user is permitted to access. The system must support the following searches:
find all words specified.
find any word specified.
find the exact phrase.
• The software is built on a framework or architect which lets it run on distributed systems.
• Secure impersonation inside its execution model for added security and offers data encryption and masking services.
• The system has connectors to share the output to third-party BI tools such as Power BI, or Looker.
Job Requirements
• Experienced architecting and developing software for scalable, distributed systems.
• Understanding the current architecture of the application and assisting in scaling the architecture cost effectively.
• Develop orchestration in the application.
• Enable users to schedule jobs on the basis of time and event.
• Containerization/packaging of the application so the application is not reliant on any local systems.
• Cost analysis of current configuration & architecture of the application.
• Comparative analysis of applications with various configuration with the same architecture.
• Writing efficient and modular code.
• Experience with cloud technologies and distributed system servers.
• Ability to facilitate demonstrations, proof of concepts.
• Deep understanding of spark application.
• Able to expertly convey ideas and concepts to others.
• Understanding of the public cloud market and pain points driving enterprise cloud adoption.
• In-depth understanding and the ability to demonstrate expertise in designing, deploying, and maintaining custom enterprise web applications.
• Prepare a high-level PowerPoint presentation and detailed word document of the application on completion of the project.
Nice to Have
• Strong knowledge of Python Machine Learning standard libraries.
• Strong understanding of all commonly used Machine Learning models and the main algorithms that compose the models.
• Good understanding of the built-in data types. (Lists, dictionaries, tuples sets).
ID dự án: #32246717
Về dự án
16 freelancer chào giá trung bình$2597 cho công việc này
Hi, This is M Zahid I went through the details of your project and would like to offer our services to deliver the results that you expect and we are sure you will not be disappointed. I got interested in your proje Thêm
Hi, I have very a good understanding of data processing systems, in fact I developed an application which allowed to gather data from 1 million shopify stores, which was used to later to apply machine learning usage. Thêm
Hi, I’m Senior Data Architect working for Globant company with having many Hadoop ecosystems like HDFS, Spark, Hive, ETL, Azure etc Please contact me for more details
Hey there ! Good day. I am Pankaj Ranga I would like to work with your project and I can surely provide the details you have specified on the project description. I can confidently say that I am knowledgeable enough to Thêm
Hello Greetings for the day, Am a professional freelancer expertized in various MSBI ,Power platform, Python and Azure data tools , with hands-on experience around 5 years in different segments of business, I will pro Thêm
Hi there, I am a Senior Software Engineer, having 9+ years of top experience in .NET Web, SaaS, Database, Migration, API, Library, Services, Reports, Dashboards. I can analyze, design, develop and manage small to larg Thêm
Hello there, Hope you are doing well! I have 5+ years of strong experience as Python Developer. I have expertise with Django, Flask, Python, AI, ML, NLP and Data Science. Can we have a quick chat/call to discuss thi Thêm
