Đã Đóng

Create Production Ready ETL Pipeline using Python, Docker and Dagster

We need to create a dockerizable ETL Pipeline for our data using Python and Dagster. Our requirement / data flowchart is very clear.

The basic flow is read data from CSV, save to Kafka. Apply multiple transforms using functions (function code will be provided by us). Use intermediate kafka queues after each transform. Write final output to Postgres DB. Use Dagster for the data orchestration.

We just need to implement it in a robust manner. We need :-

1. Docker-compose to setup the Kafka, Postgres and other services.

2. Create a DAG for orchestration of the pipeline using Dagster (using placeholders for the transform functions which our team will add later).

Kĩ năng: Python, Docker, Docker Compose, ETL, Apache Kafka

Xem nhiều hơn: read csv save database using php, create app using python, how to create a dynamic website using python, python etl pipeline, how to create a print ready pdf file using adobe illustrator, how to create a website using python, how to create vm using python, how to create front end using python, how to create a tool using python, how to create a software using python, how to create dashboard using python, how to create a mobile app using python, how to create a social network site using python, how to create a website using python pdf, how to create windows application using python, create ppt using python, etl automation using python, how to create haar cascade xml for using with opencv python, how to create a registration form using python, create table in sql server using python

Về Bên Thuê:
( 0 nhận xét ) New Delhi, India

ID dự án: #32203643

3 freelancer chào giá trung bình₹22667 cho công việc này

f9programmer

Hello , I am a developer based out of Hyderabad , India . I have developed and hosted a few applications for my clients on PHP , Laravel , Nodejs , Vuejs , React and React Native . I have also helped my clients Thêm

₹35000 INR trong 7 ngày
(3 Nhận xét)
3.1
singhyamini1821

** Can start immediately once you hire me. Hello, I am an experienced mobile app developer with an experience of 5+ years. I’m pretty confident that I will be able to create your app as required. I am an expert in ma Thêm

₹20000 INR trong 5 ngày
(0 Nhận xét)
0.0
poiyutchain

Expertise in Python and Microservices: specialization: - Python, Django, Spark, ETL, Hadoop, BigData processing and analysis SQL and NoSQL - MySQL, PostgreSQL, MongoDB, Cassandra, Redis, etc - Docker - JMS, ActiveMQ Thêm

₹13000 INR trong 7 ngày
(0 Nhận xét)
0.0