Airflow , Kafka Kinesis Project

$8-15 USD / hour

Đã đóng

Đã đăng vào

hơn 3 năm trước

$8-15 USD / hour

A>Airflow 1. Take any small source CSV file and copy it into target S3 using Airflow 2. Take any CSV in S3 and copy that into target local server using airflow 3. On local server, airflow calls a simple python file , that reads say a json and pushes data into output say CSV schedule above one after anotther to run every ten minutes ( since the files are small, we can run them back to back) B> Kafka (and / or) Kinesis 1. Take any Kafka producer code that writes into topic , pyspark consumer reads from topic 2. Say the kafka producer has hundred records. consumer reads say thirty records and goes down. Now the kafka consumer should be restarted from thirty first record. This is part of off-set management. Checkpoints to be created. feel free to take up PartA> Airflow or PartB> Kafka &/Kinesis ,separately based on your preference.

Python

Software Architecture

Linux

Data Processing

ETL

Mã dự án: 28252219

Về dự án

4 đề xuất

Dự án từ xa

Hoạt động 3 năm trước

Bạn muốn kiếm tiền?

Địa chỉ email

Lợi ích khi chào giá trên Freelancer

Thiết lập ngân sách và thời gian

Nhận thanh toán cho công việc

Phác thảo đề xuất của bạn

Miễn phí đăng ký và cháo giá cho công việc

4 freelancer chào giá trung bình $18 USD/giờ cho công việc này

@maadirajudata

Hi I am interested in part b Kafka producer and consumer. Hope it can be achieved in local environment please confirm I have expertise in Kafka based applications thanks maadiraju

$22 USD trong 20 ngày

4,9

(31 nhận xét)

5,5

@anzarulislam

Hello, I want to do your Airflow , Kafka Kinesis Project But I have few questions. Can I ask? Let's talk and start now. Best, Anzarul.

$15 USD trong 40 ngày

4,7

(19 nhận xét)

5,1

@DevraniAnuj

Hi I have worked with Airflow extensively. I am interested in the A part. Please explain in detail so that we can get started quickly.

$17 USD trong 10 ngày

0,0

(0 nhận xét)

0,0

@bharatthe1

Greetings, How are you! After going through your requirements, I am interested in Part A and I would like to share that I can fulfill your requirement. Below is brief description about my self. I am having 8+ years of working experience in Technologies like Python, Java and other big data technologies like Spark, Hive and Hadoop. Also developed many spark applications in AWS by implementing AWS Services like Lambda, Step Functions, SNS, Glue, Redshift. Also I have created Airflow DAGs by creating custom hooks and operators to connect to s3, Lambda, EMR, and SSM(System Manager) and deployed these schedulers in our prod implementation. With above experience and skillset, I strongly feel that you will be highly satisfied with the quality of work and on time delivery. You can reach out to me any time. Looking forward to hear from you! Thanks.

$18 USD trong 40 ngày