Đã Đóng

Pyspark Data engineer for a test project in Spark

Looking for a pySpark developer to help with a pyspark application for demonstration purpose. The goal is to read 3 small json files each less than 1 mb into a single data frame. Do preprocessing and then fetch some metrics based on regular expression search. ETL task is easy and I am able to achieve it. However the end goal is written below that is also the acceptability criteria for this project.

The developer should be able to write code with following points in mind:

Well structured, object-oriented and maintainable code.

Unit tests for the different components.

Documentation, comments and Proper exception handling.

Solution is deployable and we can run it (locally and on a cluster)

Config management. (separate folders and files like [login to view URL], [login to view URL] etc. rather than one python script).

Logging and alerting.

Data quality checks (like input/output dataset validation).

Kĩ năng: Hadoop, Spark, Python

Xem nhiều hơn: simple test project php, data logger keil project, engineer plc project, data processing test, online test project aspnet, data mining poker project, ruby rails test project, bss engineer nortel project ususauk, sample data processing test, create test project, master data management sigma project, need sample data processing test, freelancer board test project, looking for data conversion outsourcing project, module 8 course project t test project plan research question and data assignment, data engineer project examples, data engineer side project, glider data engineer test, data engineer test

Về Bên Thuê:
( 0 nhận xét ) Pune, India

ID dự án: #29872024

1 freelancer đang chào giá trung bình ₹1050 cho công việc này

singhrahul2016

Hi, I am working in MNC as Data Engineer and currently working on Big Data Fields using PySpark and Hadoop Frameworks. Having more than 4 years of experience in Big Data Field in production and certified pyspark d Thêm

₹1050 INR trong 7 ngày
(4 Nhận xét)
2.2