Đã Đóng

[PYTHON & GCP EXPERT] Data ingestion and processing from Pub/sub to Bigquery -- 2

I'm trying to automate the ingestion of data from Pub/sub to Bigquery. I need help writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I intend to use cloud scheduler, cloud pub/sub, cloud functions, and big query. Do not intend to use an apache beam as the volume of data is very low. (<100 records per day)

Deliverables

Complete writing cloud function which does the following:

1. Picks up data from pub/sub.

2. Processes the data and compares it to data in another table in BigQuery.

3. Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column that describes the status as 'Met', 'Not Met', 'Partially Met'

Please find project details in the attached excel file.

[login to view URL]

1. Data flow -> Pub/sub - Cloud function - Big Query

2. Flow logic - Every time data is pulled, the cloud function must use logic to map the ID, category, and status from the source table(highlighted in green) to Vul-CCF mapping, Vul-status to Control Health status, vul codename to app name with the environment and insert/update record in the destination table.

Kĩ năng: Python, Kiến trúc phần mềm, MySQL, Khai thác dữ liệu, Hadoop

Xem nhiều hơn: data entry processing officer, excel data entry processing controls, data entrydata processing, data entry processing skills, python script extract data web page, python script crape data, simply data email processing, python read input data file, scarborough data entry processing, python script extract data, python script read data text file, python script extract data website, expert data processing, python data stream processing, gcp pub sub ack, data being published to pub/sub must be sent as a bytestring., data being published to pub sub must be sent as a bytestring, google functions pub sub python

Về Bên Thuê:
( 56 nhận xét ) Siem Reap, Cambodia

ID dự án: #30965942

4 freelancer chào giá trung bình$88 cho công việc này

InstantCreate

I have read your job description and assure you that I am a perfect fit for the job for [PYTHON & GCP EXPERT] Data ingestion and processing from Pub/sub to Bigquery -- 2 I am well versed in various types of graphic d Thêm

$100 USD trong 1 ngày
(2 Nhận xét)
2.8
sharashkynkonst2

Hello, sir. How are you? Thank you for your job posting:<CACA> I am interested in your project. I have read your proposal and I am confident. I am ready to start just now. I have rich experience in python development. Thêm

$75 USD trong 7 ngày
(1 Nhận xét)
0.6
mjavidnaseemoff

Hi, Greetings, I have gone through the given job description and I'm very sure that I can do this. last 1 year working in GCP projects. So i can do that. current working project technology stack is NIFI, Pub/sub, cl Thêm

$75 USD trong 1 ngày
(0 Nhận xét)
0.0
agupta2812

Hello there! Good day! I am interested in this project. I can get this done for you at the earliest. I am a Python & GCP expert with more than 7 years of experience. Please reach out to me to discuss further. Thank Thêm

$100 USD trong 7 ngày
(0 Nhận xét)
0.0