GCP ETL Pipeline Development

@gogora

Hi I will build a robast end-to-end ETL pipeline, with Airflow, DataProc and Bigquery. Please provide information on 1) Source type and data structure. 2) SQL model specifications and schema. 3) Transformation requirements. I’m happy to discuss the details over chat, and I will respond promptly. Thanks for your attention Archil

@rishi9343

With your GCP ETL Pipeline Development project, choosing me as the developer will give you immediate access not only to advanced backend and database skills required for this task, but also to a wealth of experience in data processing, ETL design, and GCP architecture. My 8+ years in the industry has exposed me to numerous intricate projects demanding robust data pipelines via Airflow, large-scale data transformation on Dataproc using PySpark and BigQuery modeling. The timeliness and efficiency of my work are consistently praised by my previous clients. My proficiency in both native iOS and Android development reflects my attention to detail and adaptability, which I believe is crucial in translating your requirements into a smoothly running end-to-end project. In addition to delivering code that meets acceptance criteria; I won't just settle for completing this project. I will ensure that parameterised configurations are implemented to guarantee that your environment can be transitioned from dev to prod without having to deal with unnecessary code changes. We can even evaluate how we could integrate Firebase into the mix if it is something feasible for your project. Choose me for a well-optimised GCP ETL solution that meets both your current and future business needs.

@Hrolgar

Three validated historical loads is a specific requirement that deserves a clear acceptance criteria definition before the pipeline starts. Row counts plus a hash sample against the on-prem source is the baseline. For anything with business-critical data you'd typically add null checks on key columns and a uniqueness assertion at the BigQuery destination. I'd pin those criteria in the repo README so sign-off is straightforward. On the pipeline itself: Airflow DAGs with parameterized GCS bucket names, BQ dataset IDs, and Dataproc cluster configs pulled from Airflow Variables, so dev/prod promotion is a variable swap rather than file edits. Each DAG triggers an ephemeral Dataproc cluster that spins up for the job and terminates on completion, so you're not paying for idle Spark infrastructure between loads. Retry logic and alerting are in the DAG definition; dead letter handling goes to a separate GCS prefix. PySpark jobs are tuned per-job, not generic templates. On-prem schemas don't always translate cleanly to BigQuery types, so the extraction layer includes explicit type coercions and a schema validation step before any data lands in BQ. Clean git repo with dev/prod configs, documented DAGs, PySpark job modules, and three historical loads passing the agreed validation criteria. 12 days, 63,750 INR, one milestone. What's the on-prem source DB? That shapes how I write the extraction layer.

@mahmod25

Hello, I can help build your end-to-end ETL pipeline on Google Cloud using Airflow, Dataproc (PySpark), and BigQuery. My experience includes designing data pipelines, building ETL/ELT workflows in Python, orchestrating jobs with Airflow, developing PySpark transformations, and creating analytical models in SQL. For this project, I can deliver: * Secure ingestion from on-prem databases to GCS staging. * Airflow DAGs with scheduling, retries, logging, monitoring, and alerting. * Optimized PySpark jobs on Dataproc for scalable transformations. * BigQuery SQL models for refined and reporting layers. * Environment-based configuration for seamless promotion between dev, test, and prod. * Data validation and reconciliation checks to ensure row counts and key metrics match source systems. Deliverables: * Git-ready repository with clean project structure. * Airflow DAGs, PySpark scripts, and BigQuery SQL models. * Requirements, configuration files, and deployment documentation. * Validation and testing procedures for historical loads. I am available to start immediately and would be happy to discuss source systems, data volume, refresh schedules, and architecture requirements. Best regards, Mahmoud Hassan

@rajc6h6

Getting on-prem data into BigQuery smoothly requires an Airflow and Dataproc pipeline that won't fail silently or spike compute costs. I build production-grade Python and SQL data pipelines every single day. I engineered the entire backend ETL infrastructure for my startup (Synlitics) to process live, messy inputs automatically, so I know how to handle tight logging and heavy PySpark transformations. I can map out your DAGs and have a reliable dev pipeline running in under 5 days. Happy to do a quick, free audit first so you know exactly what you're getting. What database engine is your on-premise data currently sitting in?

@paleshapps

Hi there, I'm a Senior Data Engineer with 7 years of experience developing, maintaining, and optimizing ETL pipelines. I have worked extensively with Dataproc (PySpark) and self-hosted Airflow to automate workflows, implement error alerting, and ensure pipeline reliability. I have delivered data to multiple destinations, including Parquet files on GCS, BigQuery, and PostgreSQL. I also have experience optimizing ETL pipelines, achieving significant performance improvements. I can set up automated data pipelines tailored to your requirements. Let's connect to discuss your project in more detail. I'm available to start immediately. Thank you. Best regards, Charles

@JoseRCarvas

Hello, this is squarely my lane - on-prem to GCS to BigQuery, with Airflow orchestrating Dataproc PySpark jobs and BigQuery SQL for the modelling layer. I deliver it as production-ready code, not a black box - Airflow DAGs with retries, logging and alerting, parameterised PySpark transforms tuned for Dataproc, secure GCS staging ingestion from your on-prem source, and BigQuery SQL models exposing the refined tables, all driven by config so environments and tables switch cleanly. I validate the transform and SQL logic against representative sample data before deployment, so you see correct output early, then we deploy and tune cost and performance on your GCP with access shared here on Freelancer. Milestone 1 is the core path end to end - one source ingested to GCS, one Dataproc PySpark transform, and one BigQuery model, with a working Airflow DAG - then we extend to the full set of sources and models. I bid at your budget floor and keep all work and communication on Freelancer, starting once the first milestone is funded. Happy to align on your source database and target schema first. Thanks, Ricardo

@venkateshprabhas

I have 4+ years of experience building scalable ETL/ELT pipelines on Google Cloud using Python, SQL, BigQuery, Airflow, and PySpark. I have hands-on experience designing end-to-end data migration and transformation workflows from on-prem systems to GCP. I can build a robust pipeline to securely ingest data into GCS, orchestrate workflows using Airflow DAGs with retries, logging, and alerts, perform large-scale transformations using PySpark on Dataproc, and model refined datasets in BigQuery. I will ensure parameterized configurations for seamless dev-to-prod promotion, complete validation against source metrics, and deliver production-ready Git-managed code with deployment documentation.