Data platforms where pipelines fail loudly — not silently
I design and build cloud data platforms that survive partial failures, schema drift, and operational scale — incremental ingestion with checkpoint recovery, contract-driven data quality, medallion lakehouse patterns, and CI-validated platform automation on AWS, Azure, Databricks, and Snowflake.
Senior data engineering, to me, is judgment under constraint: intentional trade-offs, failure-aware design, and systems the next team can actually operate.
Design for failure.
Automate repeatable work.
Measure data quality.
Document decisions.
Keep systems understandable.
| Signal | How it shows up |
|---|---|
| System design | Architecture docs, ADRs, and clear boundaries between ingestion, transform, quality, and platform layers |
| Failure handling | Checkpoint recovery, idempotent loads, quality gates before bronze, operations runbooks |
| Operational proof | pytest, GitHub Actions CI, smoke tests, alert routing, and honest trade-off documentation |
| ⭐ Lakehouse |
lakehouse-platform-starter — Runnable reference architecture: Airflow + Cosmos · dbt · Iceberg · Trino · OpenLineage · Great Expectations
|
Connected layers — not isolated demo repos:
| Layer | Project | Focus |
|---|---|---|
| Flagship · Lakehouse | lakehouse-platform-starter | Iceberg + Trino + Cosmos dbt + Airflow + Marquez + GE · Docker stack · CI · hosted docs |
| Ingest & transform | production-data-pipeline | Incremental API · PostgreSQL bronze · dbt · Airflow · v0.1.0 |
| Quality & observability | data-quality-observability | YAML contracts · schema/freshness checks · run history · alerts |
| Platform & governance | cloud-lakehouse-blueprint | Medallion manifests · Terraform · IAM · lineage · CI validation |
| Domain | Technologies & practices |
|---|---|
| Data engineering | Python · SQL · incremental ingestion · ETL/ELT · API pipelines · checkpointing · idempotent loads |
| Data architecture | Medallion lakehouse · Iceberg · bronze/silver/gold · lineage · governance · cost modeling |
| Orchestration | Apache Airflow · Cosmos · dbt · Prefect · Spark |
| Cloud platforms | AWS · Azure · Databricks · Snowflake |
| Quality & observability | Data contracts · Great Expectations · OpenLineage · schema validation · CI/CD |
Production operations knowledge contributed upstream:
| Project | PR | Change |
|---|---|---|
| Airflow | #71158 | Clarify metrics vs traces otel_* config options (#43366) |
| Airflow | #70171 | Surface dbt Cloud failure details in Airflow task logs |
| Prefect | #22500 ✓ merged | Kubernetes readiness vs liveness probes |
| Prefect | #22533 | Global concurrency limit setup docs (re-review requested) |
| dbt docs | #9606 ✓ merged | Prefixed custom schema troubleshooting |
Since 2026 — I publish production-style data platform work in the open: portfolio repos, upstream contributions, and technical writing. Prior employer work lived in private GitLab/Azure DevOps; this GitHub profile is my public proof of craft, not a full career timeline.
| What you'll find here | Where |
|---|---|
| Lakehouse + pipeline + quality + platform stack | Pinned repos |
| Upstream OSS (Prefect, dbt, Airflow) | OSS table above |
| Architecture write-ups | Dev.to · br413.github.io |
| 90-day contribution plan | docs/90-day-contribution-plan.md |
| Daily activity log (automated) | docs/contribution-log.md |
Each flagship repo includes ADRs, pytest coverage, GitHub Actions CI, operations runbooks, and documented trade-offs — not toy demos.
Current focus (next 90 days): land Airflow #70171 + Prefect #22533, open Airflow #43366 + Meltano #6289 PRs, release production-data-pipeline v0.2.0, publish article #2 on data quality contracts.
| Article | Topic |
|---|---|
| Building a Production Data Pipeline with Incremental Loading and dbt | Incremental checkpoints, idempotent loads, medallion layering, Airflow orchestration, failure modes |
More at br413.github.io · series: Cloud Data Platform Patterns
Open to senior data engineering roles, data platform architecture discussions, and technical collaboration.
Website: br413.github.io · Flagship: lakehouse-platform-starter · dbt docs: live
data engineering · lakehouse · dbt · Airflow · Iceberg · Trino · OpenLineage · Terraform