Databricks Engineer

Position Overview

We are seeking a Senior Databricks Engineer to lead the design and implementation of a modern Databricks Lakehouse architecture . This role is focused on migrating legacy data solutions into scalable, governed, and performance-optimized pipelines using Databricks, PySpark, and Delta Lake.

The engineer will play a key role in defining and building Bronze, Silver, and Gold data layers , enabling both batch and near real-time processing, and improving the reliability and usability of enterprise data platforms. This position will work closely with data engineers, architects, and stakeholders while providing technical leadership and guidance to more junior team members.

Key Responsibilitie

sArchitecture & Engineerin

• gArchitect and implement end-to-end data solutions within the Databricks Lakehouse environmen
• tDesign and build ingestion, transformation, and serving pipelines using PySpar k, Spark SQ L, and Delta Lak
• eEstablish and enforce medallion architectur e practices across Bronze, Silver, and Gold layer

sMigration & Modernizatio

• nLead the migration of legacy Azure SQ L, SSI S, and AD F pipelines into Databricks-native workflow
• sConvert existing T-SQ L and ETL logic into scalable Spark-based processin
• gSupport modernisation of data platforms with a focus on governance, scalability, and maintainabilit

yStreaming, Governance & Optimizatio

• nImplement Auto Loade r, Delta Lake Change Data Feed (CDF ), and streaming/batch ingestion pattern
• sDefine and enforce governance using Unity Catalo g, including RBAC and data lineag
• eOptimise performance through Photo n, partitionin g, Z-Order indexin g, and cluster tunin

gValidation & Technical Leadershi

• pGuide validation and reconciliation strategies between legacy SQL outputs and Delta-based pipeline
• sPerform code reviews and promote engineering best practice
• sMentor mid-level and junior engineers across design, development, and troubleshooting activitie

s
Required Experience & Qualificatio

• nsBachelor’s degree in Computer Science, Engineering, Information Systems, or equivalent experien
• ce6+ years of experien ce in data engineering, wit h 3+ years of hands-on Databricks experien
• ceStrong expertise i n PySpa rk , Spark S QL, an d Delta La
• keProven experience migrating legacy data platforms such a s SQL Serv er , SS IS, an d A
• DFStrong understanding o f ETL/ELT desi gn , data modeli ng, and pipeline architectu
• reExperience designing data solutions usin g Bronze / Silver / Go ld laye
• rsHands-on experience with th e Azure ecosyst em, includin g AD LS , A DF, and related servic
• esStrong knowledge of performance tuning techniques for distributed data processi
• ngExperience leading technical implementation efforts and supporting junior enginee

rs
Preferred Qualificati

• onsExperience wi th Unity Cata lo g, Lakehouse Federat ion, or CDC framewo
• rksDatabricks certifications (Associate or Professional lev
• el)Exposure to large-scale regulated datasets such as healthc ar e, cla ims, or financial d
• ataExperience with bo th ba tch a nd stream ing data pipeli
• nesFamiliarity with enterprise-grade data governance and lineage practi