Data Engineer (Scala Spark)

Logeek Magazine:
~/Luxoft Alumni Club:

/
Project Descriptio

~ n: We are seeking a highly experienced Databricks Architect to lead the design, implementation, and optimization of enterprise-grade data platforms and AI-enabled workloads for a global food corporation. This role requires a hands-on technical leader who can translate business and data requirements into scalable Databricks-based solutions while ensuring robust governance, performance, and operational resilience. ies: • Lead the architecture, design, and implementation of Databricks-based data platforms, pipelines, and AI/ML workl
Define scalable lakehouse patterns using core Databricks capabilities such as Apache Spark, Delta Lake, Databricks SQL, MLflow, and governance compon
Design robust batch and streaming data pipelines for high-volume, business-critical workl
Establish architecture standards for performance optimization, workload orchestration, reliability, observability, and cost con
Collaborate with data engineers, analysts, data scientists, DevOps teams, and business stakeholders to deliver end-to-end solut
Ensure strong data governance, security, and compliance practices across the Databricks environ
Support AI and advanced analytics use cases by enabling reliable feature engineering, model lifecycle practices, and production-ready data foundat
Strong command of Apache Spark and distributed data processing concepts, including performance tuning and optimization for complex data engineering wor
Proven experience designing and delivering enterprise data solutions using Delta Lake and lakehouse architecture prin
Strong proficiency in Python and SQL; Scala is an adv
Demonstrated capability in building and optimizing ETL/ELT pipelines, data models, and large-scale ingestion fram
Experience supporting AI/ML workloads on Databricks, including MLflow, model lifecycle considerations, and production-ready data prepa
Solid knowledge of cloud-native architecture patterns on AWS, Azure, or GCP, including storage, networking, identity, and security integ
Experience with data governance, access control, lineage, and compliance frameworks in enterprise enviro
Familiarity with CI/CD, infrastructure-as-code, monitoring, and operational best practices for data pla
Ability to engage in deep technical problem-solving and contribute immediately to complex delivery scenarios with minimal ramp-u