We are looking for an experienced Azure Databricks Data Engineer to help build and operate scalable, production-grade data platforms on Azure Databricks and Microsoft Azure.
The ideal candidate combines strong Spark engineering expertise with a production engineering mindset, treating data pipelines as reliable, enterprise-grade products rather than ad-hoc solutions. This role requires hands-on experience delivering secure, governed, high-performance batch and streaming data platforms at scale.
- Design, build, and maintain scalable batch and streaming pipelines using bronze/silver/gold architectures
- Develop reusable transformation frameworks and production-grade Spark workloads
- Implement and optimize Delta Lake solutions, including: ACID transactions, Partitioning strategies, Schema evolution, OPTIMIZE/ZORDER, Incremental and CDC processing patterns
- Configure and manage Unity Catalog permissions, governance, lineage, and auditability
- Perform Spark performance tuning and observability monitoring
- Integrate Databricks with Azure services including ADLS, ADF/Synapse, Key Vault, and Microsoft Entra ID
- Implement CI/CD pipelines and deployment automation using Azure DevOps or GitHub Actions
- Collaborate with architects, analysts, data scientists, and engineers to align technical implementation with business needs
- Continuously improve platform performance, reliability, cost efficiency, and standardization
Strong hands-on expertise with Azure Databricks engineering
Advanced Apache Spark experience using:
PySpark or Scala
Spark SQL
Deep understanding of Delta Lake architecture and optimization techniques
Experience building scalable enterprise data platforms and real-time pipelines
Strong governance and security experience in multi-team environments
Experience with service principals, managed identities, lineage, and auditability
Strong troubleshooting and Spark performance optimization skills
Azure integration experience across storage, orchestration, and security services
Solid CI/CD and Infrastructure-as-Code exposure
Comfortable working independently in consulting-style delivery environments
