Principal Databricks Data Engineer
Experience Required: 12–18 Years
Key Requirements
- 12–18 years of overall Data Engineering experience
- 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
- 5+ years of hands-on experience with Databricks and Apache Spark at scale
- Strong experience modernizing legacy Cloudera platforms including:
- Modernize Cloudera platforms to Databricks Lakehouse architecture
- Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
- Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
- Ensure data reconciliation, audit integrity, and consistency during migration
- Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
- Implement layered data architecture including:
- Curated / Conformed Layer
- Semantic / Consumption Layer
- Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
- Strong experience with finance and risk data models including:
- Enable reporting capabilities including:
- Build and manage semantic and consumption layers for BI, reporting, and analytics
- Define business metrics, dimensions, hierarchies, and KPIs
- dbt or similar frameworks
- Develop and optimize large-scale data pipelines using:
- Implement Medallion Architecture including:
- Optimize workloads using:
- Cluster Configuration Tuning
- Establish data lineage and metadata management
- Ensure data security, access control, and compliance standards
- Experience with AWS or Azure cloud platforms
- Experience with CI/CD pipelines using:
- Experience with dbt is an advantage
- Act as a technical authority and lead enterprise architecture decisions
- Mentor senior engineers and establish engineering standards
- Collaborate with finance, risk, analytics, and governance stakeholders
- Translate complex data structures into business-ready insights
Nice-to-Have Skills
- Experience in BFSI, Capital Markets, or Regulatory Reporting
- Experience supporting AI/ML workloads
- Databricks and Cloud Certifications
Key Responsibilities
- Lead Cloudera to Databricks transformation initiatives
- Shape enterprise finance and risk data platforms
- Support regulatory, management, and analytical reporting systems
Essential Skills
- Data Warehouse & Data Lake
- Airflow / Databricks Workflows
- Finance & Risk Data Models
- Enterprise Data Architecture
