Key Responsibilities

• Spark Implementation & Optimization: Act as a senior technical resource for
• Spark internals, applying best practices in memory management, shuffle tuning, and partitioning to ensure our Spark-based processing is performant and cost-effective.
• Cloud-Agnostic Development: Develop and maintain data pipelines usingPython, PySpark, and Delta Lake/Parquet, adhering to our strategy of decoupling code from specific cloud provider services and reducing reliance on GUI-based tools like ADF.
• Refactoring & Modernization: Contribute to the ongoing evolution of our platform by refactoring complex SQL-based ETL into modular, testable, and maintainablePython libraries.
• Lakehouse Engineering: Build and optimize Medallion Architecture(Bronze/Silver/Gold) layers using Delta Lake, ensuring efficient data versioning, schema evolution, and storage performance (Z-Ordering, Vacuuming).
• Code-First Orchestration: Support the transition toward code-centre orchestration patterns (e.g., Airflow, Dagster, or Python-based wrappers) that prioritize portability and reduce dependency on cloud-specific orchestration tools.
• Technical Excellence: Participate in code reviews, mentor junior engineers inPySpark best practices, and contribute to the development of automated testingframeworks (Pytest) to ensure high-quality, reliable data delivery.

• Senior-Level Spark: 6+ years of experience with Spark/PySpark, with the ability to diagnose performance bottlenecks via the Spark UI and optimize complex DAGs.
• Advanced Python: Strong proficiency in production-grade Python, with experience building reusable libraries and implementing automated testing.
• SQL Proficiency: Solid T-SQL skills to accurately interpret and migrate existing logic into the new Python-centric environment.
• Azure Synapse Analytics & Dedicated SQL Pools: Solid experience working within Azure Synapse, specifically with Dedicate SQL Pools and Data Factory working with complex pipelines
• Data Lakehouse Expertise: Hands-on experience with Delta Lake and Parquet, specifically in high-volume environments.
• Agnostic Mindset: Experience working with open-source standards and containerization (Docker) to ensure data workloads can be ported across different cloud infrastructures.
• Collaborative Engineering: Proven track record of working within a team to successfully deliver and modernize large-scale data workloads.
• Ensure high levels of security, compliance, and data governance across data engineering processes.

• Work closely with data scientists, analysts, and business stakeholders to understand data requirements and develop solutions that meet business objectives.
• Lead initiatives to enhance data engineering capabilities, introduce new technologies, and drive best practices.
• Mentor junior engineers, conduct code reviews, and contribute to building a culture of technical excellence.
• Communicate effectively with technical and non-technical stakeholders, translating complex data concepts into actionable insights.

Required Qualifications:

• Bachelor’s degree in computer science, Information Systems, Engineering, or a closely related field.

Why you’ll love working for us

We believe in taking care of our team and creating an environment where you can thrive.

As part of our company, you’ll enjoy:

• Flexible Working Arrangements: Whether you’re a night owl or an early bird, we offer hybrid and remote options to suit your lifestyle.
• Comprehensive Benefits: From a wellness program to home office reimbursements and continuous learning opportunities, we’ve got you covered.
• Team Culture: Fun team-building activities, regular socials, and a supportive, inclusive culture that values transparency, accountability, and work-life balance.
• Performance Incentives: Competitive salaries, ESOP, and recognition for your hard work.