Position title
Lead Data Engineer
Description

We are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team.

Responsibilities
  • Roles and Responsibilities

    1. Design and Architecture:
    • Lead the design and implementation of data pipelines using Databricks and Snowflake.
    • Design data lakehouse architectures for efficient data storage and processing.
    • Stay updated on the latest trends and best practices in data engineering.
    Coding and Development:
    • Write high-quality, efficient, and maintainable code for data pipelines and transformations.
    • Conduct code reviews to ensure code quality, adherence to best practices, and maintainability.

    2. Technical Support:
    • Provide technical support and guidance to junior data engineers on Databricks and Snowflake.
    • Troubleshoot data engineering problems and identify root causes.
    Technical Team Management (No People Management responsibilities)
    • Mentor and guide junior data engineers in their professional development.
    • Foster a collaborative and learning environment within the data engineering team.

    3. Hands-on Coding: A lead data engineer in this role is expected to be a strong coder who can contribute to the team's development efforts. This includes:
    • Developing data pipelines and transformations in both Databricks and Snowflake.
    • Writing unit tests for data engineering code to ensure quality and reliability.
    • Staying up-to-date with the latest libraries and frameworks for data engineering

    By combining technical expertise, strong leadership skills, and a hands-on coding approach, this lead data engineer can effectively guide the team in building and maintaining a robust data infrastructure using Azure Databricks and Snowflake.

Skills
  • Technical Skills:

    1. Databricks:
    • In-depth knowledge of Apache Spark and its ecosystem (SQL, Python, Scala) for data processing and analytics.
    • Experience with Databricks notebooks, clusters, and runtime configuration.
    • Familiarity with Delta Lake and its functionalities.

    2. Snowflake:
    • Expertise in writing SQL queries for data warehousing and analytics.
    • Understanding of Snowflake's architecture, including virtual warehouses and schemas.
    • Experience with data ingestion and transformation techniques in Snowflake.

    3. Cloud Technologies:
    • Familiarity with Azure cloud services for Databricks deployments (optional).

    4. Programming:
    • Strong proficiency in Python, Scala, or SQL for data manipulation and analysis.
    • Experience with data engineering libraries (e.g., Pandas, Koalas).
    Version Control Systems:
    • Proficiency with Git for code version control and collaboration.

    Soft Skills:

    1. Communication: Excellent written and verbal communication skills to collaborate with technical and non-technical audiences.

    2. Leadership: Ability to mentor and guide junior data engineers.

    3. Problem-solving: Strong analytical and problem-solving skills to tackle complex data engineering challenges.

    4. Teamwork Ability to work effectively in a team environment and foster collaboration.

DER/BA/00047
Job Location
Bengaluru, India
Experience
5
Date posted
June 18, 2024
Share
Close modal window
Position: Lead Data Engineer
Job ID: DER/BA/00047

Agree*

*Required fields

Loading...

Thank you! Your profile has been shared with HR.