Lead Data Engineer (Azure)

Job description

2 days ago Be among the first 25 applicants

Get AI-powered advice on this job and more exclusive features.

About Fusemachines

Fusemachines is a leading AI strategy, talent, and education services provider.

Founded by Sameer Maskey Ph.D., Adjunct Associate Professor at Columbia University, Fusemachines has a core mission of democratizing AI.

With a presence in 4 countries (Nepal, United States, Canada, and Dominican Republic and more than 400 full-time employees).

Fusemachines seeks to bring its global expertise in AI to transform companies around the world.

About the role

Location: Remote (Contract)

This is a full-time position responsible for leading, designing, building, and maintaining the infrastructure required for data integration, storage, processing, and analytics (BI, visualization and Advanced Analytics) using Microsoft Azure in the Media domain.

We are seeking a Lead Data Engineer with hands-on Python, Spark experience and proven abilities to lead software development on Data and Analytics products using Agile methodology.

We are seeking a well-rounded senior data engineer to lead a cloud based Big Data product team using a variety of technologies.

The ideal candidate will possess strong technical, analytical, and interpersonal skills.

In addition, the candidate will lead data engineers, data scientists on the team to achieve architecture and design objectives as agreed with stakeholders.

Qualifications & Experience

• Must have a full-time Bachelor's degree in Computer Science or similar from a top tier school
• 4+ years of experience with Azure DevOps, Azure Cloud Platform, or other hyperscalers
• At least 4 years of experience as a data engineer with strong expertise in Azure, working on generation of big datasets using different data sources, in the Media industry
• Proven experience delivering projects and products for Data and Analytics as a data engineer
• Following certifications:
• Microsoft Certified: Azure Fundamentals
• Microsoft Certified: Azure Data Engineer Associate
• Microsoft Certified: Azure Solutions Architect Expert: nice to have
• Databricks Certified Associate Developer for Apache Spark
• Databricks Certified Data Engineer Associate, nice to have

Required Skills / Competencies

• Strong programming skills in Python (must have), Scala, and proficiency in writing efficient and optimized code for data integration, storage, processing and manipulation
• Strong experience using Markdown to document code or automated documentation tools (e.g., PyDoc)
• Strong experience with scalable and distributed data processing technologies such as Spark/PySpark (must have; Azure Databricks is a plus), DBT and Kafka
• Expert in designing and implementing efficient ELT/ETL processes in Azure (Azure Data Factory is a plus) and using open source solutions
• Skilled in data integration from APIs, databases, flat files, event streaming with Azure Data Factory
• Expertise in data cleansing, transformation, and validation
• Hands-on experience with Jupyter Notebooks and Python packaging and dependency management: Poetry, PipEnv
• Proficiency with Relational Databases (Oracle, SQL Server, MySQL, Postgres, or similar) and Non-SQL Databases (MongoDB)
• Good understanding of data modeling and database design principles; ability to design efficient database schemas
• Strong understanding and experience with SQL and advanced SQL queries
• Strong experience in designing and implementing data warehousing solutions in Azure with Azure Synapse Analytics and/or Snowflake
• Familiarity with migration of code from SAS, R, Julia, SPSS to Python
• Proven technical leadership on prior Big Data projects
• Strong understanding of the SDLC, especially Agile methodologies
• Strong knowledge of SDLC tools and technologies (Azure DevOps, Jira, Azure Boards, GitHub, Azure Repos, Bitbucket, CI/CD systems, Azure Artifacts)
• Strong understanding of DevOps principles, including CI/CD, IaC, automated testing, and cost management
• Strong knowledge of Azure data and analytics services (Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Stream Analytics, SQL Server, Azure Storage options)
• Experience in orchestration using Apache Airflow
• Strong analytical and problem-solving skills; ability to debug and troubleshoot in complex data environments
• Good understanding of data quality and governance, including data quality checks and monitoring
• Good understanding of BI solutions including Power BI and Tableau
• Knowledge of containers and their environments (Docker, Kubernetes, etc.) is a plus
• Effective written and verbal communication; ability to collaborate with cross-functional teams
• Ability to document processes, procedures, and deployment configurations
• Leadership skills with the ability to lead, generate ideas, and be assertive
• Understanding of Azure security practices and compliance standards; ability to implement security controls within data solutions
• Willingness to stay updated with the latest Azure services and industry trends
• Well organized, able to work in a fast-paced environment and prioritize effectively
• Team player, self-motivated, able to work independently while maintaining attention to detail
• Commitment to agility and continuous learning

Responsibilities

• Lead engineers on the team to meet product deliverables
• Architect, design, develop, test, optimize and maintain large-scale data architectures for data intake, validation, mining, and engineering for delivering data products
• Support data integration (batch and real-time), storage, processing, and infrastructure to ensure scalability and performance
• Provide mentorship and guidance to junior data engineers
• Collaborate with cross-functional teams to drive discovery and requirements for data management and analytics
• Work with product management to understand roadmap commitments and communicate milestones
• Identify and solve code/design optimization
• Learn and integrate with various systems, APIs, and platforms
• Establish a QA process for data validations and quality control
• Continuously evaluate and implement new technologies; promote reusable components
• Design and maintain data governance, cataloging, lineage, data quality, and security
• Work in Agile development environment and participate in Agile ceremonies
• Contribute to continuous improvement and stay updated on market trends

Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.

Powered by JazzHR

S5Q7radK8J

Required Skill Profession

Informática Y Tecnología