Assistant Manager

Here's an overview of what it typically involves, along with the qualifications and skills needed Role and Responsibilities: • Data Pipeline Development: Design, develop, and maintain scalable batch and streaming ETL pipelines using Databricks, PySpark, Python, and SQL. • Lakehouse Implementation: Build and manage data solutions leveraging Delta Lake and medallion architecture principles to support trusted and reusable data products. • Data Integration: Integrate data from enterprise applications, databases, APIs, files, and cloud platforms to enable unified analytics and reporting. • Data Modeling & Transformation: Develop efficient data models and transformation frameworks that support business intelligence, advanced analytics, and AI/ML use cases. • Platform Optimization: Monitor and optimize Databricks workloads, cluster configurations, and data processing performance to ensure scalability and cost efficiency. • Data Quality & Governance: Implement data quality checks, monitoring, lineage, security controls, and governance practices in alignment with enterprise standards. • DevOps & Automation: Contribute to CI/CD implementation, automated deployments, testing frameworks, and release management for data engineering solutions. • Cross-functional Collaboration: Partner with business stakeholders, data architects, analytics teams, and AI engineers to deliver data products that support business outcomes. • AI & Analytics Enablement: Enable downstream AI, machine learning, and Generative AI initiatives by building reliable, high-quality, and governed data foundations. • Continuous Improvement & Innovation: Stay current with advancements in Databricks, cloud technologies, and modern data engineering practices to continuously enhance platform capabilities. Qualifications: • Experience: 4 - 6 years of experience in data engineering, ETL development, and cloud-based data platforms. • Technical Skills: • Proficiency in Python, PySpark and SQL • Strong expertise in Databricks Data Engineering capabilities, including: Databricks Lakehouse Architecture, ETL/ELT Design and Development, Delta Lake, Data Modeling, Batch and Streaming Data Pipelines, Spark Optimization and Performance Tuning • Experience in designing and implementing: Medallion Architecture, Data Quality Frameworks, Data Governance Controls, Metadata and Lineage Solutions • Experience with orchestration and workflow management tools such as Databricks Workflows, Azure Data Factory, or equivalent. • Experience implementing CI/CD pipelines, Git-based source control, automated testing, and deployment automation. • Strong understanding of data warehousing, dimensional modeling, and modern Lakehouse architectures. • Exposure to Databricks Mosaic AI, Vector Search, MLflow, feature engineering, or GenAI data preparation workflows (Preferred). • Familiarity with data preparation for RAG, LLM, and AI/ML use cases (Preferred). • Experience with infrastructure-as-code and containerization technologies (Preferred). • Certifications: Relevant certifications in data engineering, such as Databricks Certified Data Engineer Associate/Professional, or similar. (Preferred not mandatory) • Problem-Solving: Strong analytical and problem-solving skills, with a proven ability to design scalable data solutions and resolve complex data engineering challenges. • Communication: Excellent communication skills with experience working directly with clients and business stakeholders to deliver data and analytics solutions. • Stakeholder Management: Experience engaging with business stakeholders to understand data and analytics requirements and translate them into scalable data engineering solutions.