Job Information
ENTERTAINER FZ LLC Senior Data Engineer in Lahore, Pakistan
Job Summary:
The Senior Data Engineer will play a crucial role in the design, mapping, documentation, and development of the enterprise data warehouse. This encompasses overseeing the entire lifecycle of data, including the creation and management of end-to-end job mechanisms for data extraction, transformation, and loading functions. Additionally, the incumbent should possess the ability to analyze business requirements and adeptly translate them into ETL job functions, leveraging industry best practices.
Job Responsibilities:
As a Senior Data Engineer , your primary responsibilities will include:
Developing data architecture and relational data models, designing database repositories, capturing data requirements, and preparing data dictionaries.
Develop and maintain scalable data pipelines using Apache Spark, Python, and related technologies to process large volumes of structured and unstructured data efficiently.
Design, implement, and optimize ETL processes to extract, transform, and load data from various sources into our data lake or data warehouse.
Collaborate with cross-functional teams including data scientists, analysts, and software engineers to understand data requirements and deliver robust data solutions.
Fine-tune Spark jobs and optimize performance by identifying and resolving bottlenecks in data processing pipelines.
Implement data quality checks and ensure data integrity throughout the data lifecycle.
Optimizing database throughput by fine-tuning connection pools for varying application needs.
Participating in all aspects of agile software development, including design, implementation, and deployment.
Developing Data Models and Data Marts from various source systems, creating and maintaining ETL scripts using data flow design.
Identifying and leveraging objects and code to reduce development effort and enforce consistent business rules.
Identifying opportunities for analysis and reporting solutions within the company.
Participating in the evaluation and selection of Data Modeling, BI, and DW tools.
Translating business requirements into technical Data Models and Data Marts, understanding key business drivers.
Clearly documenting all development activities to company standards, communicating task status, progress, and problems effectively to the project lead.
Qualification :
Successfully completed a degree in BS in Computer Science, Software Engineering, IT, BBIT or related field
3-5 years of experience in Data Analytics Field in the following Tool & Technologies:
Strong analytical and problem-solving skills
Excellent communication and interpersonal skills
Tools & Technologies
Proficiency in Azure Data Factory, Databricks, Azure Datawarehouse and SQL
Expertise in Apache Spark and Python for developing scalable data processing pipelines.
Familiarity with Azure Synapse Analytics for data warehousing solutions.
Experience in cloud implementations, with a preference for Microsoft Azure, showcasing practical expertise in leveraging cloud technologies.
Hands-on experience working with NoSQL databases, highlighting a comprehensive understanding of their structure and functionality.
Powered by JazzHR