Position: Lead Data Engineer (AWS Data Platform)
Experience: 5 to 8+ Years
Location: Hybrid (Pakistan)
Note: (Candidates must be available during UAE business hours and follow UAE public holidays)
Job Summary
We are seeking a highly skilled Lead Data Engineer with 5 to 8+ years of experience in designing, developing, and optimizing large-scale data platforms and ETL/ELT pipelines. The ideal candidate will have strong hands-on expertise in PySpark, AWS Glue, Amazon EMR, Amazon Redshift, and SQL-based data warehousing , along with proven experience in performance tuning and data optimization.
The candidate will work closely with a UAE-based customer and must be comfortable collaborating with distributed teams while adhering to UAE working hours and holiday schedules.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using PySpark , AWS Glue , and Amazon EMR .
- Build and optimize data ingestion, transformation, and processing frameworks for structured and semi-structured data.
- Develop and maintain enterprise data warehouse solutions using Amazon Redshift .
- Write complex SQL queries, stored procedures, and data transformations to support analytics and reporting requirements.
- Implement ETL/ELT processes to move data efficiently across multiple systems and platforms.
- Perform performance tuning and optimization of Spark jobs, ETL pipelines, SQL queries, and Redshift workloads.
- Ensure data quality, integrity, security, and governance across data platforms.
- Troubleshoot production issues and perform root cause analysis for data-related incidents.
- Collaborate with business stakeholders, analysts, architects, and engineering teams to understand data requirements.
- Participate in code reviews, technical design discussions, and best-practice implementation.
- Monitor data pipelines and proactively identify opportunities for performance improvements and automation.
- Create and maintain technical documentation, data models, and operational procedures.
Required Skills & Experience
Must-Have Skills
- 5 to 8+ years of experience in Data Engineering and Data Warehousing.
- Strong hands-on experience with PySpark .
- Extensive experience with AWS Glue .
- Experience building and managing workloads on Amazon EMR .
- Strong expertise in Amazon Redshift .
- Excellent SQL development and query optimization skills.
- Strong understanding of Data Warehousing concepts, dimensional modeling, and ETL/ELT processes.
- Experience in performance tuning of Spark jobs, SQL queries, ETL pipelines, and data warehouse workloads.
- Experience handling large-scale datasets and distributed data processing.
- Strong debugging, troubleshooting, and analytical skills.
Preferred Skills
- Experience with additional AWS services such as S3, IAM, CloudWatch, Lambda, and Step Functions.
- Knowledge of CI/CD pipelines and DevOps practices for data platforms.
- Experience with workflow orchestration tools.
- Familiarity with data governance, security, and compliance practices.
- Exposure to Agile/Scrum development methodologies.
Qualifications
- Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field.
- Relevant AWS certifications will be considered an advantage.
Soft Skills
- Strong communication and stakeholder management skills.
- Ability to work independently in a remote environment.
- Excellent problem-solving and analytical thinking abilities.
- Ability to collaborate effectively with cross-functional and geographically distributed teams.
- Strong ownership mindset and commitment to delivering high-quality solutions.
How many years of hands-on experience do you have in Data Engineering and Data Warehousing?
* How many years of hands-on experience do you have with PySpark? Can you describe the scale of data and types of pipelines you have built using PySpark?*
What is your experience with AWS Glue and Amazon EMR? Have you developed and managed production-grade ETL/ELT pipelines using these services?*
How many years of hands-on experience do you have with Amazon Redshift?*
Which of the following have you worked on? (Select all that apply)* Data Warehouse Design Dimensional Modeling (Star/Snowflake Schema) SQL Query Optimization Redshift Performance Tuning Stored Procedures Data Migration to Redshift
Can you provide an example where you optimized the performance of a Spark job, ETL pipeline, SQL query, or Redshift workload? What improvements were achieved?*
Are you comfortable to UAE working hours and holiday schedules?*
What is your current monthly gross salary?*
What is your expected monthly gross salary?*
What is duration of your notice period with current employer?*
#J-18808-Ljbffr
