Data Engineer Agility UAE

Agility is presently hiring suitably qualified and experienced candidate for job role of Data Engineer. The selected candidate will be responsible for the development, performance, quality, and scaling of data pipelines, with a special focus on distributed processing and data quality, will be working with stakeholders throughout the company.

Duties of A Data Engineer

• Build solutions and tools to enable, organize and process large amounts of data.
• Design and implement data ingestion and transformation in cloud-based environment and in on-premise platform.
• Create and maintain optimal data pipeline architecture for both batch and real-time processing.
• Developing scalable, flexible, and high-performance distributed data pipelines with real time stream processing.
• Identify, design, and implement internal process improvements, automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
• Work closely with data scientists to define the best way to ingest and aggregate required data and process it efficiently, and how to get their data science models to run in production.
• Collaborate with business stakeholders to ingest new sources, cleaning and preparing the data and writing the code that connect it to the data lake or to specific application(s).
• Evaluate new technologies and build prototypes for continuous improvements in Data Engineering.

Eligibility Requirements

The eligibility requirements for role of a Data Engineer are as under:

• MSc degree (or equivalent) in Computer Science, Informatics, Information Systems or another quantitative field.
• 4+ years’ experience in data engineering / architecting in big data environment both on-premise and in the cloud.

Skills & Knowledge Desired

• Building and optimizing complex data pipelines and architectures.
• Experience architecting, designing, and implementing cloud solutions on AWS environment is a must.
• Orchestration and workflow management tools such as Airflow, AWS Step Function, MLflow, Kubeflow, etc.
• Developing and managing stream-processing systems Hadoop/Spark-Streaming, Kafka/Flink, AWS Kinesis…
• Object-oriented/object function scripting languages Python, Java, Scala, etc.
• Deployment and provisioning automation tools e.g., Docker, Kubernetes, Terraform.
• Knowledge of search engines such as Elastic search.
• Technical aspects of Data Governance including glossary, metadata, data quality, master data management & data lifecycle management.
• Security, authentication and authorization.
• DevSecOps / MLOps and CI / CD frameworks.

Group Data Engineer Dubai Ports (DP) World