Data Engineer – AWS

The Data Engineer will have a key role in helping to define and develop a new Data Lake to drive a new data-driven business strategy. The Data Engineer will help propose and implement solutions primarily based on AWS, including:

• designing and implementing ETL tasks.
• identifying key data relationships.
• cleansing data.
• building a framework for BI developers to generate reporting solutions.
• orchestrating the flow of data.
• Some experience in Map Reduce and Machine Learning is desirable.

The role will involve working with colleagues, both IT and business, to help define the longer-term Data Lake strategy., with a focus on agile working and continuous delivery. The initial scope is to focus on retail data, followed by financial data. Currently, AWS tools are being used, however, open-source solutions may be considered in the near future.

Key Skills:

• Cloud environments.
• ETL, data migration, and cleansing.
• Hands-on Apache Spark using Python or Scala, creating data frames and applying transformations/actions.
• Data Lakes, data analytics, and working with large data sets.
• Relational databases (particularly Oracle and PostgreSQL)
• Working independently.
• Agile working.
• Customer-first approach to development.

Experience:

• Hands-on experience with AWS.
• Hands-on experience with AWS Glue, Lambda, and Step functions.
• Hands-on experience with SQL, Python, and Scala
• Familiarity with AWS data lake-related components such as S3, Athena, Presto, QuickSight, EMR, Redshift, Kinesis, etc.
• Familiarity with structured RDBMS.
• Familiarity with large data techniques (Hadoop, MapReduce, Spark, etc.)
• Familiarity with providing data via a microservice API.
• Experience in analyzing and cleansing data using a variety of tools and techniques.
• Experience in working with a variety of data formats, JSON, XML, CSV, parquet, etc.
• Experience with building and maintaining data dictionaries / meta-data.
• Experience in Linux and cloud environments.
• Desirable, but not essential, to have retail experience.