Purpose
Reporting to the Manager – Data Engineering, the Data Engineer is responsible for documenting data models, architecting distributed systems, creating reliable data pipelines, combining data sources, architecting data stores, and collaborating with the data science teams to build the right solutions for them.
Responsibilities
- Assisting in quality control of quantitative and qualitative research projects.
- Designing, building and support data processing pipelines to transform data using Cloud technologies e.g. AWS, Azure, GCP etc.
- Working with business analysts to understand business requirements and use cases.
- Developing and executing quality assurance and test scripts.
- Building survey and voter file data pipelines.
- Designing, building data assets in cloud solutions and/or oracle ODI.
- Designing and developing tools for internal team use to simplify complex task flows.
- Designing schemas, data models and data architecture for cloud environments.
- Implementing data flow scripts using Unix and/or scripting.
- Creating scripts and workflows to automate repeated data processing tasks.
- Working with data team to identify opportunities for task automation and helpful tools
Skills
- Ability to work across regional teams with strong communication and collaboration skills
- Tendency to go above and beyond to make things work; manage own and others work to meet the deadline and assist other team members in their deliverables
- Ability to identify solutions and make the complex simple.
- Ability to breakdown complex problems into simple solutions
Qualifications
- Bachelor’s degree in computer science or related fields
- Minimum three years’ experience in a data engineer role or similar.
- Experience with data pipeline and workflow management tools: Airflow, etc.
- Some experience and knowledge of coding language such as Python.
- Good experience and knowledge of the SQL query language.
- Some understanding of star schemas and data warehouse concepts.
- Some knowledge of AWS/Azure tools and technologies Beneficial – ETL and ELT experience – both batch and microservices-led.