Toptal
Senior Data Engineer – AWS Data Lake & Pipeline Architecture
About this role
Summary We are seeking a Data Engineer to support the development of a Data Intelligence Platform. This role focuses on data modeling, data services, data pipelines, and cloud-based data infrastructure for reporting, analytics, and data science. General information This Data Engineer will support data-related operations across a broader Data Intelligence Platform team. The work includes building and consuming web services, integrating search technologies, supporting personalization and recommendation engines, and applying data engineering best practices across the platform. The role spans cloud-based data architecture, database performance, pipeline automation, and support for data scientists, researchers, and internal business units. The environment includes AWS-based infrastructure and a mix of structured and unstructured data sources. Tasks and Deliverables Required experience Engagement highlights
- Participate in the architecture design and implementation of high-performance, scalable, and optimized data solutions.
- Create data models from scratch using strong SQL fundamentals.
- Write and optimize in-application SQL statements.
- Ensure the performance, security, and availability of databases.
- Prepare documentation and technical specifications.
- Handle database procedures such as upgrades, backups, recovery, and migration.
- Profile server resource usage and optimize configurations as necessary.
- Design, build, and automate the deployment of data pipelines and applications.
- Integrate data from on-premise databases and external data sources using REST APIs and harvesting tools.
- Collaborate with business units and data science teams on data access, transformation, processing, and reporting needs.
- Support implementation, technical issues, and training related to the data lake ecosystem.
- Work with the team to manage AWS resources, including EMR and ECS clusters.
- Support provisioning, monitoring, configuration, and maintenance of AWS tools.
- Evaluate and promote new cloud technologies that improve capabilities and lower operating costs.
- Support automation efforts using Infrastructure as Code with Terraform and CI/CD tools such as Jenkins.
- Work with the team to implement data governance, access control, and security risk reduction.
- 7-9 years of experience designing and developing cloud-based data models, ETL pipelines, and infrastructure.