Category: Jupyter

Zealpath and Trivago: case for AWS Cloud Engineer position

Tl;dr: https://github.com/markokole/trivago-cicd-pipeline-aws

Trivago uses Zealpath to find potential engineers to join their team. Zealpath is a website which hosts challenges that everyone can solve and submit, and with that apply for a job.

This is my first time using Zealpath and approach seems very practical. In worst case you learn about company’s technology stack (or some of it) and the way they think and solve problems. I have “applied” for the position AWS Cloud Engineer and 72 hours were given to submit the solution. My intention, honestly, was not to apply for Trivago job but to learn something new about automation and pipelines in AWS.

The case is described here. I am aware that they might remove the link at some point so I copied the text to the GitHub repository where the solution is.

Once you apply, the clock start ticking. You download a data.zip file and follow the instructions.

The confusion

The zip file itself is a bit confusing since all the files in the top directory appear to be in the two folders as well. I have removed all the duplicates from the home directory which left me with only README file.

The technology stack

The AWS services making up the pipeline are:

  • Athena
  • Cloudformation
  • Glue
  • S3

A DockerFile has been created to automate the provision of the pipeline.

The solution

My solution is in a GitHub repository. Hopefully it is well enough documented for anyone to understand it. It should be quite simple once you have an AWS account and Docker on Windows 10 installed. I have not tested it on Linux system.

All one needs to do is copy the DockerFile to a folder on a local machine, add a file called aws_cred.env and build the container.

But! Before all that is done, the variable s3_bucket in the Jupyter Notebook needs to be updated to the bucket name you plan to use. I really didn’t understand why the duplicates in the zip file. That is also the reason why I created the tar.gz file with the code from Zealpath’s zip file. I have also taken out the files I assume are duplicates.