airflow-ecs

Setup to run Airflow in AWS ECS containers

Requirements

Local

• Docker

AWS

• AWS IAM User for the infrastructure deployment, with admin permissions
• awscli, intall running pip install awscli
• terraform >= 0.13
• setup your IAM User credentials inside ~/.aws/credentials
• setup these env variables in your .zshrc or .bashrc, or in your the terminal session that you are going to use

Local Development

• Generate a Fernet Key:

  More about that here

• Start Airflow locally simply running:

If everything runs correctly you can reach Airflow navigating to localhost:8080. The current setup is based on Celery Workers. You can monitor how many workers are currently active using Flower, visiting localhost:5555

Deploy Airflow on AWS ECS

To run Airflow in AWS we will use ECS (Elastic Container Service).

Deploy Infrastructure using Terraform

Run the following commands:

or alternatively

By default the infrastructure is deployed in us-east-1.

When the infrastructure is provisioned (the RDS metadata DB will take a while) check the if the ECR repository is created then run:

By default the repo name created with terraform is airflow-dev Without this command the ECS services will fail to fetch the latest image from ECR

Deploy new Airflow application

To deploy an update version of Airflow you need to push a new container image to ECR. You can simply doing that running:

The deployment script will take care of:

• push a new ECR image to your repository
• re-deploy the new ECS services with the updated image

TODO

• Create Private Subnets
• Move ECS containers to Private Subnets
• Use ECS private Links for Private Subnets
• Improve ECS Task and Service Role