Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
MIT License
Setup to run Airflow in AWS ECS containers
pip install awscli
~/.aws/credentials
Generate a Fernet Key:
More about that here
Start Airflow locally simply running:
If everything runs correctly you can reach Airflow navigating to localhost:8080. The current setup is based on Celery Workers. You can monitor how many workers are currently active using Flower, visiting localhost:5555
To run Airflow in AWS we will use ECS (Elastic Container Service).
Run the following commands:
or alternatively
By default the infrastructure is deployed in us-east-1
.
When the infrastructure is provisioned (the RDS metadata DB will take a while) check the if the ECR repository is created then run:
By default the repo name created with terraform is airflow-dev
Without this command the ECS services will fail to fetch the latest
image from ECR
To deploy an update version of Airflow you need to push a new container image to ECR. You can simply doing that running:
The deployment script will take care of: