Airflow with k8s executor

This project is built on top of Apache Airflow on Kubernetes. The vision is operating airflow on k8s with:

• k8s executor
• operatior UX
  • uses kustomize
  • separate configuration for dev and production
  • improves log aggregation - to avoid all pods try to write to a single pvc airflow-logs
• security enhancements
  • picks official airflow container image - apache/airflow
  • k8s RBAC
  • security context for all pods
  • secrets management

Prerequisites

• kustomize CLI - see How to install

How to generate k8s objects:

$ kustomize build dev/

Design

It uses kustomize overlays to separate dev/prod configuration.

• development

  • in default namespace
  • service account is binding to clusteradmin
  • dags not in image
  • keeps logs in a pvc
  • uses local postgres db

• production

  • in a given namespace
  • has its own service account and role.
  • dags in custom build image
  • remote logging
  • remote db
  • no secrets in code