My Apache Airflow docker-compose file for running LocalExecutor with postgres using official production Dockerfile

docker-compose.yaml

x-airflow-environment: &airflow-environment
  environment:
    - HOST_HOME=${HOME}
    - HOST_PROJECT_PATH=${PWD}
  env_file: airflow.env
  image: feluelle/airflow:latest
x-airflow-volumes: &airflow-volumes
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - ~/.aws:/home/airflow/.aws
    - ../../1-orchestration:/opt/airflow/1-orchestration
    - ../../2-expectations:/opt/airflow/2-expectations

version: '3.8'
services:
  airflow-db:
    image: library/postgres:latest
    container_name: airflow_db
    env_file: postgres.env
    ports:
      - 35432:5432
    restart: always
  airflow-db-init:
    <<: *airflow-environment
    build:
      context: https://github.com/apache/airflow.git
      args:
        PYTHON_BASE_IMAGE: python:3.8-slim-buster
        PYTHON_MAJOR_MINOR_VERSION: 3.8
        AIRFLOW_EXTRAS: amazon,postgres,google,docker,virtualenv
    container_name: airflow_db_init
    command: db upgrade
    depends_on:
      - airflow-db
  airflow-users-init:
    <<: *airflow-environment
    container_name: airflow_users_init
    command: users create --role Admin --username airflow --email [email protected] --firstname Airflow --lastname Apache --password airflow
    depends_on:
      - airflow-db-init
  airflow-webserver:
    <<: *airflow-environment
    container_name: airflow_webserver
    command: webserver
    ports:
      - 38080:8080  # airflow webserver
      # TODO: Remove great_expectations port after moving from python dep to great_expectations docker image
      - 38888:8888  # great_expectations jupyter notebooks
    <<: *airflow-volumes
    depends_on:
      - airflow-db-init
    restart: always
  airflow-scheduler:
    <<: *airflow-environment
    container_name: airflow_scheduler
    command: scheduler
    <<: *airflow-volumes
    depends_on:
      - airflow-db-init
    restart: always

Additional notes:

• using Airflow (1-orchestration) for ELT with great_expectations (2-expectations) and dbt (3-transformations)
• using AWS System Manager - Parameter Store for Airflow variables and connections (using volume .aws folder to authenticate)
• using my edited requirements due to latest Airflow and Great Expectations incompatibilities of marshmallow dependency. (ge uses marshmallow>3.0 and airflow uses marshmallow==2.21.0)
• using dind - NOTE that you need to add permission to /var/run/docker.sock to be able to use it from inside the airflow containers.
• accessing dbt via Docker, for example:

task_dbt_test = DockerOperator(
    image='fishtownanalytics/dbt:0.17.0',
    environment=dict(
        DBT_PROFILES_DIR='/usr/.dbt'
    ),
    volumes=[
        f'{getenv("HOST_HOME")}/.dbt:/usr/.dbt',
        f'{getenv("HOST_ENVIRONMENT_PATH")}/3-transformations/examples/dbt_starter:/usr/app'
    ],
    network_mode='0-environment_default',
    auto_remove=True,
    command='test',
    task_id='dbt_test',
    dag=dag
)

In airflow.env I have also the following env variables:

AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
# TODO: Remove "GE_JUPYTER_CMD" env variable after moving from python dep to great_expectations docker image
GE_JUPYTER_CMD='jupyter notebook --ip 0.0.0.0'

The GE_JUPYTER_CMD is needed to get great_expecations docs working.