Skip to content

StyleShare/airflow-ecs-codedeploy

 
 

Repository files navigation

Container Based Airflow Development Lifecycle

Requirements

  • AWS IAM

    • S3 Read
    • S3 Write (optional)
    • ECS Full (optional)
    • ECR Full (optional)
    • ElasticCache Full (optional)
  • Set environment variables

export AWS_ACCOUNT=123456789
export AWS_DEFAULT_REGION=ap-northeast-2-or-something-else
export AWS_ACCESS_KEY_ID=123456789
export AWS_SECRET_ACCESS_KEY=123456789
  • Docker
  • Docker Compose
  • Python 3.8.2

Local Environment

user >> shell script >> env from s3 (KMS Encrypted) >> docker-compose >> app start
  1. Local env upload to s3
python commands/write_env_file_command_line.py "$ENV_PATH"

Python Click으로 만들어진 CLI로 envfile을 작성하고, $ENV_PATH에 dotenv 파일을 만듭니다.

python commands/upload_env_to_s3_command_line.py \
  --bucket_name "$BUCKET_NAME" \
  --key_name "$KEY_NAME" \
  "$ENV_PATH"

$ENV_PATH의 dotenv 파일을 s3의 특정 위치에 업로드합니다. (기본 KMS 암호화)

or just run

sh scripts/update_env.sh $BUCKET_NAME $KEY_NAME $ENV_PATH
  1. Pull Environment Variables on docker-compose runtime
sh scripts/local-compose.sh $BUCKET_NAME $KEY_NAME $ENV_FILE

Staging Environment

[master-push] user >> docker build >> ECR push >> ecs update staging cluster (terraform)

Production Environment

[release-branch push] >> docker build >> ECR push >> ecs update production cluster (terraform

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • HCL 66.6%
  • Python 16.1%
  • Dockerfile 8.7%
  • Shell 6.8%
  • Makefile 1.8%