AWS CLI

Hosting Drupal 9 on AWS Fargate Part 2: Bitbucket Pipeline

nigel — Sun, 17 Oct 2021 08:31:46 +0000

Hosting Drupal 9 on AWS Fargate Part 2: Bitbucket Pipeline nigel Sun, 17/10/2021 - 09:31

Introduction

In the first instalment of this sequence of blogs we talked about how imperative it is to create an industrial strength local dev environment when AWS Fargate is your target deployment platform. In this blog we discuss something equally important - the deploy and build process. This needs to be automated for Drupal since security patches are issued on a weekly basis by the Drupal Security Group, and an automated solution is the only way to keep on top of these updates.

Over the last two years I have become accustomed to using Atlassian's Bitbucket Pipelines, and I am really liking this CI/CD delivery mechanism. It is particularly suited to containerised solutions since the pipeline itself runs in a Docker container, and often times the solution Docker image can be used for the build process - case in point this project! This tutorial will show how the official Drupal PHP Docker container can be used as the pipeline for delivering Drupal 9 projects to AWS Fargate. Let's see how!

Bitbucket Pipeline

The diagram above shows the basic steps the pipeline follows. Let's have a look at these in turn with narrative and then see how that transposes into the Pipeline code.A few trivial housekeeping activities have been omitted from the diagram for clarity but will be discussed when we look at the code.

Since my Badzilla blog is a hobby project, I don't have multiple environments, and merge all new features into the git master branch. Any merge into master triggers the build process. Commercial pipelines tend to trigger builds on feature branch commits too which may be a consideration depending upon your circumstances.
I've already mentioned that the Docker image selected for the pipeline will be the official Drupal / PHP release - and thus contains composer which is used to build out the codebase.
Once the codebase has been built the testing can be undertaken. I have identified two parallel steps in the pipeline - automated testing and static testing. I have covered the configuration for these in earlier blogs - so click through for further reading.
The next steps involve building the two Docker image artefacts we need for our Fargate bundles - the PHP / FPM / Codebase image, and the NGINX Web Server image. The latter is required since we need to load a configuration for Drupal 9 and for serving the S3 / CloudFront assets.
The artefacts are pushed into AWS ECR after being tagged latest.
We are now ready to deploy the images to our ECS Cluster - but before we do this, we provide the opportunity to run any pre-deploy hooks. This would typically be activities such as drush putting the web app into maintenance mode to prevent public access during updates.
One the pre-deploy scripts have completed, the deploy script is invoked. This forces an ECS service update on the cluster. The script loops to determine whether the update fails or completes, and upon a successful completion will move onto the final step.
The post-deploy scripts are now run. These would be further drush commands to update the database schema for example, and to remove the maintenance mode flag.

Pipeline Code

The pipeline code is saved in bitbucket-pipelines.yml and is show below.

image:
  name: drupal:9.2.6-php7.4-fpm
 
clone:
  depth: full
 
definitions:
  services:
    docker:
      memory: 2048
 
  steps:
    - step: &build-environment
        name: Build Environment
        script:
          # Install git & unzip for composer
          - apt-get update
          - apt-get install git unzip -y
          # Install dependencies
          - export COMPOSER_ALLOW_SUPERUSER=1
          - composer install
          # Copy project settings file in place
          - cp storage/settings.php web/sites/default/settings.php
          # Remove all non-essential dirs and files from top level directory
          - rm -rf assets backup databases .git
          - rm Makefile README.md docker-compose.yml composer.json composer.lock .env .csslintrc .eslintignore .eslintrc.json
          - rm .gitignore
          # Add the bind mount VOLUME so in Fargate the nginx container can do a VOLUME_FROM and load /opt/drupal docroot
          # from the core-drupal container
          - sed -i '/^FROM.*/a VOLUME ["/opt/drupal"]' Dockerfile > /dev/null 2>&1
        caches:
          - composer
        artifacts:
          - '**'
 
    # Automated Tests placeholder
    - step: &automated-tests
        name: Automated Tests
        script:
          - echo "Add automated tests here"
 
    # Static Analysis tests placeholder
    - step: &static-tests
        name: Static Analysis Tests
        script:
          - echo "Add static analysis here"
 
pipelines:
  # Run when code is merged / pushed into specific branches.
  branches:
    master:
      - step: *build-environment
 
      - parallel:
          - step: *automated-tests
          - step: *static-tests
 
      - step:
          name: Build & Deploy
          deployment: AWS Prod
          script:
            # Install AWS CLI
            - apt update && apt-get install -y jq unzip python3 python-dev python3-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev python-pip
            - curl "https://s3.amazonaws.com/aws-cli/awscli-bundle-1.18.200.zip" -o "awscli-bundle.zip"
            - unzip awscli-bundle.zip
            - ./awscli-bundle/install -b ~/bin/aws
            - export PATH=~/bin:$PATH
            # Build the codebase + FPM
            - docker build -t badzilla-core-drupal .
            # Authenticate with ECR
            - aws configure set aws_access_key_id "${AWS_ACCESS_KEY}"
            - aws configure set aws_secret_access_key "${AWS_SECRET_KEY}"
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${CORE_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-core-drupal:latest "${CORE_DRUPAL_REPO_URL}"
            - docker push "${CORE_DRUPAL_REPO_URL}"
            # NGINX_DRUPAL_REPO_URL
            - cd docker/nginx/fargate
            - docker build -t badzilla-nginx-drupal .
            - cd -
            # Authenticate with ECR
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${NGINX_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-nginx-drupal:latest "${NGINX_DRUPAL_REPO_URL}"
            - docker push "${NGINX_DRUPAL_REPO_URL}"
            # Pre deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/pre-deploy/
            - cd -
            # Deploy script
            - ./deploy-scripts/ecs-service-deploy.sh
            # Post deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/post-deploy/
            - cd -
          services:
            - docker
          caches:
            - docker

Now let's inspect the code per section. Firstly the declaration of the Docker image to use for the pipeline.

image:
  name: drupal:9.2.6-php7.4-fpm

You'll note that I'm using the same image as I used for my development environment with one small change. In the pipeline I have abandoned the use of the ultra lightweight Alpine image - the image being used is standard Ubuntu. Why so? Firstly there is less necessity for keeping the image small since Atlassian don't charge for storage of the image, and secondly I will want to add additional Linux packages to this image for the pipeline process, and thus I want to use a package manager I am familiar with - apt.

clone:
  depth: full
 
definitions:
  services:
    docker:
      memory: 2048

Here I'm saying I want a full clone of the repository in the pipeline. This is a little unnecessary and causes an overhead. In the future I will consider a more shallow clone for greater performance. The Docker memory setting should provide enough headroom for Drupal builds.

  steps:
    - step: &build-environment
        name: Build Environment
        script:
          # Install git & unzip for composer
          - apt-get update
          - apt-get install git unzip -y
          # Install dependencies
          - export COMPOSER_ALLOW_SUPERUSER=1
          - composer install

Steps are defined in the pipeline that are invoked later in the script. This step &build-environment will build out the environment by installing the packages I will need and then running composer install to build out the codebase.

          # Copy project settings file in place
          - cp storage/settings.php web/sites/default/settings.php
          # Remove all non-essential dirs and files from top level directory
          - rm -rf assets backup databases .git
          - rm Makefile README.md docker-compose.yml composer.json composer.lock .env .csslintrc .eslintignore .eslintrc.json
          - rm .gitignore

Every Drupal site needs a settings.php file. I decided to have mine saved in the repository in the directory storage since there is no confidential data. All settings use environmental variables. Your mileage may vary here - other strategies are possible for injecting the settings.php into the build.

I then delete files and directories from the build which won't be needed in a production environment.

          # Add the bind mount VOLUME so in Fargate the nginx container can do a VOLUME_FROM and load /opt/drupal docroot
          # from the core-drupal container
          - sed -i '/^FROM.*/a VOLUME ["/opt/drupal"]' Dockerfile > /dev/null 2>&1

This snippet is a kludge to fix a difference between the local environment docker-compose and the production Fargate Task Definition. Fargate shared volumes use bind mounts which are created by initially using the VOLUME instruction in the Dockerfile and settings in the Task Definition (to be discussed in a later blog). Here I use interactive sed to inject the VOLUME into the Dockerfile

        caches:
          - composer
        artifacts:
          - '**'

Closing the build step I ensure that Docker is cached to save time next time the pipeline is invoked and the artifact (American spelling) setting tells the pipeline to save the all my work in that pipeline step for steps that follow it.

    # Automated Tests placeholder
    - step: &automated-tests
        name: Automated Tests
        script:
          - echo "Add automated tests here"
 
    # Static Analysis tests placeholder
    - step: &static-tests
        name: Static Analysis Tests
        script:
          - echo "Add static analysis here"
 
pipelines:
  # Run when code is merged / pushed into specific branches.
  branches:
    master:
      - step: *build-environment
 
      - parallel:
          - step: *automated-tests
          - step: *static-tests

Here I have defined the testing steps (both of which are currently placeholders), and then started the 'implementation' section of the code. The steps will be invoked on the master branch, initially the build and then the the two testing steps in parallel.

      - step:
          name: Build & Deploy
          deployment: AWS Prod
          script:
            # Install AWS CLI
            - apt update && apt-get install -y jq unzip python3 python-dev python3-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev python-pip
            - curl "https://s3.amazonaws.com/aws-cli/awscli-bundle-1.18.200.zip" -o "awscli-bundle.zip"
            - unzip awscli-bundle.zip
            - ./awscli-bundle/install -b ~/bin/aws
            - export PATH=~/bin:$PATH

Once testing has completed successfully we commence the inline deploy step. We are going to need the AWS CLI installing here since the CLI is necessary to issue the correct sequence of commands to AWS to commit and deploy.

Note I am also installing the amazing utility jq - a JSON query language utility. This is imperative for inspecting the output of the CLI commands since their responses are sent in JSON format.

            # Build the codebase + FPM
            - docker build -t badzilla-core-drupal .
            # Authenticate with ECR
            - aws configure set aws_access_key_id "${AWS_ACCESS_KEY}"
            - aws configure set aws_secret_access_key "${AWS_SECRET_KEY}"
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${CORE_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-core-drupal:latest "${CORE_DRUPAL_REPO_URL}"
            - docker push "${CORE_DRUPAL_REPO_URL}"

Now I've got all I need installed I can build the Docker image artefacts. Firstly I build out the Drupal codebase image which also contains PHP and FPM. I then authenticate with AWS ECR where I will push the artefact once its been tagged.

            # NGINX_DRUPAL_REPO_URL
            - cd docker/nginx/fargate
            - docker build -t badzilla-nginx-drupal .
            - cd -
            # Authenticate with ECR
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${NGINX_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-nginx-drupal:latest "${NGINX_DRUPAL_REPO_URL}"
            - docker push "${NGINX_DRUPAL_REPO_URL}"

I now repeat the process for the NGINX web server Docker image. Note I have the configuration for this in my codebase repo under the directory docker/nginx/fargate

            # Pre deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/pre-deploy/
            - cd -
            # Deploy script
            - ./deploy-scripts/ecs-service-deploy.sh
            # Post deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/post-deploy/
            - cd -

This is the fun part! I now run the deploy shell scripts which are listed with narratives later in the blog. Firstly I run the pre-deploy commands, then the deploy commands and the post-deploy commands. All will be revealed...

          services:
            - docker
          caches:
            - docker

To use Docker commands in a pipeline, you need to list Docker under the services setting. Installing Docker in the pipeline itself and then invoking it will not work! I tried and failed! Note I am also using caching for greater performance.

Repository Environmental Variables

The pipeline and the shell scripts heavily use environmental variables that are set in the repository in Bitbucket. Their names are self-evident and listed in their entirety above.

Pre- and Post-deploy shell script

The pre- and post-deploy shell script is defined below. Currently it only works on drush commands but could be extended to include shell scripts easily enough. The idea is it scans the directory passed as a parameter and it looks for lines in any file that starts with the word drush. It then takes the runtime arguments of that drush command and converts it into a Docker cmd syntax. Once that is done the command is executed as a Fargate task using the codebase image with parameter overrides. This task runs to completion and exits. The status of the command can be captured and reported back to the pipeline. Obviously this isn't as quick as running a drush command in a terminal shell since the Fargate task needs to be provisioned which takes time. It is however a fiendish way of reusing an existing Docker image since the official Drupal Docker image already contains the drush command.
deploy-scripts/deploy-parser.sh

#!/bin/bash
 
# Parse the passed files and issue commands in accordance.
# Currently only supports drush commands.
# This can be expanded out to other shell scripts.
 
# Output directory to be parsed
echo "Directory: $1 to be parsed"
 
set_network_configuration()
{
NETWORK=$(cat <
)
echo "${NETWORK}"
}
 
 
 
set_overrides_command()
{
OVERRIDES=$(cat <
)
echo "${OVERRIDES}"
}
 
 
 
# Process each line in a selected file
parse_line()
{
    # All lines to be processed must start with the word drush to be considered
    # This grep / sed combination will turn the drush command into docker cmd syntax
    grep -e "^drush" < "${1}" | while read -r line ; do
        cmd=`echo $line | sed -r 's/[^ ][^ ]*/"&"/g' | sed -r 's/\s+/,/g' | sed -r 's/drush/vendor\/bin\/drush/'`
 
        # @BUG with quoting in dotenv and how I load into environment. Appears to add newlines making it difficult to add quotes later.
        # So already added quotes for $CORE_TASK_DEFINITION_NAME in .env file but need the unquoted version in
        # --task-definition flag so unquote here
        TASK_DEF_UNQUOTE=`echo ${CORE_TASK_DEFINITION_NAME} | sed s/\"//g`
 
        # Now execute the command to start the container with the drush cmd
        FARGATE_TASK=`aws ecs run-task \
            --cluster "${ECS_CLUSTER_NAME}" \
            --task-definition "${TASK_DEF_UNQUOTE}" \
            --network-configuration "$(set_network_configuration  ${ECS_SUBNET_1} ${ECS_SUBNET_2}  ${ECS_SECURITY_GROUP})" \
            --launch-type FARGATE \
            --platform-version '1.4.0' \
            --region "${AWS_DEFAULT_REGION}" \
            --overrides "$(set_overrides_command ${CORE_TASK_DEFINITION_NAME} ${cmd})" | jq .tasks[0].taskArn -r`
 
        if [[ $? != 0 ]]; then
            echo "Fargate Run Task Failed."
            exit $?
        elif [[ -z  "${FARGATE_TASK}" ]]; then
            echo "Unspecified Fargate Run Task Failure."
            exit 1
        else
            echo "Cmd run is ${cmd}"
        fi
 
        # Loop round to check on the status of the task. Can't run in parallel so have to wait for each task to finish
        while true; do
            FARGATE_STATUS=`aws ecs describe-tasks --cluster "${ECS_CLUSTER_NAME}" --tasks "${FARGATE_TASK}"  | jq .tasks[0].lastStatus -r`
 
            if [[ -z "${FARGATE_STATUS}" ]]; then
                echo "Unspecified Fargate Describe Task Failure."
                exit 1
            fi
 
            echo "${FARGATE_STATUS}"
 
            if [[ "${FARGATE_STATUS}" == "STOPPED" ]] || [[ "${FARGATE_STATUS}" == "DEPROVISIONING" ]]; then
                  break
            fi
 
            sleep 20
        done
 
	echo "Drush command completed"
    done
}
 
 
# Get a list of all the files in the directory and process
parse_directory()
{
    # loop though all the files in the directory
    for file in "${1}"/*; do
        base="$(basename ${file})"
        echo "${base} being parsed"
        parse_line "${file}"
    done
}
 
parse_directory "${1}"

ECS Deploy Script

The main ECS deploy shell script uses AWS CLI ecs update-service to force a new deployment. This means that when the Fargate bundles are recycling any new codebase and NGINX Docker image will be used from AWS ECR. The process can take a few minutes so I have added a loop facility which uses AWS CLI ecs describe-services to capture the current deployment status and act accordingly. Note I am using the previously mentioned jq utility to interrogate the long JSON response from the CLI and whittle it down to the status field which should contain the word "COMPLETED" or "FAILED" or "IN_PROGRESS".
deploy-scripts/ecs-service-deploy.sh

#!/bin/bash
 
 
# Use the aws cli to force deploy Fargate task running in ECS service
aws ecs update-service --cluster "${ECS_CLUSTER_NAME}" --service "${ECS_SERVICE_NAME}" --force-new-deployment --region "${AWS_DEFAULT_REGION}"
 
# Give it a few seconds to take effect
sleep 5
 
# Get the deployment status by looping over the deployment
# Try for a maximum of 20 minutes then give up. Chances are after 20 minutes of incomplete
# deployment there is a problem in the task definition
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
do
    if [[ "${i}" -eq 21 ]]; then
        echo "Timed out before could determine if deployment succeeded"
        exit 1
    fi
 
    STATUS=`aws ecs describe-services --cluster "${ECS_CLUSTER_NAME}" --services "${ECS_SERVICE_NAME}" --region "${AWS_DEFAULT_REGION}" | jq .services[0].deployments[0].rolloutState -r`
 
    # Was it successful?
    if [[ "${STATUS}" = "COMPLETED"  ]]; then
        echo "Deployment succeeded"
        exit 0
    # Anything we need to be concerned about?
    elif [[ "${STATUS}" = "FAILED"  ]]; then
        echo "Deployment failed"
        exit 2
    # If still deploying echo this
    elif [[ "${STATUS}" = "IN_PROGRESS"  ]]; then
        echo "Deployment in progress"
    else
        echo "Unknown status "${STATUS}" so quitting"
        exit 3
    fi
 
    # Wait a minute before retrying
    sleep 60
done

Ad hoc Drush Commands

The solution provides the capability to run drush during the pre- and post-deploy scripts. But what about running drush commands on an ad hoc basis from a remote laptop? This is often required by developers or devops and would be crucial for the management of a Drupal 9 website. Stay tuned for a future blog on how to achieve this!

blog terms

AWS AWS CLI devops PHP Drupal Drupal 9

Hosting Drupal 9 on AWS Fargate Part 1: Local Development

nigel — Sun, 19 Sep 2021 08:46:14 +0000

Hosting Drupal 9 on AWS Fargate Part 1: Local Development nigel Sun, 19/09/2021 - 09:46

Introduction

Welcome to a series of blogs on hosting Drupal 9 (or legacy Drupal 8) sites on AWS Fargate. Fargate can now be considered a mature product, and provides a simple serverless containerised platform for hosting web apps. It offers considerable savings over traditional Virtual servers without, if configured correctly, loss of performance. Furthermore it offers greater simplicity of operation and setup then its AWS sibling offering ECS over EC2, and again a cost saving.

The blogs in this series will cover important considerations such as the pipelines to deploy your codebase within Docker images, and their storage; set up of the Drupal S3FS module along with the CloudFront accelerator and S3 object store; auto scaling of the Fargate tasks; load testing your app to ensure you are fully optimised; and how to run your Drush commands in short living Fargate tasks.

This first blog will cover your team's local environment and how its build will ensure your path to Fargate deploy is smooth, with as much reuse of the configuration in production as possible.

Objective

The objective is to stand up a Fargate task running as a service which will be the web app, and it will persist and auto scale. Also have Fargate tasks to run which will run Drush commands, stood up on a per Drush command basis, and terminating after each Drush command completes. The diagram above shows the Docker images requires to achieve our goal.

Drupal Core + FPM PHP + Codebase

The intention is to use the official Drupal FPM Docker image using Alpine, and be as up to date as possible with the versioning. Note that the Alpine build has been chosen since it's an extremely lightweight version of Linux, based on busybox which has a very small footprint. This comes at a price - those with Ubuntu experience will have to learn new commands. For instance Alpine uses the apk package manager instead of apt-get. The web app's codebase - ie all the configuration and custom modules and themes created by the dev team - will be copied into this image.

Drupal Core + FPM PHP + Codebase + Exec command

This will be the Fargate task to run one off Drush commands. Since the codebase will already contain Drush binaries in vendor/bin/drush, it will be the same as the previously mentioned image. Notwithstanding this, it won't persist. It will use the Fargate equivalent of the Docker command to exec a particular command and then quit.

Memcached

I have elected to run individual Memcached caching in each task running in the Web App service. Memcached will be used for caching the Drupal MySQL cache tables and for page caching for anonymous users. There is an alternative strategy to this - centralise the caching using for instance the AWS ElastiCache with Memcached product. Each task in the web service would have an endpoint to the same centralised ElastiCache instance. This would be the preferred option if your website is largely serving logged in traffic, and if you are running an eCommerce site with the users' baskets stored in Memcached. Since that doesn't apply to my own site, and I would like to keep costs down too, my feeling is having Memcached as part of the Fargate task is a better fit.

nginx

A web server is needed, and I've elected for nginx although Apache would also be a good choice. My rationale for selecting nginx is I would like to add the feature of nginx serving slightly stale cached pages when the origin is returning 5xx errors. This would give a seamless switch-over when a task becomes unhealthy - during the period AWS determines that a task is returning errors and has crossed the unhealthy threshold, nginx could still serve pages which would be less than 5 minutes old. The task would then be drained and removed from the pool with no problem noticeable to the end user. This will be the subject of a blog at a later date.

Using a local Virtual Machine

Those who know me well will know I am not an advocate of using my laptop's OS for development purposes. The reason being that with multiple clients and with multiple projects in each, my host OS would quickly become a mess of different development tooling and versions which could collide and could also generate inconsistencies with other devs in the team should say different versions of bin files end up in the $PATH.

My ethos has always been do not pollute your host OS! Always use VMs for development!

I have created my own VM for Docker based implementations. My belief is if the target deploy is using Docker containers then the local development environment should match this as much as is practicable. In fact it's impossible to match the ECS/Fargate environment exactly, but it is certainly possible to use Docker with docker-compose to simulate closely what will be seen in prod.

My VM can be found in GitHub by clicking on this link. It is a very lightweight VM and can be orchestrated easily using Vagrant and Ansible. PHP and MySQL are deliberately not included in the playbook since all development work will be undertaken by using Docker containers.

All my localhost config examples in this blog will have come from using my DockerVM mentioned above. If you intend to follow this blog closely, you will need to install it, but read the instructions in the README.md carefully!

Composer

Once you've got a lightweight VM up and running (hopefully mine), and ssh'd into it, and cloned your repo, you will need to build out the dependencies using composer install. So how do we go about this when there is no composer and no PHP in the VM? There are two choices:

Use the official Docker Composer image
Use the latest official Docker Drupal image (9.2.6-php7.4-fpm-alpine.3.14 at time of writing this blog)

I chose the official Docker Composer image purely because (a) there was more documentation on it readily available, and (b) I wasn't sure at the time whether the Drupal image bundled Composer. It does, and it will be used in the Bitbucket Pipeline later so you may want to select that instead for consistency, but if you do, you'll have to change my composer commands slightly.

Ok so let's build our composer image.

$ docker run --rm --interactive --tty --volume $PWD:/app composer install --ignore-platform-reqs
Unable to find image 'composer:latest' locally
latest: Pulling from library/composer
a0d0a0d46f8b: Pull complete 
153eea49496a: Pull complete 
11efd0df1fcb: Pull complete 
b3f3214c344d: Pull complete 
9abd2f85688c: Pull complete 
83d85b95eb4c: Pull complete 
923d73ddadfa: Pull complete 
711b5c4b02a7: Pull complete 
ee08fa481788: Pull complete 
5f9f812bf61d: Pull complete 
28b3ecec72d5: Pull complete 
a5ffb14bc4f9: Pull complete 
d92aff3e5aec: Pull complete 
cd0274c84e75: Pull complete 
Digest: sha256:33d2c1d9fb7311cb8f215a8f52669894ef534292ac5d331347f929c27e317bb1
Status: Downloaded newer image for composer:latest
> DrupalProject\composer\ScriptHandler::checkComposerVersion
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
  - Downloading composer/installers (v2.0.1)
<---- SNIPPED ---->
  - Copy [web-root]/profiles/README.txt from assets/scaffold/files/profiles.README.txt
  - Copy [web-root]/themes/README.txt from assets/scaffold/files/themes.README.txt
> DrupalProject\composer\ScriptHandler::createRequiredFiles
Created a sites/default/files directory with chmod 0777

Success! Note that there is a requirement to use the composer flag --ignore-platform-reqs since it expects ext-gd locally which obviously we don't have.

docker-compose: Codebase + FPM

In preparation for creating a docker-compose file we'll need to build out an image containing the latest Drupal 9 Docker image + our codebase once composer install has been run. This will need a Dockerfile, and we can add APCu caching in the Dockerfile for greater Drupal performance. I also noted that Memcached API library needs to be added to Dockerfile since it is my intention to connect to a Memcached server for cache management in the container bundle. The Dockerfile is listed below.

FROM drupal:9.2.6-php7.4-fpm-alpine3.14
 
# Copy the artefact (on Bitbucket) or local files into the image
COPY ./ /opt/drupal/.
 
# PHP Memcached which needs the libmemcached API
# Bug in Apline - creates extensions in the wrong directory so add them in correct location
RUN apk add php7-igbinary php7-pecl-memcached libmemcached \
    && echo "extension=memcached.so" > /usr/local/etc/php/conf.d/docker-php-ext-memcached.ini \
    && echo "extension=igbinary.so" > /usr/local/etc/php/conf.d/docker-php-ext-igbinary.ini \
    && mv /usr/lib/php7/modules/memcached.so /usr/local/lib/php/extensions/no-debug-non-zts-20190902 \
    && mv /usr/lib/php7/modules/igbinary.so /usr/local/lib/php/extensions/no-debug-non-zts-20190902
 
# APCu cache
RUN apk add --update --no-cache --virtual .build-dependencies $PHPIZE_DEPS \
    && pecl install apcu \
    && docker-php-ext-enable apcu \
    && pecl clear-cache \
    && apk del .build-dependencies

docker-compose: nginx image

We will also need to build out an nginx image with the standard Drupal configuration. To do this I created a docker directory at the top level of the repo (i.e. a sibling of web directory) as such:

$ tree docker
docker
├── core-drupal
├── memcached
└── nginx
    ├── fargate
    │   ├── default.conf
    │   ├── Dockerfile
    │   └── nginx.conf
    └── local
        ├── default.conf
        ├── Dockerfile
        └── nginx.conf
 
5 directories, 6 files

You'll note that I have two sets of config files - one for AWS Fargate and one for local development. In fact the differences are minimal and will be discussed later. The local files are below
nginx.conf

user  nginx;
worker_processes  auto;
 
error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
 
 
events {
    worker_connections  1024;
}
 
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    fastcgi_buffers 16 16k;
    fastcgi_buffer_size 32k;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    #tcp_nopush     on;
 
    keepalive_timeout  65;
 
    #gzip  on;
 
    include /etc/nginx/conf.d/*.conf;
}

default.conf

  server {
    server_name badzilla-fargate.test;
    root /opt/drupal/web; ## <-- Your only path reference.
 
    location = /favicon.ico {
      log_not_found off;
      access_log off;
    }
 
    location = /robots.txt {
      allow all;
      log_not_found off;
      access_log off;
    }
 
    location ~ \..*/.*\.php$ {
      return 403;
    }
 
    location ~ ^/sites/.*/private/ {
      return 403;
    }
 
    # Block access to scripts in site files directory
    location ~ ^/sites/[^/]+/files/.*\.php$ {
      deny all;
    }
 
    # Allow "Well-Known URIs" as per RFC 5785
    location ~* ^/.well-known/ {
      allow all;
    }
 
    # Block access to "hidden" files and directories whose names begin with a
    # period. This includes directories used by version control systems such
    # as Subversion or Git to store control files.
    location ~ (^|/)\. {
      return 403;
    }
 
    location / {
      # try_files $uri @rewrite; # For Drupal <= 6
      try_files $uri /index.php?$query_string; # For Drupal >= 7
    }
 
    location @rewrite {
      #rewrite ^/(.*)$ /index.php?q=$1; # For Drupal <= 6
      rewrite ^ /index.php; # For Drupal >= 7
    }
 
    # Don't allow direct access to PHP files in the vendor directory.
    location ~ /vendor/.*\.php$ {
      deny all;
      return 404;
    }
 
    # Protect files and directories from prying eyes.
    location ~* \.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|/(\.(?!well-known).*)|Entries.*|Root|/#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$ {
      deny all;
      return 404;
    }
 
    # In Drupal 8, we must also match new paths where the '.php' appears in
    # the middle, such as update.php/selection. The rule we use is strict,
    # and only allows this pattern with the update.php front controller.
    # This allows legacy path aliases in the form of
    # blog/index.php/legacy-path to continue to route to Drupal nodes. If
    # you do not have any paths like that, then you might prefer to use a
    # laxer rule, such as:
    #   location ~ \.php(/|$) {
    # The laxer rule will continue to work if Drupal uses this new URL
    # pattern with front controllers other than update.php in a future
    # release.
    location ~ '\.php$|^/update.php' {
      fastcgi_split_path_info ^(.+?\.php)(|/.*)$;
      # Ensure the php file exists. Mitigates CVE-2019-11043
      try_files $fastcgi_script_name =404;
      # Security note: If you're running a version of PHP older than the
      # latest 5.3, you should have "cgi.fix_pathinfo = 0;" in php.ini.
      # See http://serverfault.com/q/627903/94922 for details.
      include fastcgi_params;
      # Block httpoxy attacks. See https://httpoxy.org/.
      fastcgi_param HTTP_PROXY "";
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
      fastcgi_param QUERY_STRING $query_string;
      fastcgi_intercept_errors on;
      # PHP 5 socket location.
      #fastcgi_pass unix:/var/run/php5-fpm.sock;
      # PHP 7 socket location.
#      fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
      fastcgi_pass fpm:9000;
    }
 
    # S3FS Module
    location ~ ^/s3/files/styles/ {
      try_files $uri @rewrite;
    }
 
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg)$ {
      try_files $uri @rewrite;
      expires max;
      log_not_found off;
    }
 
    # Fighting with Styles? This little gem is amazing.
    # location ~ ^/sites/.*/files/imagecache/ { # For Drupal <= 6
    #location ~ ^/sites/.*/files/styles/ { # For Drupal >= 7
    #  try_files $uri @rewrite;
    #}
 
    # Handle private files through Drupal. Private file's path can come
    # with a language prefix.
    location ~ ^(/[a-z\-]+)?/system/files/ { # For Drupal >= 7
      try_files $uri /index.php?$query_string;
    }
 
    # Enforce clean URLs
    # Removes index.php from urls like www.example.com/index.php/my-page --> www.example.com/my-page
    # Could be done with 301 for permanent or other redirect codes.
    if ($request_uri ~* "^(.*/)index\.php/(.*)") {
      return 307 $1$2;
    }
  }

Dockerfile

FROM nginx:1.21.1-alpine
COPY ./default.conf /etc/nginx/conf.d/
COPY ./nginx.conf /etc/nginx/

Some commentary is required here. You'll see in the default.conf file there is a snippet:

    # S3FS Module
    location ~ ^/s3/files/styles/ {
      try_files $uri @rewrite;
    }

This is because the assets are not served locally - they are stored in an AWS S3 bucket with the AWS CloudFront accelerator providing edge capabilities. I will provide a further blog at a later date on setting all this up but in essence, assets shouldn't be kept in individual Fargate tasks, so either an AWS S3 / CloudFront solution needs to be utilised or an AWS EFS drive.

Now let's see what the differences are between the local configuration and Fargate.

$ diff docker/nginx/local/default.conf docker/nginx/fargate/default.conf 
2c2
<     server_name badzilla-fargate.test;
---
>     server_name badzilla.co.uk www.badzilla.co.uk;
4a5,10
>     # User Agent ELB-HealthChecker sends no headers with the request so nginx by default will respond with a 400
>     # Fix with this
>     location = /health {
>         return 200;
>     }
> 
92c98
<       fastcgi_pass fpm:9000;
---
>       fastcgi_pass 127.0.0.1:9000;

Ok there are three differences. The first is obvious - I am using different server names dependent on the environment. The second difference is the AWS health checking capability that I have included for Fargate but not needed locally. I use the Drupal health_check module by the way, but the problem resides with the AWS Application Load Balancer health check feature. It sends requests to nginx without headers which nginx doesn't like - and always replies with a 400 response. This snippet fixes that problem. The third difference relates to how Fargate handles Docker networking. It is much simplified and all containers in the Fargate task use 127.0.0.1 for communication. Locally I am using a label named fpm for fastcgi_pass communication between nginx and FPM.

docker-compose: MariaDB and Memcached

The use of MariaDB is the most significant difference between the local environment and Fargate. Fargate connects to a MariaDB instance in AWS RDS, whilst the local MariaDB persists its data in the VM. I did consider using another RDS instance for local development, but that would require public access to the database which I wasn't keen to do for security reasons. In addition there is a cost implication which isn't viable for a hobby project like my blog. I am using the standard Docker images for both MariaDB and Memcached locally.

docker-compose Configuration

docker-compose

version: "3"
 
services:
  mariadb:
    image: "mariadb:${MARIADB_VERSION}"
    restart: 'always'
    container_name: mariadb-drupal
    volumes:
      - "/var/lib/mysql/data:${MARIADB_DATA_DIR}"
      - "/var/lib/mysql/logs:${MARIADB_LOG_DIR}"
      - "/var/docker/mariadb/conf:/etc/mysql"
    environment:
      MYSQL_ROOT_PASSWORD: "${MYSQL_ROOT_PASSWORD}"
      MYSQL_DATABASE: "${MYSQL_DATABASE}"
      MYSQL_USER: "${MYSQL_USER}"
      MYSQL_PASSWORD: "${MYSQL_PASSWORD}"
  drupal:
    image: core-drupal:latest
    container_name: core-drupal
    depends_on:
      - mariadb
      - memcached
    volumes:
      - ./:/opt/drupal
    restart: 'always'
    environment:
      MYSQL_DATABASE: "${MYSQL_DATABASE}"
      MYSQL_USER: "${MYSQL_USER}"
      MYSQL_PASSWORD: "${MYSQL_PASSWORD}"
      MYSQL_HOSTNAME: "${MYSQL_HOSTNAME}"
      MYSQL_NAMESPACE: "${MYSQL_NAMESPACE}"
      MYSQL_DRIVER: "${MYSQL_DRIVER}"
      MYSQL_PORT: "${MYSQL_PORT}"
      AH_SITE_ENVIRONMENT: "${AH_SITE_ENVIRONMENT}"
      S3FS_ACCESS_KEY: "${S3FS_ACCESS_KEY}"
      S3FS_SECRET_KEY: "${S3FS_SECRET_KEY}"
      S3FS_BUCKET: "${S3FS_BUCKET}"
      S3FS_AWS_REGION: "${S3FS_AWS_REGION}"
      S3FS_CLOUDFRONT_CNAME: "${S3FS_CLOUDFRONT_CNAME}"
      S3FS_USE_S3_PUBLIC: "${S3FS_USE_S3_PUBLIC}"
      S3FS_USE_CNAME: "${S3FS_USE_CNAME}"
      S3FS_USE_HTTPS: "${S3FS_USE_HTTPS}"
      S3FS_TWIG_PATH: "${S3FS_TWIG_PATH}"
 
  nginx:
    image: nginx-drupal:latest
    container_name: nginx-drupal
    links:
      - drupal:fpm
    ports:
      - "80:80"
      - "443:443"
    depends_on:
      - drupal
    restart: 'always'
    volumes:
      - ./:/opt/drupal
  memcached:
    image: memcached:alpine3.14
    restart: 'always'
    container_name: memcached-drupal
    ports:
      - "11211:11211"

Note I have all my settings held in the environment. I also use the vlucas/phpdotenv composer dependency so these environmental variables are accessible in Drupal through a dot env file.

Shortcuts

Remembering commands to build the Docker images, gain command line access to MariaDB, importing and exporting databases etc is an exercise in tedium I'd rather avoid. I have built a make file with shortcuts although command line aliases could also have been used.
Makefile

.DEFAULT_GOAL := help
 
 
help:
	@echo "See the README.md for available options"
 
exportdb: set-env
	@echo "Exporting the database"
	docker exec -i mariadb-drupal bash -c 'mysqldump -u $$MYSQL_USER -p$$MYSQL_PASSWORD $$MYSQL_DATABASE' > $(file)
 
importdb: set-env
	@echo "Importing the database"
	docker exec -i mariadb-drupal bash -c 'mysql -u $$MYSQL_USER -p$$MYSQL_PASSWORD $$MYSQL_DATABASE' < $(file)
 
core-drupal: set-env
	@echo "Building Drupal Core Docker from Official Image"
	docker build -t core-drupal .
 
composer-require:
	@echo "Composer require running"
	docker run --rm --interactive --tty --volume $$PWD:/app  --volume $${COMPOSER_HOME:-$$HOME/.composer}:/tmp composer require "$(module)"  --ignore-platform-reqs
 
build-all: set-env
	@echo "Building all non-standard docker images ready for spinning up docker-compose"
	cd docker/nginx/local && docker build -t nginx-drupal .
	docker build -t core-drupal .
 
start-web: set-env
	docker-compose up
 
stop-web:
	@echo "Stopping the web app gracefully. Please wait"
	docker-compose down
 
set-env:
	@echo "Setting the env variables"
	export $(cat .env | grep -v '^#' | grep '\S' | tr '\r' '\0' | xargs -0 -n1)

So for instance should I want to build the codebase and nginx Docker images I would simply type:

make build-all

To add a new module using composer, the syntax would be:

make composer-require module=drupal/module_name

Note the requirement to use key=value for parameter substitution in make. Similarly to import a database dump it would be:

make importdb file=mydump.sql

Not included in the make commands because of its interactive nature, to gain access to MariaDB do:

docker exec -it mariadb-drupal mysql -uusername -ppassword

Finally when you do:

make start-web

you should see output similar to the screenshot above!

Drush

It's important to have Drush available in your local environment. To achieve this, ensure that drush is a dependency requirement in your composer.json file. If not add it using my shortcut mentioned earlier. This will place drush in path /opt/drupal/vendor/bin inside the drupal-core container which obviously must be running for it to work. Exec-ing a command from the VM command line and escaping it correctly is hard work - so to save you the heartache I've created a shell function in my VM to make light work of it. If you are not using my VM, add this to the bottom of your /etc/bash.bashrc file.

drush() {
  PARAMS=$@
  CONTAINER="core-drupal"
  CMD="sh -c \"/opt/drupal/vendor/bin/drush ${PARAMS}\""
  DCMD="docker exec -it $CONTAINER $CMD"
  eval $DCMD
}

Now any Drush command is available to you (when the container is running!)

drush cr
 [success] Cache rebuild complete.

blog terms

AWS AWS CLI devops PHP Drupal Drupal 9

AWS Fargate Static Analysis with PHP

nigel — Mon, 13 Jan 2020 14:28:49 +0000

AWS Fargate Static Analysis with PHP nigel Mon, 13/01/2020 - 14:28

In my earlier blog on AWS Fargate Automated Testing I spoke about creating an AWS Fargate task definition which would include a web app's codebase and the Cucumber / Gherkin / maven / Java runtime stack. This tutorial is an opportunity to look at the static analysis stage of the pipeline where code is inspected for vulnerabilities, coding standards, compilation errors, copy and paste detection amongst others. Since my background is in Drupal and PHP, my tests will be focused around the PHP language and will use the excellent Static Analysis Tools for PHP repo.

Looking at the jakzal/phpqa repo, it is clear there is a great deal of tooling available. The purpose of this tutorial is take a small sample of these tools and create a Fargate task to run them against my codebase. My somewhat arbitrary selection is:

phpcpd - Copy/paste detection
phpcs - Codesniffer that detects coding standards violations
phplint - PHP syntax error checker.

Ok - let's get started.

Create the Dockerfile and the Entrypoint

As I have previously mentioned, Fargate is a black box and it doesn't lend itself to command line invocation of the tooling. Typically locally on a laptop we would run the jakzal/phpqa Docker image like:

$  docker run -it --rm -v $(PWD):/project -w /project  jakzal/phpqa phpcpd web/core/modules/views/src
$  docker run -it --rm -v $(PWD):/project -w /project  jakzal/phpqa phpcs web/core/modules/views/src
$  docker run -it --rm -v $(PWD):/project -w /project  jakzal/phpqa phplint web/core/modules/views/src

This could obviously be placed in a shell script. Now in Fargate we don't have access to the command line and we have to create a Fargate definition up front before that can be provisioned and then run. So it's a little bit more involved.

The simplest approach is to create a Dockerfile based on the jakzal/phpqa repo, and an extrypoint script that will run each one of the tools I have identified.

The Dockerfile is defined below. Note that I have selected PHP version 7.2. This is the closest supported version of PHP to what I'm running on my production box. Think I need to upgrade that at some point..

FROM jakzal/phpqa:1.28.1-php7.2
ADD entry.sh /entry.sh
RUN /bin/bash -c 'chmod +x /entry.sh'
ENTRYPOINT ["/entry.sh"]

Next up is the entry.sh file. First thing to note is I have added the set +e flag, which is default anyway. This is included to ensure that even if I get some pernickety error messages the script won't terminate immediately, and it acts as an aide memoire that we don't want error termination so don't accidentally copy paste in a competing set -e flag.

There is then a check whether the shared ECS volume has been loaded - this is the responsibility of the codebase container, but because containers can start at varying times, I have to ensure I don't run the PHP analysis before the volume is ready.

The path to the source code to perform the analysis on is defined as /var/static_tests/web/core/modules/views/src. This has two components. The first part /var/static_tests/ is the mount point which will correspond to my codebase top level directory. The second component, web/core/modules/views/src points to the core contributed Drupal Views module. This is purely there as a reference to some arbitrary code. I am using the codebase to my own Badzilla which is a site build with no custom code and thus I have none of my own code to use in this sample.

#!/usr/bin/env sh
 
set +e
 
while [ ! -d /var/static_tests/web ]
do
  sleep 5
  echo "Waiting for volume to be mounted and files copied into position"
done
 
phpcpd /var/static_tests/web/core/modules/views/src
phpcs /var/static_tests/web/core/modules/views/src
phplint /var/static_tests/web/core/modules/views/src

Building the Docker image locally

It isn't mandatory to test everything works before constructing the Fargate task definition, but it makes sense to me to test my work locally first. If the Fargate task subsequently fails then I know the problem area will be localised to the AWS eco-system, and probably to Fargate. So I always build locally and run locally first. To build I cd into my Dockerfile / entry.sh entry and issue:

$ docker build -f Dockerfile -t static-fargate  .

To run it's

$ docker run -it --rm -v $(PWD):/var/static_tests  static-fargate

You can see above I am establishing the Docker volume which will be shared with my codebase.

The Docker Hub repo for this is at https://hub.docker.com/repository/docker/sanddevil/static-fargate and the GitHub repo for the code is at https://github.com/sanddevil/devops-static-php

Define the Fargate task

With everything working locally it's time to build the Fargate task. This is very similar to the script I created in my earlier automated testing blog. There are a few differences obviously. My container stack comprises of a combined PHP + Apache + my own codebase image, and the PHP static analysis jakzal/phpqa obviously. I use the shell script heredoc convention to define each container. You should note I have now expanded my codebase container to include two shared volumes - one for the automated tests (not discussed here but discussed previously) and the static tests.

#!/bin/bash
 
 
BRANCH="master"
REPO="php7.0-apache/${BRANCH}"
 
# Repo account and repo
ECR_ACCOUNT="xxxxxxxxxxx.dkr.ecr.eu-west-2.amazonaws.com"
ECR_REPO="${ECR_ACCOUNT}/${REPO,,}"
ECS_CLUSTER="FargatePrototype"
 
# Get the Task Role ARN for executing the task
TASK_ARN=`aws iam get-role --role-name ecsTaskExecutionRole | jq .Role.Arn -r`
 
 
# Execution ARN appears to be the same as the Task ARN
EXECUTION_ARN="${TASK_ARN}"
 
 
# Determine if the log group already exists: NOTE: "aws ecs register-task-definition" will not create the log group for you!
LOG_EXISTS=`aws logs describe-log-groups | jq ".logGroups[] | select(.logGroupName == \"/ecs/"${BRANCH}"\") | .logGroupName" -r | wc -l`
if [[ "${LOG_EXISTS}" -eq 0 ]]; then
	aws logs create-log-group --log-group-name "/ecs/${BRANCH}"
fi
 
CONTAINER_CODEBASE=$(cat <
)
 
CONTAINER_STATIC_PHP=$(cat <
)
 
 
CONTAINERS_STATIC=$(cat <
)
 
 
aws ecs register-task-definition \
  --family "STATIC-${BRANCH}" \
  --volumes "name=static_tests,host={}" "name=automated_tests,host={}" \
  --task-role-arn "$TASK_ARN" \
  --execution-role-arn "${EXECUTION_ARN}" \
  --network-mode "awsvpc" \
  --container-definitions "${CONTAINERS_STATIC}" \
  --cpu "256" \
  --memory "512" \
  --requires-compatibilities "FARGATE"

I'm not quite done yet though. Somehow I need to copy my codebase into the shared Docker volume so that the PHP static analysis container can see it. This is done in the entry.sh script of my PHP + Apache + Codebase image (not discussed here) but the entry.sh file is shown below for completeness.

#!/bin/bash
set -e
 
# Copy the automated tests into a shared area accessible to maven
cp -R /var/www/html/test  /var/automated_tests/.
 
# Copy the PHP codebase into a shared area accessible to the PHP static analysis container
cp -R /var/www/html/web  /var/static_tests/.
 
# Run Apache in the foreground to prevent the container from quitting
apache2-foreground

Here you can see the copying I do for both the automated tests and the static analysis.

Running the Fargate task

Normally I would invoke the Fargate task programmatically using AWS CLI but since that leads to a rather dry image-free blog, I will use the console in this instance. My previous blog shows how to navigate around the ECS dashboard to run the task, then you can see the task running in the first screenshot above. It will only run until the entrypoint script terminates as per the rules on how Docker works, so to capture that scrennshot I had to be quick. The second screenshot shows the CloudWatch logs for the task, and you can see there are a few copy/paste errors detected.

blog terms

devops AWS AWS CLI ECS Docker bash

AWS Fargate: CI/CD Architecture for Drupal and PHP Frameworks

nigel — Sat, 04 Jan 2020 13:45:57 +0000

AWS Fargate: CI/CD Architecture for Drupal and PHP Frameworks nigel Sat, 04/01/2020 - 13:45

Introduction

AWS Fargate is a compute engine for Amazon ECS which allows the running of Docker containers without having to provision and manage the underlying EC2 servers. This makes the standing up of Docker bundles immeasurably easier, and thankfully that ease comes with very few limitations. It is often considered the Docker equivalent of AWS Lambda.

Fargate's ease of use and convenience makes it a natural fit for CI/CD architecture. Apps can be defined as Fargate tasks which are analogous to Docker Bundles, and pipeline stages can be engineered to perform standard activities such as build, automated testing, static analysis, dev playground, and deploy.

This architecture statement explains how this can be applied to the world of PHP frameworks and Drupal CMS. With a little imagination, and a few tweaks to the ecosystem and Fargate tasks, and the architecture would be valid for other high level languages and frameworks.

Pipeline Stages

Before we get into the physical AWS ecosystem, it's worth discussing what we are trying to achieve. I have identified the four main stages in the pipeline in the diagram above. There will be two entry points into the build stage: it will be either automated on a git pull request creation and approval, but also a manual invocation is possible for times when a dev or QA wants to check progress on a feature after a commit but before a PR is created. In PHP frameworks and Drupal 8+ codebases, it is customary to use the dependency management tool composer which obviates the need to check into vcs contributed plugins and modules. The build stage will also compile the front end themes and JavaScript and minify / uglify at the same - therefore production ready CSS and JS should not be committed into vcs by developers. Once the codebase has been built, it will be copied into a PHP/Apache Docker image and pushed into the AWS ECR repository for later use.

The automated testing will cover both functional BDD and unit TDD testing against the codebase image that has just been built. Writing TDD against the largely procedural Drupal 7 was onerous and challenging and therefore overlooked in the main. Thankfully Drupal 8 is OOP which makes writing back-end tests with PHPUnit much more straightforward.

The static analysis will check for code quality, vulnerability, smell, copy and paste detection, and basic lint type checking.

The deploy step can mean different things to different people. Deployment can mean pushing your built codebase to the production server, or it can mean simply creating a semantic versioning tag, and pushing the codebase artefact to an upstream repository, ready for a manual deployment to prod by Ops staff. I have opted for the latter interpretation in this instance.

CI/CD Ecosystem

The physical ecosystem is described below and shown in diagrammatic form above. Whilst on first glance this may appear quite complex, in fact it is largely self-evident with a little explanation.

Jenkins

Jenkins will be the automation tool since it is ubiquitous across the industry and there is a great deal of expertise in the market place. Jenkins has pipeline capability, and new projects should elect to use the Groovy-based declarative scripting DSL. This allows stages to run in parallel, and thus once the build stage is completed, automated testing and static analysis can be concurrent. In addition I propose the standing up of a 'persisting playground' - the built feature branch of the web app with a DNS qualified relevant url which can be shared between devs, QA, and stakeholders. This would be useful for troubleshooting, demonstrations of progress, show and tells etc., and would be torn down at close of business to reduce costs. My tutorial here shows how a friendly branch specific url can be created and propagated by AWS Route 53 within 60 seconds.

The Jenkins pipeline script is responsible for control flow only; all the actual 'doing' activities, such as building the codebase Docker image on the fly and committing it to ECR, is undertaken by shell scripts spawned from the pipeline. This means the power of AWS CLI can be leveraged to undertake all AWS specific tasks.

The flow of the pipeline will vary depending upon the digital team's needs. I have elected for the following:

On PR Creation, run build, automated testing, static analysis, persisting playground
On PR Approval, run build, automation testing (and fail the pipeline if testing fails), deploy

Obviously you may have different requirements here such as building on each feature branch commit, but the re-configuration effort will be minimal regardless. Adding conditions on the Jenkins pipeline stages is trivial.

AWS ECR

The PHP-Apache Docker image is a starting point for creating an image that will include the feature branch built by Jenkins. Detailed instructions on how to achieve this are covered in my earlier blog, and once the image is created it is committed to ECR. When a Fargate task is run, it must first be provisioned by fetching all the Docker images it needs, and having the codebase + Apache/PHP local to AWS will improve start-up.

AWS Fargate

Jenkins will run up to three concurrent AWS Fargate tasks - a task which includes all the Docker containers to run the automated tests, a task that has the static analysis containers, and the basic web app containers for the persisting playground which will contain Apache+PHP along with probably Memcached or Redis, and any other products and services required by your web app. More on the AWS Fargate task definitions later.

The automated tests and the static analysis run to conclusion and automatically die once their entry point scripts have run. The persisting playground will run Apache in the foreground to prevent termination.

AWS RDS

It is impractical to import db dumps into Fargate tasks, so all Fargate tasks will use one shared database resource. Obviously sharing a database can feasibly lead to schema inconsistencies when different schema changes are being developed on different feature branches. In Drupal these are implemented by drush updatedb and in PHP frameworks there is always specific command line tooling to perform database migrations. To mitigate against schema inconsistencies, there would need to be communication between developers, and the facility to reset the database to a known develop branch datum with a Jenkins job.

AWS S3

An export of the develop branch database should be held locally in AWS S3 to facilitate quick import into RDS as required, and updated nightly.

AWS S3 & AWS Cloudfront

File assets such as jpg and png images are normally held in /sites/{default|domain name}/files in Drupal but are not committed to vcs. These need to be synced from the host's server nightly into S3, and for greater performance the use of Cloudfront acceleration is recommended. Configuration changes will be required in the Drupal or PHP framework codebase to point to the Cloudfront endpoints. Drupal will require the use of the Storage API module or the S3 File System module.

AWS CloudWatch

Fargate integrates with CloudWatch easily, and output from each Docker container in the Fargate task definition will send stdout and stderr to a CloudWatch log group. This is imperative not just for troubleshooting, but also to determine the automated test results. By parsing the logs, a full test output can be created.

Slack Integration

It is desirable for the CI/CD pipeline to be hands-free, i.e. Jenkins login not required unless a manual pipeline invocation is necessary. To achieve this goal, Slack integration is required with job statuses being sent to dedicated Slack channels, and output from the test results also sent through Slack.

Existing Host

Of course to seed RDS and the file asset S3, we need to get database exports and assets from our existing host. Exactly how to achieve this will vary dependent upon the host. If possible, a slave Jenkins can execute remote jobs on the host to upload the db exports and file assets to RDS and S3 respectively, and that is how I have drawn my diagram. If this is not possible, a combination of scp and rsync will suffice.

Fargate Task Definitions

Fargate tasks can be defined in the AWS Console, but I've already identified that AWS CLI is the tool of choice for programmatic creation. The CLI requires the container definitions in JSON format, and for the pipeline discussed above there would be three Fargate definitions per feature branch. The first is the standard web app which would typically have the PHP/Apache container with the codebase copied into it during a docker build command. The second is the automated testing stack which includes Maven (which has the Java runtime bundled) along with Selenium grid and two nodes: Firefox and Chrome. I spoke about this Fargate task at length in an earlier blog here. The third definition is for static analysis and I am suggesting the excellent jakzal/phpqa repository for this.

The three tasks would be defined and then run within the Jenkins pipeline. It must be noted that once the tasks have been run they should be immediately deleted from AWS ECR. Storage in ECR is costed, and in the unlikely event of the tasks being required at a later date, they can always be recreated by invoking the pipeline again manually and selecting the feature branch from a drop down list.

The use of Drupal Drush

Fargate is a black box, without access to the underlying EC2 infrastructure, and therefore it is not possible to ssh into EC2 and then perform a docker exec command. Furthermore drush aliases require an ssh daemon running on the target alias to be able to execute remote commands from say a developer working on premises. So how can we run the normal set of everyday drush commands such as to apply schema changes, apply features, split the configuration, flush the caches etc?

Ok so we have PHP installed on Jenkins since we have to run composer install in the build stage. We also have an RDS instance with the database. We therefore have what we need to bootstrap Drupal - which is a requirement of drush.

Therefore all drush commands should be run on Jenkins during the build stage. These commands should be listed in a sibling directory to the codebase's docroot or web directory, in say devops_hooks or some such. As part of the Jenkins build stage, the existence of directory devops_hooks will be checked, and all shell scripts within it will be run.

blog terms

AWS AWS CLI devops PHP Drupal

AWS Fargate Automated Testing with Cucumber, Gherkin, Java, Maven, Selenium Grid Hub and Chrome and Firefox Nodes

nigel — Mon, 09 Dec 2019 08:37:50 +0000

AWS Fargate Automated Testing with Cucumber, Gherkin, Java, Maven, Selenium Grid Hub and Chrome and Firefox Nodes nigel Mon, 09/12/2019 - 08:37

Tutorial for creating an automated testing AWS Fargate task that will include Docker containers for a full testing framework:

Maven for build automation
Java Runtime to enable the execution of maven and the junit unit testing framework
Java Cucumber - a tool for BDD testing
Gherkin to define behaviours in the English language
Selenium Hub being the central loading point for the tests
Selenium Chrome - one of the nodes
Selenium Firefox - a second node
Apache / PHP - the web app and web server.

This is not a trivial exercise but by the end of the blog I will have a working automated testing environment which can be slotted into a DevOps pipeline. Please note this blog is not a tutorial on how to write Cucumber / Gherkin tests, however I will create a few sample Gherkin behaviours and their respective Java Cucumber stepdefs as a starting point. These can then be expanded out by your QA team.

Starting Point

The starting point will be the simple PHP/Apache web app I put together in my previous blog here - this is a single url website which merely shows the phpinfo() status page. But it's ideal for our automated testing because our testing framework is plenty complex already, and we can still run a few tests against this simple landing page such as whether the additional PHP extensions I included in the Dockerfile are loaded correctly.

Now we need to build out our testing tree structure.

At a sibling level to the docroot directory I created in the previous blog, create the following:

$ mkdir -p mytests/src/test/java/mytests
$ mkdir -p mytests/src/test/resources/mytests

Now it's time to create the files I need

$ cd mytests/
$ touch pom.xml
$ touch src/test/java/mytests/RunCucumberTest.java
$ touch src/test/java/mytests/Stepdefs.java
$ touch src/test/java/mytests/SetupTestDriver.java
$ touch src/test/resources/mytests/home_page.feature

This will create a tree structure like this:

$ tree
.
|-- pom.xml
`-- src
    `-- test
        |-- java
        |   `-- mytests
        |       |-- RunCucumberTest.java
        |       |-- SetupTestDriver.java
        |       `-- Stepdefs.java
        `-- resources
            `-- mytests
                `-- home_page.feature
 
6 directories, 5 files

Now we need to flesh out those files we created. Firstly the pom file which contains a list of the project's dependencies. Careful inspection will show you that I used the hellocucumber template as my starting point.
pom.xml

 version="1.0" encoding="UTF-8"?>
 xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    >4.0.0>
 
    >hellocucumber>
    >mytests>
    >1.0.0-SNAPSHOT>
    >jar>
 
    >
        >4.2.6>
        >UTF-8>
    >
 
    >
        >
            >io.cucumber>
            >cucumber-java>
            >${cucumber.version}>
            >test>
        >
 
        >
            >io.cucumber>
            >cucumber-junit>
            >${cucumber.version}>
            >test>
        >
 
        >
            >junit>
            >junit>
            >4.12>
            >test>
        >
 
    
    >
        >io.github.bonigarcia>
        >webdrivermanager>
        >3.7.1>
    >
 
	
	>
    		>org.seleniumhq.selenium>
    		>selenium-java>
    		>3.141.59>
	>
 
    
    >
        >org.seleniumhq.selenium>
        >selenium-chrome-driver>
        >3.141.59>
    >
 
    
    >
        >org.apache.maven.plugins>
        >maven-surefire-plugin>
        >2.22.2>
    > 
 
    >
        >org.slf4j>
        >slf4j-api>
        >1.5.6>
    >
 
    >
        >org.slf4j>
        >slf4j-log4j12>
        >1.5.6>
    > 
 
    >
 
    >
        >
            >
                >org.apache.maven.plugins>
                >maven-compiler-plugin>
                >3.7.0>
                >
                    >UTF-8>
                    >1.8>
                    >1.8>
                >
            >
        >
    >
>

Now my Gherkin tests themselves. These are rather simple - I am checking for whether my GD and Memecached PHP extensions have been loaded, i.e. they are listed on the front page which just shows the entire PHP configuration. Under normal circumstances Gherkin test tend to be GIVEN..WHEN..THEN constructs, but since I am not actually doing anything (e.g. clicking a button to go somewhere) I only need WHEN..THEN
home_page.feature

Feature: Are we on the Home Page?
 
 
  Scenario: On the home page
    When User is on the Home Page
    Then Message Displayed Memcached
 
    When User is on the Home Page
    Then Message Displayed GD

Next is the StepDef file where I convert the Gherkin into Cucumber Java containing the actual business logic behind the quasi-English Gherkin. I don't pretend to be an expert Java coder - I have the sum total of about 4 days experience. Therefore my code is probably sub-optimal, but it does the job and an in-house Java professional could whip it into shape. I have defined two browsers - Firefox and Chrome as per our Selenium configuration, although please note there is currently no semblance of parallel running of the tests in here. That again has been left to more talented Java practitioners than myself.
StepDefs.java

package mytests;
 
import base.SetupTestDriver;
 
import java.util.concurrent.TimeUnit;
 
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import io.github.bonigarcia.wdm.WebDriverManager;
 
import cucumber.api.java.en.Given;
import cucumber.api.java.en.Then;
import cucumber.api.java.en.When;
 
import java.net.MalformedURLException;
 
import static org.junit.Assert.*;
 
 
public class Stepdefs {
 
	public static WebDriver[] drivers;
	public static String[][] browsers;
 
    @When("User is on the Home Page")
    public void user_is_on_Home_Page() throws Throwable {
       	browsers = new String[][] {
       		{"linux", "firefox", "http://127.0.0.1", "http://127.0.0.1:4444"},
            {"linux", "chrome", "http://127.0.0.1", "http://127.0.0.1:4444"}
        };
 
		drivers = new WebDriver[browsers.length];
        for (int i = 0; i < browsers.length; i++) {
        	drivers[i] = new SetupTestDriver(browsers[i][0], browsers[i][1], browsers[i][2], browsers[i][3]).getDriver();
        }
    }
 
    @Then("Message Displayed Memcached")
    public void message_displayed_memcached() throws Throwable {
        for (int i = 0; i < drivers.length; i++) {
        	Boolean exists = !drivers[i].findElements(By.name("module_memcached")).isEmpty();
        	drivers[i].quit();
        	assertTrue(exists);
        }
    }
 
	@Then("Message Displayed GD")
	public void gd() throws Throwable {
        for (int i = 0; i < drivers.length; i++) {
			Boolean exists = !drivers[i].findElements(By.name("module_gd")).isEmpty();
        	drivers[i].quit();
        	assertTrue(exists);
        }
	}
}

The StepDef file doesn't contain any Firefox or Chrome webdriver startup configuration. This has been placed in a separate Java class.
SetupTestDriver.java

// src/main/java/base/SetupTestDriver.java
package base;
 
import org.openqa.selenium.Platform;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.firefox.FirefoxProfile;
import org.openqa.selenium.firefox.FirefoxDriverLogLevel;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.Dimension;
 
import java.net.MalformedURLException;
import java.net.URL;
import java.util.concurrent.TimeUnit;
 
public class SetupTestDriver {
    private WebDriver driver = null;
    private String browser = null;
    private String baseUrl = null;
    private String os = null;
    private String node = null;
 
    public SetupTestDriver(String os, String browser, String baseUrl, String node) throws MalformedURLException {
        this.browser = browser;
        this.os = os;
        this.baseUrl = baseUrl;
        this.node = node;
 
        Platform platform = Platform.fromString(os.toUpperCase());
        if(browser.equalsIgnoreCase("chrome")) {
            ChromeOptions chromeOptions = new ChromeOptions();
            chromeOptions.addArguments("--headless");
            chromeOptions.setCapability("platform", platform);
            this.driver = new RemoteWebDriver(new URL(node + "/wd/hub"), chromeOptions);
        } else if (browser.equalsIgnoreCase("firefox")) {
            FirefoxOptions firefoxOptions = new FirefoxOptions();
            firefoxOptions.setProfile(new FirefoxProfile());
            firefoxOptions.setCapability("platform", platform);
            firefoxOptions.setHeadless(true);
            this.driver = new RemoteWebDriver(new URL(node + "/wd/hub"), firefoxOptions);
        }
 
        this.driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
        // Maximizing the screen doesn't appear to work.
        this.driver.manage().window().setSize(new Dimension(1600,900));
        this.driver.get(baseUrl);
 
    }
 
    public String getOs() {
        return this.os;
    }
 
    public String getBrowser() {
        return this.browser;
    }
 
    public String getBaseUrl() {
        return this.baseUrl;
    }
 
    public String getNode() {
        return this.node;
    }
 
    public WebDriver getDriver() {
        return this.driver;
    }
}

Finally there's some code to perform the glue between Gherkin and Cucumber.
RunCucumberTest.java

package mytests;
 
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
 
@RunWith(Cucumber.class)
@CucumberOptions(plugin = {"pretty"})
public class RunCucumberTest {
}

Build the Docker Codebase + PHP/Apache

In my previous blog here I showed how to add the codebase to the php:7.2-apache image. The codebase has now changed, and I now have two top level sibling directories docroot and mytests - which contains the automated test code I've just added. There is a change we need to make to the Dockerfile in this revision, and the inclusion of an entrypoint script called entry.sh. The entry.sh file will copy the test code suite into a shared Fargate non-persistent drive so that the Maven container can access the codebase and run the tests against it. Once the copy of files has been completed, Apache must be run in foreground mode to prevent the container from quitting. Below we have the new version of the Dockerfile, and the entry.sh file. The image needs to be rebuilt, but since that was covered in the previous blog, it is omitted here.

Dockerfile

FROM php:7.2-apache
RUN apt-get update && apt-get install -y libmemcached-dev zlib1g-dev \
    libfreetype6-dev \
    libjpeg62-turbo-dev \
    libpng-dev \
    libwebp-dev \
    && pecl install memcached \
    && docker-php-ext-enable memcached \
    && docker-php-ext-configure gd --with-gd --with-webp-dir --with-jpeg-dir \
       --with-png-dir --with-zlib-dir --with-freetype-dir \
    && docker-php-ext-install gd
COPY --chown=www-data:www-data ./ /var/www/html/.
COPY entry.sh /usr/local/bin/.
ENV AH_SITE_ENVIRONMENT devops
ENV APACHE_DOCUMENT_ROOT /var/www/html/docroot
RUN sed -ri -e 's!/var/www/html!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's!/var/www/!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf
RUN /bin/bash -c 'chmod +x /usr/local/bin/entry.sh'

entry.sh

#!/bin/bash
set -e
 
# Copy the automated tests into a shared areaaccessible to maven
cp -R /var/www/html/mytests  /var/automated_tests/.
 
# Run Apache in the foreground to prevent the container from quitting
apache2-foreground

Maven:3.6.2-jdk-13

The Java runtime is bundled with the Maven image if the correct tag is selected. I have opted for maven:3.6.2-jdk-13 which is up to date and supported. This requires a small amount of customisation, so again a Dockerfile will be needed. In this instance I want to put a wait loop in the entrypoint script to ensure that there's no attempt to run the tests before the non-persistent shared drive containing the test suite has been mounted by the d8codebase container. At the end I exec the mvn clean install command to run the tests.

The easiest way of achieving this is to edit the mvn-entrypoint.sh script which is in /usrlocal/bin. I copied out of the image to edit and the Dockerfile copies it back into place.

Using my previous blog I built this, tagged and pushed to ECR using the image name d8maven. The instructions aren't copied here for brevity reasons - please refer to my previous blog if you need to.

Dockerfile

FROM maven:3.6.2-jdk-13
COPY mvn-entrypoint.sh /usr/local/bin/mvn-entrypoint.sh
RUN /bin/bash -c 'chmod +x /usr/local/bin/mvn-entrypoint.sh'

mvn-entrypoint.sh

#! /bin/sh -eu
 
# Copy files from /usr/share/maven/ref into ${MAVEN_CONFIG}
# So the initial ~/.m2 is set with expected content.
# Don't override, as this is just a reference setup
 
copy_reference_files() {
  local log="$MAVEN_CONFIG/copy_reference_file.log"
  local ref="/usr/share/maven/ref"
 
  if mkdir -p "${MAVEN_CONFIG}/repository" && touch "${log}" > /dev/null 2>&1 ; then
      cd "${ref}"
      local reflink=""
      if cp --help 2>&1 | grep -q reflink ; then
          reflink="--reflink=auto"
      fi
      if [ -n "$(find "${MAVEN_CONFIG}/repository" -maxdepth 0 -type d -empty 2>/dev/null)" ] ; then
          # destination is empty...
          echo "--- Copying all files to ${MAVEN_CONFIG} at $(date)" >> "${log}"
          cp -rv ${reflink} . "${MAVEN_CONFIG}" >> "${log}"
      else
          # destination is non-empty, copy file-by-file
          echo "--- Copying individual files to ${MAVEN_CONFIG} at $(date)" >> "${log}"
          find . -type f -exec sh -eu -c '
              log="${1}"
              shift
              reflink="${1}"
              shift
              for f in "$@" ; do
                  if [ ! -e "${MAVEN_CONFIG}/${f}" ] || [ -e "${f}.override" ] ; then
                      mkdir -p "${MAVEN_CONFIG}/$(dirname "${f}")"
                      cp -rv ${reflink} "${f}" "${MAVEN_CONFIG}/${f}" >> "${log}"
                  fi
              done
          ' _ "${log}" "${reflink}" {} +
      fi
      echo >> "${log}"
  else
    echo "Can not write to ${log}. Wrong volume permissions? Carrying on ..."
  fi
}
 
# Wait until we see the test files that are being copied from the PHP/Apache/Codebase container to a shared volume.
while [ ! -d /var/automated_tests/mytests ]
do
  sleep 2
done
 
owd="$(pwd)"
copy_reference_files
unset MAVEN_CONFIG
 
cd "${owd}"
unset owd
 
exec mvn -f /var/automated_tests/mytests -q clean install

Define the Fargate Task

I now have all parts I need apart from a shell script to actually define the Fargate task which will contain all the config for the containers. I have made extensive use of the heredoc convention in the shell script - one per container definition - which makes it easier to read, and also means when I come to make up other tasks (such as static analysis, and persistent playground) I can simple swap out those containers I don't need.

There are a few comments here to note.

You cannot create a Fargate Task without the IAM ecsTaskExecutionRole, but that will not be created for you. It is however created when you manually create your first Fargate task in the AWS console. Of course ecsTaskExecutionRole can be created programmatically too - and will be the subject of a block at another time.
The Fargate task definition will not create your CloudFront log group for you! Therefore I do a check in the script to see if it already exists, and if not then create.
I have defined a mount point to where I copy the test suite. This is because the test suite is in the code repository which is copied into the PHP/Apache container, but is needed by the Maven container also.
The Selenium Chrome browser is using a non-standard port 5556. This is because it would clash with Firefox otherwise which also defaults to port 5555. You must remember to use an environmental variable NODE_PORT to change the port as well as the hostPort and containerPort.
I have defaulted the task's compute to extremely high cpu and memory. They should be reduced using trial and error to avoid incurring high compute fees.

#!/bin/bash
 
# Rewrite branch name so no feature/ hotfix/ forward slash
BRANCH="selenium-testing"
REPO="d8codebase"
 
# Repo account and repo
ECR_ACCOUNT="XXXXXXXXXXX.dkr.ecr.eu-west-2.amazonaws.com"
ECR_REPO="${ECR_ACCOUNT}/${REPO,,}"
ECS_CLUSTER="default"
 
# Get the Task Role ARN for executing the task
TASK_ARN=`aws iam get-role --role-name ecsTaskExecutionRole | jq .Role.Arn -r`
 
 
# Execution ARN appears to be the same as the Task ARN
EXECUTION_ARN="${TASK_ARN}"
 
 
# Determine if the log group already exists: NOTE: "aws ecs register-task-definition" will not create the log group for you!
LOG_EXISTS=`aws logs describe-log-groups | jq ".logGroups[] | select(.logGroupName == \"/ecs/"${BRANCH}"\") | .logGroupName" -r | wc -l`
if [[ "${LOG_EXISTS}" -eq 0 ]]; then
	aws logs create-log-group --log-group-name "/ecs/${BRANCH}"
fi
 
CONTAINER_CODEBASE=$(cat <
)
 
CONTAINER_SELENIUM_HUB=$(cat <
)
 
CONTAINER_SELENIUM_FIREFOX=$(cat <
)
 
CONTAINER_SELENIUM_CHROME=$(cat <
)
 
CONTAINER_MAVEN=$(cat <
)
 
 
CONTAINERS_AUTOMATIC=$(cat <
)
 
 
aws ecs register-task-definition \
  --family "AUTOMATION-${BRANCH}" \
  --volumes "name=automated_tests,host={}" \
  --task-role-arn "$TASK_ARN" \
  --execution-role-arn "${EXECUTION_ARN}" \
  --network-mode "awsvpc" \
  --container-definitions "${CONTAINERS_AUTOMATIC}" \
  --cpu "4096" \
  --memory "30720" \
  --requires-compatibilities "FARGATE"

Running the Fargate Task

Once the script is executed, the task will be defined and ready to run. This can be run by creating a new shell script (or extending the one above) or by using the console. Since this blog is becoming too code focused, I'll document the running of tasks programmatically in another blog later. For now you should be able to see your task in the console and run it. See screenshot 1 above.

Screenshot 2 show that the task has reached RUNNING state. If you are quick you can click on the task identifier and be taken to the more detailed screen shown in image 3 which shows all the containers that are running.

Once one of the Docker images completes its activity, a signal will be sent to all the other containers to stop. In the task described in this blog, the first task to complete is the running of the automated tests. Once the tests complete, then all other containers will stop. See image 4.

Did the Tests Actually Succeed?

A valid question! The output of the Maven container is written to a CloudWatch log stream which can be parsed and echoed back to the screen, or sent to a Slack channel (more on this in a later blog). For now I have created a sequence of commands using pipes which includes the open source awslogs utility.

$ aws logs describe-log-streams --log-group-name "/ecs/selenium-testing" --log-stream-name-prefix "ecs/Maven" | jq '.logStreams|=sort_by(-.creationTime)|.logStreams[0].logeamName' -r | xargs -I{} awslogs get "/ecs/selenium-testing" {} -s1d | cut -d ' ' -f 1,2 --complement
mvn
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running mytests.RunCucumberTest
Dec 10, 2019 3:43:46 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Dec 10, 2019 3:43:47 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Dec 10, 2019 3:43:49 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Dec 10, 2019 3:43:50 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Feature: Are we on the Home Page?
  Scenario: On the home page         # mytests/home_page.feature:4
    When User is on the Home Page    # Stepdefs.user_is_on_Home_Page()
    Then Message Displayed Memcached # Stepdefs.message_displayed_memcached()
    When User is on the Home Page    # Stepdefs.user_is_on_Home_Page()
    Then Message Displayed GD        # Stepdefs.gd()
1 Scenarios (1 passed)
4 Steps (4 passed)
0m6.835s
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.055 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Troubleshooting Selenium Grid

Getting the Selenium stack working can be fiddly, although the CloudWatch logs are a very good way of diagnosing problems. If you want to satisy yourself that Selenium Hub and its two nodes (Firefox and Chrome) are actually there and available, there is a way but you'll have to work fast on the console since the Fargate tasks completes and shuts down once the tests are complete.

When the Fargate task transitions from PENDING to RUNNING, click on the task identifier. Copy the public IP address into your clipboard - you'll need it shortly. Now click on the ENI (Elastic Network Interface) link which will launch a new tab. Then click on the security group. Now you need to create a new rule. By default port 4444 is blocked on the public IP address - and to see Selenium working I need that to be open.

Edit the rules, create a new custom TCP/IP rule, add 4444 as the port and select anywhere for the CIDR, then save. See screenshot 1 above.

You now instantly have an open port 4444. Open a new browser tab, then paste the IP address you copied a few steps ago, then append /grid/console to it and enter. You should see your Grid + Node configuration as per the second screenshot above.

blog terms

devops ECS AWS AWS CLI Development Docker bash

Programmatically Create AWS ECR Repository and Commit Docker Image to ECR

nigel — Sat, 07 Dec 2019 07:56:15 +0000

Programmatically Create AWS ECR Repository and Commit Docker Image to ECR nigel Sat, 07/12/2019 - 07:56

This is a tutorial for using AWS CLI to programmatically create an AWS ECR repository and then commit a Docker image to that repository. This is the first step in a series of tutorials for creating an automated testing framework in AWS Fargate which can be one step of a DevOps pipeline.

As an example I am going to use the Docker php:7.2-apache image and to make it a little more complex, add a few more PHP extensions using a new Dockerfile so that my new image will be Drupal 8 compliant.

Let's get started.

Pull Base Image and Define Dockerfile

The base php:7.2-apache image needs to be pulled from Docker Hub. Issue the following command.

$ docker pull php:7.2-apache
7.2-apache: Pulling from library/php
000eee12ec04: Pull complete 
8ae4f9fcfeea: Pull complete 
60f22fbbd07a: Pull complete 
ccc7a63ad75f: Pull complete 
a2427b8dd6e7: Pull complete 
91cac3b30184: Pull complete 
d6e40015fc10: Pull complete 
240e21c03bb4: Pull complete 
504858e1e4aa: Pull complete 
9a0523b2d73f: Pull complete 
e3acb84829f4: Pull complete 
28a372733f87: Pull complete 
62ee66cdb80a: Pull complete 
c3f368ddc7aa: Pull complete 
Digest: sha256:602bef9acc8d54527ae5b7ee0f16e5cdb6ab81364310f5123b93b95e357f608d
Status: Downloaded newer image for php:7.2-apache
docker.io/library/php:7.2-apache
$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
php                 7.2-apache          fec8a1aaffb8        31 hours ago        410MB
lambci/lambda       go1.x               8d962cde7e27        3 months ago        707MB
sls-docker          latest              8d962cde7e27        3 months ago        707MB

Yes the docker image is there. Now I need to create a directory for the Dockerfile and a sample index.php file which will prove our image is working.

$ mkdir automated && cd automated

By default the Docker php image uses /var/www/html as Apache's docroot. However, I'm going to alter this to something more standard for Drupal web apps - a directory called docroot will be created and will contain a simple index.php containing a call to phpinfo();

$ mkdir docroot && echo " > docroot/index.php

The Dockerfile should now be created and populated. There a number of activities that need to be performed on it.

Reference the base image with a FROM instruction
Install the libraries that support memcached and graphics manipulation
Enable memcached PHP extension
Configure the graphics PHP extension
Install the graphics PHP extension
Copy the files in the current directory and below into the Docker image at the /var/www/html ensuring the permissions are correct for the apache user and group
Set the environment variable AH_SITE_ENVIRONMENT to devops - this is an Acquia Cloud environment variable and only of any significance for those using Acquia as their host - or need a setting in my case to denote that devops is not running on Acquia
Change the default location for the docroot - that means setting an environment variable and using the sed editor to change the configuration in the sites files

Dockerfile

FROM php:7.2-apache
RUN apt-get update && apt-get install -y libmemcached-dev zlib1g-dev \
    libfreetype6-dev \
    libjpeg62-turbo-dev \
    libpng-dev \
    libwebp-dev \
    && pecl install memcached \
    && docker-php-ext-enable memcached \
    && docker-php-ext-configure gd --with-gd --with-webp-dir --with-jpeg-dir \
       --with-png-dir --with-zlib-dir --with-freetype-dir \
    && docker-php-ext-install gd
COPY --chown=www-data:www-data ./ /var/www/html/.
ENV AH_SITE_ENVIRONMENT devops
ENV APACHE_DOCUMENT_ROOT /var/www/html/docroot
RUN sed -ri -e 's!/var/www/html!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's!/var/www/!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf

Build the Image, Run the Container

Once the Dockerfile is complete, it is built with

$ docker build -t d8codebase .

and to confirm it's built correctly:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
d8codebase          latest              c2193f9778a6        33 minutes ago      439MB
<none>              <none>              2a71964685e0        2 hours ago         432MB
php                 7.2-apache          fec8a1aaffb8        35 hours ago        410MB
lambci/lambda       go1.x               8d962cde7e27        3 months ago        707MB
sls-docker          latest              8d962cde7e27        3 months ago        707MB

Great it's there. Now to spin up the container.

$ docker run -d -p 8080:80 d8codebase

Note I am mapping port 8080 to the container's port 80 which probably won't be pertinent to you. Long story I will try to make short. I write my blogs on my Macbook on a local network in the 192.168 range assigned by my ISP's hub. But I develop and test all the practical work in my blogs in a VM in the IP 10. range. Routing between 192.168 and 10. is difficult without port forwarding - so I port forward 127.0.0.1 8080 to my VM's 10.0.2.15 8080, so when I run my Docker Apache container I must remember incoming is on port 8080 and map that against the container's port 80. Hope that makes sense!

Now check it's running:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
9a6dc2fb88ff        d8codebase          "docker-php-entrypoi…"   44 minutes ago      Up 44 minutes       0.0.0.0:8080->80/tcp   agitated_feistel

Yes, and we can see from the screenshot above that it's picking up the memcached and the gd graphics extensions. Success.

Programmatically Create the AWS ECR Repository

This is quite easy and I've put it into a shell script that requires one parameter - the repo name to be created. The work is done by the aws ecr create-repository call. The rest of the code is simply checking for error conditions. The screenshot above was taken on the AWS console after the shell script below was successfully run with a parameter of d8codebase.

#!/bin/bash
 
#Programmatically create an ECR repo
 
# Check whether we have the args
if [[ -z $@ ]]; then
	echo "usage: [repo_name]"
	exit 1
fi
 
# Create the repo
REPO_NAME=`aws ecr create-repository --repository-name "$1" | jq .repository.repositoryName -r`
 
# Check it was created ok. We already get a quality stderr msg from aws cli so just quit on error.
# Similarly, if already exists it will tell us
if [[ "${REPO_NAME}" != $1 ]]; then
	exit 2
fi
 
echo "Successfully created ${REPO_NAME}"

Now the script to automatically login to Docker and push the image to ECR. Note that AWS ECR does require you to use the docker login command, and therefore provides a AWS CLI ECR command get-login which returns a token than can be piped into the docker login command. The token retrieved from ECR has a large payload, and it is top and tailed with other metadata beyond just the password. Therefore that needs to be removed, and if you look closely at the script below, you can see that I pipe the entire returned payload through sed with a regex, and then the result is piped onwards to docker login. Hopefully the rest is quite self explanatory - and you can see the screenshot above that the push has worked and the image can now be included in ECS / Fargate task definitions.

#!/bin/bash
 
#Programmatically push to an ECR repo
 
# Check whether we have the args
if [[ -z $@ ]]; then
	echo "usage: [repo_name]"
	exit 1
fi
 
REPO_NAME=$1
ECR_ACCOUNT="XXXXXXXXXX.dkr.ecr.eu-west-2.amazonaws.com"
ECR_REPO="${ECR_ACCOUNT}/$1"
 
# AWS Authentication - since we need to use docker push
# The sed regex will top and tail the additional meta data sent by the aws login request.
aws ecr get-login --region eu-west-2  | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS ${ECR_ACCOUNT}
 
# Get the image id of the Docker build. If there is a "latest" use that else get the first without latest
TAG=`docker images | grep -w "${REPO_NAME}" | grep latest | awk '{ print $3; }'`
if [[ -z ${TAG} ]]; then
        TAG=`docker images | grep -w "${REPO_NAME}" | awk '{ print $3; }'`
fi
 
docker tag "$TAG" "${ECR_REPO}"
docker push "${ECR_REPO}"

blog terms

AWS AWS CLI Docker devops Fargate ECS Development bash

Create Arbitrary Subdomains for AWS Fargate Tasks using AWS CLI

nigel — Fri, 01 Nov 2019 15:03:02 +0000

Create Arbitrary Subdomains for AWS Fargate Tasks using AWS CLI nigel Fri, 01/11/2019 - 15:03

AWS Fargate is a relatively new product offering the capability of running Docker bundles as an ECS compute service on EC2 without the necessity for the user to have to orchestrate the underlying EC2 infrastructure. It is perfect for DevOps, and was the technology I chose as DevOps Architect at my current client. The client wanted standard CI/CD capabilities such as spinning up containerised copies of their website environment per feature branch to perform static analysis, automated testing, and also significantly, a persisting playground that QA or development personnel could use for manual testing and troubleshooting a feature (or hotfix) branch they are currently working on.

This is similar to the offering of both Platform.sh and Amazee Labs hosting solutions - developers can spin up their feature branches. Alas my client is locked into another host yet needed on-the-fly feature branch Docker bundles.

AWS Fargate tasks can achieve this, but out of the box it isn't possible to create public domain registration via ECS Service Discovery. The Fargate task is issued a bare IPv4 address which will obviously work but our requirement is to have a subdomain created per branch that would be of the format {hotfix|feature}--{jira ticket}-{suitable branch name}.{top level domain}. So an example would be:

feature--doi-746-my-whizzy-new-feature.clientdomain.dev

So whilst we can't create a subdomain like that natively, we can do it by retrieving the IP address from the ECS Fargate task, then add that IP to a public Route 53 Hosted Zone. Furthermore, since the Fargate task is run via a Jenkins pipeline job that uses shell scripts populated with AWS CLI, the good news is we can indeed complete the registration of subdomain also using the AWS CLI *and* have the DNS propagate within 60 seconds! Amazing! Let's dive into how we achieve this.

Requirements + Assumptions

The solution will be built using the standard httpd Apache2 server sample Fargate task described on the AWS blog Tutorial: Creating a Cluster with a Fargate Task Using the AWS CLI. I added the task definition into the console, although it is also possible create a shell script and use AWS CLI. I have used the default cluster for brevity's sake, and the AWS account default VPC. I have also transferred a spare domain I had, saasidate.com, into Route 53 before I started the blog. This will be my tld and all the subdomains will be added off it. It's important that the ownership of the domain is transferred into Route 53 since otherwise subdomains will have authority issues if the parent domain is still registered to other domain providers such as GoDaddy or 123-Reg.

The solution I am putting together will use AWS CLI which by default returns JSON structured data. It is absolutely essential to use the Linux command line utility jq for parsing the JSON. Documentation on jq is readily available on the net.

Steps

The solution will need the following steps:

Get the user's default VPC (or create a custom VPC if preferred. We'll need this at a few points in the code.
Get a subnet from the VPC. Three subnets are created for use with the default VPC. I pick the first which is arbitrary.
Get a security group. A new security group is created per task when using the console but that is unnecessary for the shell scripts we will be building. So in our script, check whether our 'standard' security group has been created. If so use it, if not then create a security group with an ingress port of 80 and a CIDR of 0.0.0.0/0 which allows access worldwide on TCP port 80.
Run the Fargate task and loop / sleep until the task has a status of RUNNING.
Get the network interface id of the task and interrogate it for the IP address.
Get the hosted zone for the parent domain.
Add a record set containing the feature branch subdomain to the parent domain's hosted zone.
Create a new hosted zone using the feature branch.

Ok - let's crack on and put this into AWS CLI shell scripts.

Get the VPC

I'm using the default VPC so I can filter the list I get back from AWS using the IsDefault attribute.

VPC=`aws ec2 describe-vpcs | jq '.Vpcs[] | select(.IsDefault == true) | .VpcId' -r`

Note the use of the jq select statement. This is a familiar construct and will be used throughout this blog. Also note the -r flag - this removes the double quotes from the response.

Get the First Subnet in the Default VPC

Here I am using jq again but piping the results through head to get the first subnet. There wasn't an obvious jq query to do this natively within jq.

# Get the first subnet of the VPC
SUBNET=`aws ec2 describe-subnets | jq --arg VPC "$VPC" '.Subnets[] | select(.VpcId == $VPC) | .SubnetId' -r | head -n 1`

Get the Security Group

Here I check whether we already have a dedicated security group which gives us worldwide access to port 80. I search first for a unique security group description and if it doesn't exist, then create it.

# Check if our security group for port 80 webserver already exists. Search for our funky unique description
SECURITY=`aws ec2 describe-security-groups | jq ' .SecurityGroups[] | select (.Description == "Ingress Port 80 Anywhere") | .GroupId' -r`
 
# Let's inspect what we got back.
if [[ $? -ne 0 ]]; then
	echo "describe-security-groups failed with code $?"
	exit $?
fi
 
# If we didn't get a security group then create
if [[ -z "$SECURITY" ]]; then
	SECURITY=`aws ec2 create-security-group \
		--description "Ingress Port 80 Anywhere" \
		--group-name "Fargate Webserver Port 80" \
		--vpc-id "$VPC" | jq .GroupId -r `
 
 
	aws ec2 authorize-security-group-ingress \
		--group-id "$SECURITY" \
		--ip-permissions "FromPort=80,ToPort=80,IpProtocol=TCP,IpRanges=[{CidrIp=0.0.0.0/0}]"
fi

Run the Fargate Task

Next I run the Fargate task. This is the sample Fargate task covered in the AWS documentation and uses the httpd docker image. When I created it in the console I called it "first-run-task-definition" - for the life of me I can't remember why I chose such a crazy name! Note I am piping the result through jq as per normal and then onwards through sed to get the task id without all the arn information that precedes it. The sed command looks for the final forward slash in the task arn and crops anything before it.

# Run the task and parse the task ARN for polling later to determine its status
TASK=`aws ecs run-task \
	--cluster "default" \
	--task-definition "first-run-task-definition" \
	--network-configuration "awsvpcConfiguration={subnets=[${SUBNET}],securityGroups=[${SECURITY}],assignPublicIp=ENABLED}" \
	--launch-type FARGATE  | jq .tasks[0].taskArn -r | sed "s/.*\///" `
 
# Let's inspect what we got back. There should be a task identifier.
if [[ $? -ne 0 ]]; then
	echo "task-run failed with code $?"
	exit $?
fi
 
if [[ -z "$TASK" ]]; then
	echo "task-run could not create task"
	exit 1
fi

Loop and Wait for Fargate Task to get to RUNNING Status

Next I have to wait for the task to acquire RUNNING status, However I can't wait for ever, and thus I have wrapped a shell script rule into a timeout mechanism. I have allowed for 5 minutes for the task to get to RUNNING status and that should be more than enough.

# Loop around until we see a RUNNING status or timeout after 10 x 30 second sleeps
for i in 1 2 3 4 5 6 7 8 9 10 11
do
	if [[ $i -eq 11 ]]; then
		echo "Timed out before could establish task is running"
		exit 1
	fi
 
	STATUS=`aws ecs describe-tasks --cluster="default" --tasks $TASK  | jq .tasks[0].lastStatus -r`
 
	# Did it exit unexpectedly?
	if [[ $? -ne 0 ]]; then
		echo "describe-tasks failed with code $?"
		exit $?
	fi
 
	if [[ "$STATUS" = "RUNNING"  ]]; then
		break
	fi
 
	sleep 30
done

Obtain the Network Interface Id

The ENI is available in the Fargate task's description so I needed to parse this on my quest to get the public IP.

# Get the network Interface id
ENI=`aws ecs describe-tasks --cluster="default" --tasks $TASK  | jq '.tasks[0].attachments[0].details[] | select(.name == "networkInterfaceId") | .value' -r `
 
# Let's inspect what we got back. There should be a network interface id.
if [[ $? -ne 0 ]]; then
	echo "describe-tasks failed with code $?"
	exit $?
fi
 
if [[ -z "$ENI" ]]; then
	echo "describe-tasks could not establish eni"
	exit 1
fi

Get the Public IP Address

With the ENI it is possible to parse the the descriptions of all the network interfaces for the public IP address.

PUBLIC_IP=`aws ec2 describe-network-interfaces | jq --arg ENI "$ENI" '.NetworkInterfaces[] | select(.NetworkInterfaceId == $ENI) | .PrivateIpAddresses[0].Association.PublicIp' -r`
 
# Did we get an IP address?
if [[ $? -ne 0 ]]; then
	echo "describe-network-interfaces failed with code $?"
	exit $?
fi
 
if [[ -z "$PUBLIC_IP" ]]; then
	echo "describe-network-interfaces could not retrieve public IP address"
	exit 1
fi

Get the Parent Domain's Hosted Zone Id

The shell script needs to know about the parent domain and the subdomain name (which in my case is a feature branch name) - so those need to be retrieved from runtime arguments. This will go at the top of the shell script.

if [[ -z $1 ]] || [[ -z $2 ]]; then
	echo "usage: parent_domain_name feature_branch_name"
	exit 1
fi
 
BRANCH=$2.$1
PARENT=$1.

Then continuing at the end of the codebase, the following gets the parent domain

PARENT_ZONE=`aws route53 list-hosted-zones-by-name | jq --arg PARENT "$PARENT" '.HostedZones[] | select(.Name == $PARENT) | .Id' -r | sed "s/.*\///"`

Add the Feature Branch Name and Public IP Address to the Parent Hosted Zone

Now I created a new record set for the parent domain's hosted zone. This uses the AWS CLI command change-resource-record-sets which allows for insert, upsert and delete. I needed a DNS A record adding. The easiest way of providing the runtime parameters is via a JSON structure and that is created in shell script heredoc format to simplify shell script escaping.

BATCH=$(cat <
)
 
 
# Use the hosted zone of the tld
aws route53 change-resource-record-sets \
	--hosted-zone-id $PARENT_ZONE \
	--change-batch "$BATCH"

Create the Feature Branch Hosted Zone

The final AWS CLI call is to use create-hosted-zone, and here I needed to provide the feature branch name and an arbitrary unique caller reference for which I used a timestamp.

timestamp=$(date +%s)
 
# Now create a new hosted zone of the feature branch
aws route53 create-hosted-zone \
	--name "$BRANCH" \
	--caller-reference "$timestamp"

The Complete Script

The complete script is listed below. Note that for brevity and easy reading I have cut a few coding standards. There is repetition in the error condition checking. This should be rewritten with calls to shell script functions in a separate file.

#!/bin/bash
 
# Script to assign a public IP address issued by a Fargate task to a subdomain
 
if [[ -z $1 ]] || [[ -z $2 ]]; then
	echo "usage: parent_domain_name feature_branch_name"
	exit 1
fi
 
BRANCH=$2.$1
PARENT=$1.
 
 
# Get the default VPC.
VPC=`aws ec2 describe-vpcs | jq '.Vpcs[] | select(.IsDefault == true) | .VpcId' -r`
 
# Get the first subnet of the VPC
SUBNET=`aws ec2 describe-subnets | jq --arg VPC "$VPC" '.Subnets[] | select(.VpcId == $VPC) | .SubnetId' -r | head -n 1`
 
# Check if our security group for port 80 webserver already exists. Search for our funky unique description
SECURITY=`aws ec2 describe-security-groups | jq ' .SecurityGroups[] | select (.Description == "Ingress Port 80 Anywhere") | .GroupId' -r`
 
# Let's inspect what we got back.
if [[ $? -ne 0 ]]; then
	echo "describe-security-groups failed with code $?"
	exit $?
fi
 
# If we didn't get a security group then create
if [[ "$SECURITY" = "" ]]; then
	SECURITY=`aws ec2 create-security-group \
		--description "Ingress Port 80 Anywhere" \
		--group-name "Fargate Webserver Port 80" \
		--vpc-id "$VPC" | jq .GroupId -r `
 
 
	aws ec2 authorize-security-group-ingress \
		--group-id "$SECURITY" \
		--ip-permissions "FromPort=80,ToPort=80,IpProtocol=TCP,IpRanges=[{CidrIp=0.0.0.0/0}]"
fi
 
# Run the task and parse the task ARN for polling later to determine its status
TASK=`aws ecs run-task \
	--cluster "default" \
	--task-definition "first-run-task-definition" \
	--network-configuration "awsvpcConfiguration={subnets=[${SUBNET}],securityGroups=[${SECURITY}],assignPublicIp=ENABLED}" \
	--launch-type FARGATE  | jq .tasks[0].taskArn -r | sed "s/.*\///" `
 
# Let's inspect what we got back. There should be a task identifier.
if [[ $? -ne 0 ]]; then
	echo "task-run failed with code $?"
	exit $?
fi
 
if [[ -z "$TASK" ]]; then
	echo "task-run could not create task"
	exit 1
fi
 
# Loop around until we see a RUNNING status or timeout after 10 x 30 second sleeps
for i in 1 2 3 4 5 6 7 8 9 10 11
do
	if [[ $i -eq 11 ]]; then
		echo "Timed out before could establish task is running"
		exit 1
	fi
 
	STATUS=`aws ecs describe-tasks --cluster="default" --tasks $TASK  | jq .tasks[0].lastStatus -r`
 
	# Did it exit unexpectedly?
	if [[ $? -ne 0 ]]; then
		echo "describe-tasks failed with code $?"
		exit $?
	fi
 
	if [[ "$STATUS" = "RUNNING"  ]]; then
		break
	fi
 
	sleep 30
done
 
 
# Get the network Interface id
ENI=`aws ecs describe-tasks --cluster="default" --tasks $TASK  | jq '.tasks[0].attachments[0].details[] | select(.name == "networkInterfaceId") | .value' -r `
 
# Let's inspect what we got back. There should be a network interface id.
if [[ $? -ne 0 ]]; then
	echo "describe-tasks failed with code $?"
	exit $?
fi
 
if [[ -z "$ENI" ]]; then
	echo "describe-tasks could not establish eni"
	exit 1
fi
 
 
# Get the Public IP Address
PUBLIC_IP=`aws ec2 describe-network-interfaces | jq --arg ENI "$ENI" '.NetworkInterfaces[] | select(.NetworkInterfaceId == $ENI) | .PrivateIpAddresses[0].Association.PublicIp' -r`
 
# Did we get an IP address?
if [[ $? -ne 0 ]]; then
	echo "describe-network-interfaces failed with code $?"
	exit $?
fi
 
if [[ -z "$PUBLIC_IP" ]]; then
	echo "describe-network-interfaces could not retrieve public IP address"
	exit 1
fi
 
BATCH=$(cat <
)
 
PARENT_ZONE=`aws route53 list-hosted-zones-by-name | jq --arg PARENT "$PARENT" '.HostedZones[] | select(.Name == $PARENT) | .Id' -r | sed "s/.*\///"`
 
aws route53 change-resource-record-sets \
	--hosted-zone-id $PARENT_ZONE \
	--change-batch "$BATCH"
 
 
timestamp=$(date +%s)
 
# Now create a new hosted zone of the feature branch
aws route53 create-hosted-zone \
	--name "$BRANCH" \
	--caller-reference "$timestamp"

Invoking the Script

The script requires two parameters as previous discussed. Here's my example for the blog:

$ ./feature-branch.sh saasidate.com feature--doi-746-my-whizzy-new-feature

Currently the output of the script is the response from the AWS CLI route53 create-hosted-zone call, and here you can detect the full domain name.

Checking the Outcome

Obviously the first check is to copy and paste the url of the feature branch into a browser and see if it loads. An example of this is the blog heading image at the top of the page.

Also the AWS console is the place to check everything is as required. Navigate to ECS -> default -> Tasks and you should see a screenshot similar to the first one immediately above. This is the task we invoked from inside our shell script. Now navigate to Route 53 -> Hosted Zones -> {domain name} and you can see the feature branch and its IP address.

Steps left to do

New hosted zones will normally propagate in 60 seconds in Route 53 - but it would be good if the script above polls Route 53 every 10 seconds or so and reports back the subdomain url and the IP address once the propagation has completed. This should be surfaced on the command line as a minimum, but perhaps a notification on Slack would be even better. Both options are trivial.

Whilst the idea is that the Fargate tasks should persist, they shouldn't be around forever. Therefore there needs to be clear up activities to remove the feature subdomain from the parent domain's hosted zone, and the subdomain hosted zone. Also there obviously needs a step to stop the Fargate task.

Caveats

There are caveats to this solution - but for my use case, it's one of those rare occurrences in life that the caveats don't apply to me.

Firstly, this solution will only work with a Fargate task and not an ECS service running a Fargate task. Tasks should be run as services for production environments since services give you great things like replication, and health percentages against number of running tasks. There is therefore no scaling, no load balancing, no DDOS protection in what I'm offering here.

Fargate tasks run in isolation are perfect for spun up short living environments such as playgrounds for devs and QAs to run manual tests against or troubleshoot development issues - which is exactly my use case.

blog terms

AWS AWS CLI Docker devops Fargate ECS Development Route 53 bash