Drupal 9

Hosting Drupal 9 on AWS Fargate Part 2: Bitbucket Pipeline

nigel — Sun, 17 Oct 2021 08:31:46 +0000

Hosting Drupal 9 on AWS Fargate Part 2: Bitbucket Pipeline nigel Sun, 17/10/2021 - 09:31

Introduction

In the first instalment of this sequence of blogs we talked about how imperative it is to create an industrial strength local dev environment when AWS Fargate is your target deployment platform. In this blog we discuss something equally important - the deploy and build process. This needs to be automated for Drupal since security patches are issued on a weekly basis by the Drupal Security Group, and an automated solution is the only way to keep on top of these updates.

Over the last two years I have become accustomed to using Atlassian's Bitbucket Pipelines, and I am really liking this CI/CD delivery mechanism. It is particularly suited to containerised solutions since the pipeline itself runs in a Docker container, and often times the solution Docker image can be used for the build process - case in point this project! This tutorial will show how the official Drupal PHP Docker container can be used as the pipeline for delivering Drupal 9 projects to AWS Fargate. Let's see how!

Bitbucket Pipeline

The diagram above shows the basic steps the pipeline follows. Let's have a look at these in turn with narrative and then see how that transposes into the Pipeline code.A few trivial housekeeping activities have been omitted from the diagram for clarity but will be discussed when we look at the code.

Since my Badzilla blog is a hobby project, I don't have multiple environments, and merge all new features into the git master branch. Any merge into master triggers the build process. Commercial pipelines tend to trigger builds on feature branch commits too which may be a consideration depending upon your circumstances.
I've already mentioned that the Docker image selected for the pipeline will be the official Drupal / PHP release - and thus contains composer which is used to build out the codebase.
Once the codebase has been built the testing can be undertaken. I have identified two parallel steps in the pipeline - automated testing and static testing. I have covered the configuration for these in earlier blogs - so click through for further reading.
The next steps involve building the two Docker image artefacts we need for our Fargate bundles - the PHP / FPM / Codebase image, and the NGINX Web Server image. The latter is required since we need to load a configuration for Drupal 9 and for serving the S3 / CloudFront assets.
The artefacts are pushed into AWS ECR after being tagged latest.
We are now ready to deploy the images to our ECS Cluster - but before we do this, we provide the opportunity to run any pre-deploy hooks. This would typically be activities such as drush putting the web app into maintenance mode to prevent public access during updates.
One the pre-deploy scripts have completed, the deploy script is invoked. This forces an ECS service update on the cluster. The script loops to determine whether the update fails or completes, and upon a successful completion will move onto the final step.
The post-deploy scripts are now run. These would be further drush commands to update the database schema for example, and to remove the maintenance mode flag.

Pipeline Code

The pipeline code is saved in bitbucket-pipelines.yml and is show below.

image:
  name: drupal:9.2.6-php7.4-fpm
 
clone:
  depth: full
 
definitions:
  services:
    docker:
      memory: 2048
 
  steps:
    - step: &build-environment
        name: Build Environment
        script:
          # Install git & unzip for composer
          - apt-get update
          - apt-get install git unzip -y
          # Install dependencies
          - export COMPOSER_ALLOW_SUPERUSER=1
          - composer install
          # Copy project settings file in place
          - cp storage/settings.php web/sites/default/settings.php
          # Remove all non-essential dirs and files from top level directory
          - rm -rf assets backup databases .git
          - rm Makefile README.md docker-compose.yml composer.json composer.lock .env .csslintrc .eslintignore .eslintrc.json
          - rm .gitignore
          # Add the bind mount VOLUME so in Fargate the nginx container can do a VOLUME_FROM and load /opt/drupal docroot
          # from the core-drupal container
          - sed -i '/^FROM.*/a VOLUME ["/opt/drupal"]' Dockerfile > /dev/null 2>&1
        caches:
          - composer
        artifacts:
          - '**'
 
    # Automated Tests placeholder
    - step: &automated-tests
        name: Automated Tests
        script:
          - echo "Add automated tests here"
 
    # Static Analysis tests placeholder
    - step: &static-tests
        name: Static Analysis Tests
        script:
          - echo "Add static analysis here"
 
pipelines:
  # Run when code is merged / pushed into specific branches.
  branches:
    master:
      - step: *build-environment
 
      - parallel:
          - step: *automated-tests
          - step: *static-tests
 
      - step:
          name: Build & Deploy
          deployment: AWS Prod
          script:
            # Install AWS CLI
            - apt update && apt-get install -y jq unzip python3 python-dev python3-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev python-pip
            - curl "https://s3.amazonaws.com/aws-cli/awscli-bundle-1.18.200.zip" -o "awscli-bundle.zip"
            - unzip awscli-bundle.zip
            - ./awscli-bundle/install -b ~/bin/aws
            - export PATH=~/bin:$PATH
            # Build the codebase + FPM
            - docker build -t badzilla-core-drupal .
            # Authenticate with ECR
            - aws configure set aws_access_key_id "${AWS_ACCESS_KEY}"
            - aws configure set aws_secret_access_key "${AWS_SECRET_KEY}"
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${CORE_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-core-drupal:latest "${CORE_DRUPAL_REPO_URL}"
            - docker push "${CORE_DRUPAL_REPO_URL}"
            # NGINX_DRUPAL_REPO_URL
            - cd docker/nginx/fargate
            - docker build -t badzilla-nginx-drupal .
            - cd -
            # Authenticate with ECR
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${NGINX_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-nginx-drupal:latest "${NGINX_DRUPAL_REPO_URL}"
            - docker push "${NGINX_DRUPAL_REPO_URL}"
            # Pre deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/pre-deploy/
            - cd -
            # Deploy script
            - ./deploy-scripts/ecs-service-deploy.sh
            # Post deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/post-deploy/
            - cd -
          services:
            - docker
          caches:
            - docker

Now let's inspect the code per section. Firstly the declaration of the Docker image to use for the pipeline.

image:
  name: drupal:9.2.6-php7.4-fpm

You'll note that I'm using the same image as I used for my development environment with one small change. In the pipeline I have abandoned the use of the ultra lightweight Alpine image - the image being used is standard Ubuntu. Why so? Firstly there is less necessity for keeping the image small since Atlassian don't charge for storage of the image, and secondly I will want to add additional Linux packages to this image for the pipeline process, and thus I want to use a package manager I am familiar with - apt.

clone:
  depth: full
 
definitions:
  services:
    docker:
      memory: 2048

Here I'm saying I want a full clone of the repository in the pipeline. This is a little unnecessary and causes an overhead. In the future I will consider a more shallow clone for greater performance. The Docker memory setting should provide enough headroom for Drupal builds.

  steps:
    - step: &build-environment
        name: Build Environment
        script:
          # Install git & unzip for composer
          - apt-get update
          - apt-get install git unzip -y
          # Install dependencies
          - export COMPOSER_ALLOW_SUPERUSER=1
          - composer install

Steps are defined in the pipeline that are invoked later in the script. This step &build-environment will build out the environment by installing the packages I will need and then running composer install to build out the codebase.

          # Copy project settings file in place
          - cp storage/settings.php web/sites/default/settings.php
          # Remove all non-essential dirs and files from top level directory
          - rm -rf assets backup databases .git
          - rm Makefile README.md docker-compose.yml composer.json composer.lock .env .csslintrc .eslintignore .eslintrc.json
          - rm .gitignore

Every Drupal site needs a settings.php file. I decided to have mine saved in the repository in the directory storage since there is no confidential data. All settings use environmental variables. Your mileage may vary here - other strategies are possible for injecting the settings.php into the build.

I then delete files and directories from the build which won't be needed in a production environment.

          # Add the bind mount VOLUME so in Fargate the nginx container can do a VOLUME_FROM and load /opt/drupal docroot
          # from the core-drupal container
          - sed -i '/^FROM.*/a VOLUME ["/opt/drupal"]' Dockerfile > /dev/null 2>&1

This snippet is a kludge to fix a difference between the local environment docker-compose and the production Fargate Task Definition. Fargate shared volumes use bind mounts which are created by initially using the VOLUME instruction in the Dockerfile and settings in the Task Definition (to be discussed in a later blog). Here I use interactive sed to inject the VOLUME into the Dockerfile

        caches:
          - composer
        artifacts:
          - '**'

Closing the build step I ensure that Docker is cached to save time next time the pipeline is invoked and the artifact (American spelling) setting tells the pipeline to save the all my work in that pipeline step for steps that follow it.

    # Automated Tests placeholder
    - step: &automated-tests
        name: Automated Tests
        script:
          - echo "Add automated tests here"
 
    # Static Analysis tests placeholder
    - step: &static-tests
        name: Static Analysis Tests
        script:
          - echo "Add static analysis here"
 
pipelines:
  # Run when code is merged / pushed into specific branches.
  branches:
    master:
      - step: *build-environment
 
      - parallel:
          - step: *automated-tests
          - step: *static-tests

Here I have defined the testing steps (both of which are currently placeholders), and then started the 'implementation' section of the code. The steps will be invoked on the master branch, initially the build and then the the two testing steps in parallel.

      - step:
          name: Build & Deploy
          deployment: AWS Prod
          script:
            # Install AWS CLI
            - apt update && apt-get install -y jq unzip python3 python-dev python3-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev python-pip
            - curl "https://s3.amazonaws.com/aws-cli/awscli-bundle-1.18.200.zip" -o "awscli-bundle.zip"
            - unzip awscli-bundle.zip
            - ./awscli-bundle/install -b ~/bin/aws
            - export PATH=~/bin:$PATH

Once testing has completed successfully we commence the inline deploy step. We are going to need the AWS CLI installing here since the CLI is necessary to issue the correct sequence of commands to AWS to commit and deploy.

Note I am also installing the amazing utility jq - a JSON query language utility. This is imperative for inspecting the output of the CLI commands since their responses are sent in JSON format.

            # Build the codebase + FPM
            - docker build -t badzilla-core-drupal .
            # Authenticate with ECR
            - aws configure set aws_access_key_id "${AWS_ACCESS_KEY}"
            - aws configure set aws_secret_access_key "${AWS_SECRET_KEY}"
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${CORE_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-core-drupal:latest "${CORE_DRUPAL_REPO_URL}"
            - docker push "${CORE_DRUPAL_REPO_URL}"

Now I've got all I need installed I can build the Docker image artefacts. Firstly I build out the Drupal codebase image which also contains PHP and FPM. I then authenticate with AWS ECR where I will push the artefact once its been tagged.

            # NGINX_DRUPAL_REPO_URL
            - cd docker/nginx/fargate
            - docker build -t badzilla-nginx-drupal .
            - cd -
            # Authenticate with ECR
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${NGINX_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-nginx-drupal:latest "${NGINX_DRUPAL_REPO_URL}"
            - docker push "${NGINX_DRUPAL_REPO_URL}"

I now repeat the process for the NGINX web server Docker image. Note I have the configuration for this in my codebase repo under the directory docker/nginx/fargate

            # Pre deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/pre-deploy/
            - cd -
            # Deploy script
            - ./deploy-scripts/ecs-service-deploy.sh
            # Post deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/post-deploy/
            - cd -

This is the fun part! I now run the deploy shell scripts which are listed with narratives later in the blog. Firstly I run the pre-deploy commands, then the deploy commands and the post-deploy commands. All will be revealed...

          services:
            - docker
          caches:
            - docker

To use Docker commands in a pipeline, you need to list Docker under the services setting. Installing Docker in the pipeline itself and then invoking it will not work! I tried and failed! Note I am also using caching for greater performance.

Repository Environmental Variables

The pipeline and the shell scripts heavily use environmental variables that are set in the repository in Bitbucket. Their names are self-evident and listed in their entirety above.

Pre- and Post-deploy shell script

The pre- and post-deploy shell script is defined below. Currently it only works on drush commands but could be extended to include shell scripts easily enough. The idea is it scans the directory passed as a parameter and it looks for lines in any file that starts with the word drush. It then takes the runtime arguments of that drush command and converts it into a Docker cmd syntax. Once that is done the command is executed as a Fargate task using the codebase image with parameter overrides. This task runs to completion and exits. The status of the command can be captured and reported back to the pipeline. Obviously this isn't as quick as running a drush command in a terminal shell since the Fargate task needs to be provisioned which takes time. It is however a fiendish way of reusing an existing Docker image since the official Drupal Docker image already contains the drush command.
deploy-scripts/deploy-parser.sh

#!/bin/bash
 
# Parse the passed files and issue commands in accordance.
# Currently only supports drush commands.
# This can be expanded out to other shell scripts.
 
# Output directory to be parsed
echo "Directory: $1 to be parsed"
 
set_network_configuration()
{
NETWORK=$(cat <
)
echo "${NETWORK}"
}
 
 
 
set_overrides_command()
{
OVERRIDES=$(cat <
)
echo "${OVERRIDES}"
}
 
 
 
# Process each line in a selected file
parse_line()
{
    # All lines to be processed must start with the word drush to be considered
    # This grep / sed combination will turn the drush command into docker cmd syntax
    grep -e "^drush" < "${1}" | while read -r line ; do
        cmd=`echo $line | sed -r 's/[^ ][^ ]*/"&"/g' | sed -r 's/\s+/,/g' | sed -r 's/drush/vendor\/bin\/drush/'`
 
        # @BUG with quoting in dotenv and how I load into environment. Appears to add newlines making it difficult to add quotes later.
        # So already added quotes for $CORE_TASK_DEFINITION_NAME in .env file but need the unquoted version in
        # --task-definition flag so unquote here
        TASK_DEF_UNQUOTE=`echo ${CORE_TASK_DEFINITION_NAME} | sed s/\"//g`
 
        # Now execute the command to start the container with the drush cmd
        FARGATE_TASK=`aws ecs run-task \
            --cluster "${ECS_CLUSTER_NAME}" \
            --task-definition "${TASK_DEF_UNQUOTE}" \
            --network-configuration "$(set_network_configuration  ${ECS_SUBNET_1} ${ECS_SUBNET_2}  ${ECS_SECURITY_GROUP})" \
            --launch-type FARGATE \
            --platform-version '1.4.0' \
            --region "${AWS_DEFAULT_REGION}" \
            --overrides "$(set_overrides_command ${CORE_TASK_DEFINITION_NAME} ${cmd})" | jq .tasks[0].taskArn -r`
 
        if [[ $? != 0 ]]; then
            echo "Fargate Run Task Failed."
            exit $?
        elif [[ -z  "${FARGATE_TASK}" ]]; then
            echo "Unspecified Fargate Run Task Failure."
            exit 1
        else
            echo "Cmd run is ${cmd}"
        fi
 
        # Loop round to check on the status of the task. Can't run in parallel so have to wait for each task to finish
        while true; do
            FARGATE_STATUS=`aws ecs describe-tasks --cluster "${ECS_CLUSTER_NAME}" --tasks "${FARGATE_TASK}"  | jq .tasks[0].lastStatus -r`
 
            if [[ -z "${FARGATE_STATUS}" ]]; then
                echo "Unspecified Fargate Describe Task Failure."
                exit 1
            fi
 
            echo "${FARGATE_STATUS}"
 
            if [[ "${FARGATE_STATUS}" == "STOPPED" ]] || [[ "${FARGATE_STATUS}" == "DEPROVISIONING" ]]; then
                  break
            fi
 
            sleep 20
        done
 
	echo "Drush command completed"
    done
}
 
 
# Get a list of all the files in the directory and process
parse_directory()
{
    # loop though all the files in the directory
    for file in "${1}"/*; do
        base="$(basename ${file})"
        echo "${base} being parsed"
        parse_line "${file}"
    done
}
 
parse_directory "${1}"

ECS Deploy Script

The main ECS deploy shell script uses AWS CLI ecs update-service to force a new deployment. This means that when the Fargate bundles are recycling any new codebase and NGINX Docker image will be used from AWS ECR. The process can take a few minutes so I have added a loop facility which uses AWS CLI ecs describe-services to capture the current deployment status and act accordingly. Note I am using the previously mentioned jq utility to interrogate the long JSON response from the CLI and whittle it down to the status field which should contain the word "COMPLETED" or "FAILED" or "IN_PROGRESS".
deploy-scripts/ecs-service-deploy.sh

#!/bin/bash
 
 
# Use the aws cli to force deploy Fargate task running in ECS service
aws ecs update-service --cluster "${ECS_CLUSTER_NAME}" --service "${ECS_SERVICE_NAME}" --force-new-deployment --region "${AWS_DEFAULT_REGION}"
 
# Give it a few seconds to take effect
sleep 5
 
# Get the deployment status by looping over the deployment
# Try for a maximum of 20 minutes then give up. Chances are after 20 minutes of incomplete
# deployment there is a problem in the task definition
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
do
    if [[ "${i}" -eq 21 ]]; then
        echo "Timed out before could determine if deployment succeeded"
        exit 1
    fi
 
    STATUS=`aws ecs describe-services --cluster "${ECS_CLUSTER_NAME}" --services "${ECS_SERVICE_NAME}" --region "${AWS_DEFAULT_REGION}" | jq .services[0].deployments[0].rolloutState -r`
 
    # Was it successful?
    if [[ "${STATUS}" = "COMPLETED"  ]]; then
        echo "Deployment succeeded"
        exit 0
    # Anything we need to be concerned about?
    elif [[ "${STATUS}" = "FAILED"  ]]; then
        echo "Deployment failed"
        exit 2
    # If still deploying echo this
    elif [[ "${STATUS}" = "IN_PROGRESS"  ]]; then
        echo "Deployment in progress"
    else
        echo "Unknown status "${STATUS}" so quitting"
        exit 3
    fi
 
    # Wait a minute before retrying
    sleep 60
done

Ad hoc Drush Commands

The solution provides the capability to run drush during the pre- and post-deploy scripts. But what about running drush commands on an ad hoc basis from a remote laptop? This is often required by developers or devops and would be crucial for the management of a Drupal 9 website. Stay tuned for a future blog on how to achieve this!

blog terms

AWS AWS CLI devops PHP Drupal Drupal 9

Hosting Drupal 9 on AWS Fargate Part 1: Local Development

nigel — Sun, 19 Sep 2021 08:46:14 +0000

Hosting Drupal 9 on AWS Fargate Part 1: Local Development nigel Sun, 19/09/2021 - 09:46

Introduction

Welcome to a series of blogs on hosting Drupal 9 (or legacy Drupal 8) sites on AWS Fargate. Fargate can now be considered a mature product, and provides a simple serverless containerised platform for hosting web apps. It offers considerable savings over traditional Virtual servers without, if configured correctly, loss of performance. Furthermore it offers greater simplicity of operation and setup then its AWS sibling offering ECS over EC2, and again a cost saving.

The blogs in this series will cover important considerations such as the pipelines to deploy your codebase within Docker images, and their storage; set up of the Drupal S3FS module along with the CloudFront accelerator and S3 object store; auto scaling of the Fargate tasks; load testing your app to ensure you are fully optimised; and how to run your Drush commands in short living Fargate tasks.

This first blog will cover your team's local environment and how its build will ensure your path to Fargate deploy is smooth, with as much reuse of the configuration in production as possible.

Objective

The objective is to stand up a Fargate task running as a service which will be the web app, and it will persist and auto scale. Also have Fargate tasks to run which will run Drush commands, stood up on a per Drush command basis, and terminating after each Drush command completes. The diagram above shows the Docker images requires to achieve our goal.

Drupal Core + FPM PHP + Codebase

The intention is to use the official Drupal FPM Docker image using Alpine, and be as up to date as possible with the versioning. Note that the Alpine build has been chosen since it's an extremely lightweight version of Linux, based on busybox which has a very small footprint. This comes at a price - those with Ubuntu experience will have to learn new commands. For instance Alpine uses the apk package manager instead of apt-get. The web app's codebase - ie all the configuration and custom modules and themes created by the dev team - will be copied into this image.

Drupal Core + FPM PHP + Codebase + Exec command

This will be the Fargate task to run one off Drush commands. Since the codebase will already contain Drush binaries in vendor/bin/drush, it will be the same as the previously mentioned image. Notwithstanding this, it won't persist. It will use the Fargate equivalent of the Docker command to exec a particular command and then quit.

Memcached

I have elected to run individual Memcached caching in each task running in the Web App service. Memcached will be used for caching the Drupal MySQL cache tables and for page caching for anonymous users. There is an alternative strategy to this - centralise the caching using for instance the AWS ElastiCache with Memcached product. Each task in the web service would have an endpoint to the same centralised ElastiCache instance. This would be the preferred option if your website is largely serving logged in traffic, and if you are running an eCommerce site with the users' baskets stored in Memcached. Since that doesn't apply to my own site, and I would like to keep costs down too, my feeling is having Memcached as part of the Fargate task is a better fit.

nginx

A web server is needed, and I've elected for nginx although Apache would also be a good choice. My rationale for selecting nginx is I would like to add the feature of nginx serving slightly stale cached pages when the origin is returning 5xx errors. This would give a seamless switch-over when a task becomes unhealthy - during the period AWS determines that a task is returning errors and has crossed the unhealthy threshold, nginx could still serve pages which would be less than 5 minutes old. The task would then be drained and removed from the pool with no problem noticeable to the end user. This will be the subject of a blog at a later date.

Using a local Virtual Machine

Those who know me well will know I am not an advocate of using my laptop's OS for development purposes. The reason being that with multiple clients and with multiple projects in each, my host OS would quickly become a mess of different development tooling and versions which could collide and could also generate inconsistencies with other devs in the team should say different versions of bin files end up in the $PATH.

My ethos has always been do not pollute your host OS! Always use VMs for development!

I have created my own VM for Docker based implementations. My belief is if the target deploy is using Docker containers then the local development environment should match this as much as is practicable. In fact it's impossible to match the ECS/Fargate environment exactly, but it is certainly possible to use Docker with docker-compose to simulate closely what will be seen in prod.

My VM can be found in GitHub by clicking on this link. It is a very lightweight VM and can be orchestrated easily using Vagrant and Ansible. PHP and MySQL are deliberately not included in the playbook since all development work will be undertaken by using Docker containers.

All my localhost config examples in this blog will have come from using my DockerVM mentioned above. If you intend to follow this blog closely, you will need to install it, but read the instructions in the README.md carefully!

Composer

Once you've got a lightweight VM up and running (hopefully mine), and ssh'd into it, and cloned your repo, you will need to build out the dependencies using composer install. So how do we go about this when there is no composer and no PHP in the VM? There are two choices:

Use the official Docker Composer image
Use the latest official Docker Drupal image (9.2.6-php7.4-fpm-alpine.3.14 at time of writing this blog)

I chose the official Docker Composer image purely because (a) there was more documentation on it readily available, and (b) I wasn't sure at the time whether the Drupal image bundled Composer. It does, and it will be used in the Bitbucket Pipeline later so you may want to select that instead for consistency, but if you do, you'll have to change my composer commands slightly.

Ok so let's build our composer image.

$ docker run --rm --interactive --tty --volume $PWD:/app composer install --ignore-platform-reqs
Unable to find image 'composer:latest' locally
latest: Pulling from library/composer
a0d0a0d46f8b: Pull complete 
153eea49496a: Pull complete 
11efd0df1fcb: Pull complete 
b3f3214c344d: Pull complete 
9abd2f85688c: Pull complete 
83d85b95eb4c: Pull complete 
923d73ddadfa: Pull complete 
711b5c4b02a7: Pull complete 
ee08fa481788: Pull complete 
5f9f812bf61d: Pull complete 
28b3ecec72d5: Pull complete 
a5ffb14bc4f9: Pull complete 
d92aff3e5aec: Pull complete 
cd0274c84e75: Pull complete 
Digest: sha256:33d2c1d9fb7311cb8f215a8f52669894ef534292ac5d331347f929c27e317bb1
Status: Downloaded newer image for composer:latest
> DrupalProject\composer\ScriptHandler::checkComposerVersion
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
  - Downloading composer/installers (v2.0.1)
<---- SNIPPED ---->
  - Copy [web-root]/profiles/README.txt from assets/scaffold/files/profiles.README.txt
  - Copy [web-root]/themes/README.txt from assets/scaffold/files/themes.README.txt
> DrupalProject\composer\ScriptHandler::createRequiredFiles
Created a sites/default/files directory with chmod 0777

Success! Note that there is a requirement to use the composer flag --ignore-platform-reqs since it expects ext-gd locally which obviously we don't have.

docker-compose: Codebase + FPM

In preparation for creating a docker-compose file we'll need to build out an image containing the latest Drupal 9 Docker image + our codebase once composer install has been run. This will need a Dockerfile, and we can add APCu caching in the Dockerfile for greater Drupal performance. I also noted that Memcached API library needs to be added to Dockerfile since it is my intention to connect to a Memcached server for cache management in the container bundle. The Dockerfile is listed below.

FROM drupal:9.2.6-php7.4-fpm-alpine3.14
 
# Copy the artefact (on Bitbucket) or local files into the image
COPY ./ /opt/drupal/.
 
# PHP Memcached which needs the libmemcached API
# Bug in Apline - creates extensions in the wrong directory so add them in correct location
RUN apk add php7-igbinary php7-pecl-memcached libmemcached \
    && echo "extension=memcached.so" > /usr/local/etc/php/conf.d/docker-php-ext-memcached.ini \
    && echo "extension=igbinary.so" > /usr/local/etc/php/conf.d/docker-php-ext-igbinary.ini \
    && mv /usr/lib/php7/modules/memcached.so /usr/local/lib/php/extensions/no-debug-non-zts-20190902 \
    && mv /usr/lib/php7/modules/igbinary.so /usr/local/lib/php/extensions/no-debug-non-zts-20190902
 
# APCu cache
RUN apk add --update --no-cache --virtual .build-dependencies $PHPIZE_DEPS \
    && pecl install apcu \
    && docker-php-ext-enable apcu \
    && pecl clear-cache \
    && apk del .build-dependencies

docker-compose: nginx image

We will also need to build out an nginx image with the standard Drupal configuration. To do this I created a docker directory at the top level of the repo (i.e. a sibling of web directory) as such:

$ tree docker
docker
├── core-drupal
├── memcached
└── nginx
    ├── fargate
    │   ├── default.conf
    │   ├── Dockerfile
    │   └── nginx.conf
    └── local
        ├── default.conf
        ├── Dockerfile
        └── nginx.conf
 
5 directories, 6 files

You'll note that I have two sets of config files - one for AWS Fargate and one for local development. In fact the differences are minimal and will be discussed later. The local files are below
nginx.conf

user  nginx;
worker_processes  auto;
 
error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
 
 
events {
    worker_connections  1024;
}
 
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    fastcgi_buffers 16 16k;
    fastcgi_buffer_size 32k;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    #tcp_nopush     on;
 
    keepalive_timeout  65;
 
    #gzip  on;
 
    include /etc/nginx/conf.d/*.conf;
}

default.conf

  server {
    server_name badzilla-fargate.test;
    root /opt/drupal/web; ## <-- Your only path reference.
 
    location = /favicon.ico {
      log_not_found off;
      access_log off;
    }
 
    location = /robots.txt {
      allow all;
      log_not_found off;
      access_log off;
    }
 
    location ~ \..*/.*\.php$ {
      return 403;
    }
 
    location ~ ^/sites/.*/private/ {
      return 403;
    }
 
    # Block access to scripts in site files directory
    location ~ ^/sites/[^/]+/files/.*\.php$ {
      deny all;
    }
 
    # Allow "Well-Known URIs" as per RFC 5785
    location ~* ^/.well-known/ {
      allow all;
    }
 
    # Block access to "hidden" files and directories whose names begin with a
    # period. This includes directories used by version control systems such
    # as Subversion or Git to store control files.
    location ~ (^|/)\. {
      return 403;
    }
 
    location / {
      # try_files $uri @rewrite; # For Drupal <= 6
      try_files $uri /index.php?$query_string; # For Drupal >= 7
    }
 
    location @rewrite {
      #rewrite ^/(.*)$ /index.php?q=$1; # For Drupal <= 6
      rewrite ^ /index.php; # For Drupal >= 7
    }
 
    # Don't allow direct access to PHP files in the vendor directory.
    location ~ /vendor/.*\.php$ {
      deny all;
      return 404;
    }
 
    # Protect files and directories from prying eyes.
    location ~* \.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|/(\.(?!well-known).*)|Entries.*|Root|/#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$ {
      deny all;
      return 404;
    }
 
    # In Drupal 8, we must also match new paths where the '.php' appears in
    # the middle, such as update.php/selection. The rule we use is strict,
    # and only allows this pattern with the update.php front controller.
    # This allows legacy path aliases in the form of
    # blog/index.php/legacy-path to continue to route to Drupal nodes. If
    # you do not have any paths like that, then you might prefer to use a
    # laxer rule, such as:
    #   location ~ \.php(/|$) {
    # The laxer rule will continue to work if Drupal uses this new URL
    # pattern with front controllers other than update.php in a future
    # release.
    location ~ '\.php$|^/update.php' {
      fastcgi_split_path_info ^(.+?\.php)(|/.*)$;
      # Ensure the php file exists. Mitigates CVE-2019-11043
      try_files $fastcgi_script_name =404;
      # Security note: If you're running a version of PHP older than the
      # latest 5.3, you should have "cgi.fix_pathinfo = 0;" in php.ini.
      # See http://serverfault.com/q/627903/94922 for details.
      include fastcgi_params;
      # Block httpoxy attacks. See https://httpoxy.org/.
      fastcgi_param HTTP_PROXY "";
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
      fastcgi_param QUERY_STRING $query_string;
      fastcgi_intercept_errors on;
      # PHP 5 socket location.
      #fastcgi_pass unix:/var/run/php5-fpm.sock;
      # PHP 7 socket location.
#      fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
      fastcgi_pass fpm:9000;
    }
 
    # S3FS Module
    location ~ ^/s3/files/styles/ {
      try_files $uri @rewrite;
    }
 
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg)$ {
      try_files $uri @rewrite;
      expires max;
      log_not_found off;
    }
 
    # Fighting with Styles? This little gem is amazing.
    # location ~ ^/sites/.*/files/imagecache/ { # For Drupal <= 6
    #location ~ ^/sites/.*/files/styles/ { # For Drupal >= 7
    #  try_files $uri @rewrite;
    #}
 
    # Handle private files through Drupal. Private file's path can come
    # with a language prefix.
    location ~ ^(/[a-z\-]+)?/system/files/ { # For Drupal >= 7
      try_files $uri /index.php?$query_string;
    }
 
    # Enforce clean URLs
    # Removes index.php from urls like www.example.com/index.php/my-page --> www.example.com/my-page
    # Could be done with 301 for permanent or other redirect codes.
    if ($request_uri ~* "^(.*/)index\.php/(.*)") {
      return 307 $1$2;
    }
  }

Dockerfile

FROM nginx:1.21.1-alpine
COPY ./default.conf /etc/nginx/conf.d/
COPY ./nginx.conf /etc/nginx/

Some commentary is required here. You'll see in the default.conf file there is a snippet:

    # S3FS Module
    location ~ ^/s3/files/styles/ {
      try_files $uri @rewrite;
    }

This is because the assets are not served locally - they are stored in an AWS S3 bucket with the AWS CloudFront accelerator providing edge capabilities. I will provide a further blog at a later date on setting all this up but in essence, assets shouldn't be kept in individual Fargate tasks, so either an AWS S3 / CloudFront solution needs to be utilised or an AWS EFS drive.

Now let's see what the differences are between the local configuration and Fargate.

$ diff docker/nginx/local/default.conf docker/nginx/fargate/default.conf 
2c2
<     server_name badzilla-fargate.test;
---
>     server_name badzilla.co.uk www.badzilla.co.uk;
4a5,10
>     # User Agent ELB-HealthChecker sends no headers with the request so nginx by default will respond with a 400
>     # Fix with this
>     location = /health {
>         return 200;
>     }
> 
92c98
<       fastcgi_pass fpm:9000;
---
>       fastcgi_pass 127.0.0.1:9000;

Ok there are three differences. The first is obvious - I am using different server names dependent on the environment. The second difference is the AWS health checking capability that I have included for Fargate but not needed locally. I use the Drupal health_check module by the way, but the problem resides with the AWS Application Load Balancer health check feature. It sends requests to nginx without headers which nginx doesn't like - and always replies with a 400 response. This snippet fixes that problem. The third difference relates to how Fargate handles Docker networking. It is much simplified and all containers in the Fargate task use 127.0.0.1 for communication. Locally I am using a label named fpm for fastcgi_pass communication between nginx and FPM.

docker-compose: MariaDB and Memcached

The use of MariaDB is the most significant difference between the local environment and Fargate. Fargate connects to a MariaDB instance in AWS RDS, whilst the local MariaDB persists its data in the VM. I did consider using another RDS instance for local development, but that would require public access to the database which I wasn't keen to do for security reasons. In addition there is a cost implication which isn't viable for a hobby project like my blog. I am using the standard Docker images for both MariaDB and Memcached locally.

docker-compose Configuration

docker-compose

version: "3"
 
services:
  mariadb:
    image: "mariadb:${MARIADB_VERSION}"
    restart: 'always'
    container_name: mariadb-drupal
    volumes:
      - "/var/lib/mysql/data:${MARIADB_DATA_DIR}"
      - "/var/lib/mysql/logs:${MARIADB_LOG_DIR}"
      - "/var/docker/mariadb/conf:/etc/mysql"
    environment:
      MYSQL_ROOT_PASSWORD: "${MYSQL_ROOT_PASSWORD}"
      MYSQL_DATABASE: "${MYSQL_DATABASE}"
      MYSQL_USER: "${MYSQL_USER}"
      MYSQL_PASSWORD: "${MYSQL_PASSWORD}"
  drupal:
    image: core-drupal:latest
    container_name: core-drupal
    depends_on:
      - mariadb
      - memcached
    volumes:
      - ./:/opt/drupal
    restart: 'always'
    environment:
      MYSQL_DATABASE: "${MYSQL_DATABASE}"
      MYSQL_USER: "${MYSQL_USER}"
      MYSQL_PASSWORD: "${MYSQL_PASSWORD}"
      MYSQL_HOSTNAME: "${MYSQL_HOSTNAME}"
      MYSQL_NAMESPACE: "${MYSQL_NAMESPACE}"
      MYSQL_DRIVER: "${MYSQL_DRIVER}"
      MYSQL_PORT: "${MYSQL_PORT}"
      AH_SITE_ENVIRONMENT: "${AH_SITE_ENVIRONMENT}"
      S3FS_ACCESS_KEY: "${S3FS_ACCESS_KEY}"
      S3FS_SECRET_KEY: "${S3FS_SECRET_KEY}"
      S3FS_BUCKET: "${S3FS_BUCKET}"
      S3FS_AWS_REGION: "${S3FS_AWS_REGION}"
      S3FS_CLOUDFRONT_CNAME: "${S3FS_CLOUDFRONT_CNAME}"
      S3FS_USE_S3_PUBLIC: "${S3FS_USE_S3_PUBLIC}"
      S3FS_USE_CNAME: "${S3FS_USE_CNAME}"
      S3FS_USE_HTTPS: "${S3FS_USE_HTTPS}"
      S3FS_TWIG_PATH: "${S3FS_TWIG_PATH}"
 
  nginx:
    image: nginx-drupal:latest
    container_name: nginx-drupal
    links:
      - drupal:fpm
    ports:
      - "80:80"
      - "443:443"
    depends_on:
      - drupal
    restart: 'always'
    volumes:
      - ./:/opt/drupal
  memcached:
    image: memcached:alpine3.14
    restart: 'always'
    container_name: memcached-drupal
    ports:
      - "11211:11211"

Note I have all my settings held in the environment. I also use the vlucas/phpdotenv composer dependency so these environmental variables are accessible in Drupal through a dot env file.

Shortcuts

Remembering commands to build the Docker images, gain command line access to MariaDB, importing and exporting databases etc is an exercise in tedium I'd rather avoid. I have built a make file with shortcuts although command line aliases could also have been used.
Makefile

.DEFAULT_GOAL := help
 
 
help:
	@echo "See the README.md for available options"
 
exportdb: set-env
	@echo "Exporting the database"
	docker exec -i mariadb-drupal bash -c 'mysqldump -u $$MYSQL_USER -p$$MYSQL_PASSWORD $$MYSQL_DATABASE' > $(file)
 
importdb: set-env
	@echo "Importing the database"
	docker exec -i mariadb-drupal bash -c 'mysql -u $$MYSQL_USER -p$$MYSQL_PASSWORD $$MYSQL_DATABASE' < $(file)
 
core-drupal: set-env
	@echo "Building Drupal Core Docker from Official Image"
	docker build -t core-drupal .
 
composer-require:
	@echo "Composer require running"
	docker run --rm --interactive --tty --volume $$PWD:/app  --volume $${COMPOSER_HOME:-$$HOME/.composer}:/tmp composer require "$(module)"  --ignore-platform-reqs
 
build-all: set-env
	@echo "Building all non-standard docker images ready for spinning up docker-compose"
	cd docker/nginx/local && docker build -t nginx-drupal .
	docker build -t core-drupal .
 
start-web: set-env
	docker-compose up
 
stop-web:
	@echo "Stopping the web app gracefully. Please wait"
	docker-compose down
 
set-env:
	@echo "Setting the env variables"
	export $(cat .env | grep -v '^#' | grep '\S' | tr '\r' '\0' | xargs -0 -n1)

So for instance should I want to build the codebase and nginx Docker images I would simply type:

make build-all

To add a new module using composer, the syntax would be:

make composer-require module=drupal/module_name

Note the requirement to use key=value for parameter substitution in make. Similarly to import a database dump it would be:

make importdb file=mydump.sql

Not included in the make commands because of its interactive nature, to gain access to MariaDB do:

docker exec -it mariadb-drupal mysql -uusername -ppassword

Finally when you do:

make start-web

you should see output similar to the screenshot above!

Drush

It's important to have Drush available in your local environment. To achieve this, ensure that drush is a dependency requirement in your composer.json file. If not add it using my shortcut mentioned earlier. This will place drush in path /opt/drupal/vendor/bin inside the drupal-core container which obviously must be running for it to work. Exec-ing a command from the VM command line and escaping it correctly is hard work - so to save you the heartache I've created a shell function in my VM to make light work of it. If you are not using my VM, add this to the bottom of your /etc/bash.bashrc file.

drush() {
  PARAMS=$@
  CONTAINER="core-drupal"
  CMD="sh -c \"/opt/drupal/vendor/bin/drush ${PARAMS}\""
  DCMD="docker exec -it $CONTAINER $CMD"
  eval $DCMD
}

Now any Drush command is available to you (when the container is running!)

drush cr
 [success] Cache rebuild complete.

blog terms

AWS AWS CLI devops PHP Drupal Drupal 9