PHP

Hosting Drupal 9 on AWS Fargate Part 2: Bitbucket Pipeline

nigel — Sun, 17 Oct 2021 08:31:46 +0000

Hosting Drupal 9 on AWS Fargate Part 2: Bitbucket Pipeline nigel Sun, 17/10/2021 - 09:31

Introduction

In the first instalment of this sequence of blogs we talked about how imperative it is to create an industrial strength local dev environment when AWS Fargate is your target deployment platform. In this blog we discuss something equally important - the deploy and build process. This needs to be automated for Drupal since security patches are issued on a weekly basis by the Drupal Security Group, and an automated solution is the only way to keep on top of these updates.

Over the last two years I have become accustomed to using Atlassian's Bitbucket Pipelines, and I am really liking this CI/CD delivery mechanism. It is particularly suited to containerised solutions since the pipeline itself runs in a Docker container, and often times the solution Docker image can be used for the build process - case in point this project! This tutorial will show how the official Drupal PHP Docker container can be used as the pipeline for delivering Drupal 9 projects to AWS Fargate. Let's see how!

Bitbucket Pipeline

The diagram above shows the basic steps the pipeline follows. Let's have a look at these in turn with narrative and then see how that transposes into the Pipeline code.A few trivial housekeeping activities have been omitted from the diagram for clarity but will be discussed when we look at the code.

Since my Badzilla blog is a hobby project, I don't have multiple environments, and merge all new features into the git master branch. Any merge into master triggers the build process. Commercial pipelines tend to trigger builds on feature branch commits too which may be a consideration depending upon your circumstances.
I've already mentioned that the Docker image selected for the pipeline will be the official Drupal / PHP release - and thus contains composer which is used to build out the codebase.
Once the codebase has been built the testing can be undertaken. I have identified two parallel steps in the pipeline - automated testing and static testing. I have covered the configuration for these in earlier blogs - so click through for further reading.
The next steps involve building the two Docker image artefacts we need for our Fargate bundles - the PHP / FPM / Codebase image, and the NGINX Web Server image. The latter is required since we need to load a configuration for Drupal 9 and for serving the S3 / CloudFront assets.
The artefacts are pushed into AWS ECR after being tagged latest.
We are now ready to deploy the images to our ECS Cluster - but before we do this, we provide the opportunity to run any pre-deploy hooks. This would typically be activities such as drush putting the web app into maintenance mode to prevent public access during updates.
One the pre-deploy scripts have completed, the deploy script is invoked. This forces an ECS service update on the cluster. The script loops to determine whether the update fails or completes, and upon a successful completion will move onto the final step.
The post-deploy scripts are now run. These would be further drush commands to update the database schema for example, and to remove the maintenance mode flag.

Pipeline Code

The pipeline code is saved in bitbucket-pipelines.yml and is show below.

image:
  name: drupal:9.2.6-php7.4-fpm
 
clone:
  depth: full
 
definitions:
  services:
    docker:
      memory: 2048
 
  steps:
    - step: &build-environment
        name: Build Environment
        script:
          # Install git & unzip for composer
          - apt-get update
          - apt-get install git unzip -y
          # Install dependencies
          - export COMPOSER_ALLOW_SUPERUSER=1
          - composer install
          # Copy project settings file in place
          - cp storage/settings.php web/sites/default/settings.php
          # Remove all non-essential dirs and files from top level directory
          - rm -rf assets backup databases .git
          - rm Makefile README.md docker-compose.yml composer.json composer.lock .env .csslintrc .eslintignore .eslintrc.json
          - rm .gitignore
          # Add the bind mount VOLUME so in Fargate the nginx container can do a VOLUME_FROM and load /opt/drupal docroot
          # from the core-drupal container
          - sed -i '/^FROM.*/a VOLUME ["/opt/drupal"]' Dockerfile > /dev/null 2>&1
        caches:
          - composer
        artifacts:
          - '**'
 
    # Automated Tests placeholder
    - step: &automated-tests
        name: Automated Tests
        script:
          - echo "Add automated tests here"
 
    # Static Analysis tests placeholder
    - step: &static-tests
        name: Static Analysis Tests
        script:
          - echo "Add static analysis here"
 
pipelines:
  # Run when code is merged / pushed into specific branches.
  branches:
    master:
      - step: *build-environment
 
      - parallel:
          - step: *automated-tests
          - step: *static-tests
 
      - step:
          name: Build & Deploy
          deployment: AWS Prod
          script:
            # Install AWS CLI
            - apt update && apt-get install -y jq unzip python3 python-dev python3-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev python-pip
            - curl "https://s3.amazonaws.com/aws-cli/awscli-bundle-1.18.200.zip" -o "awscli-bundle.zip"
            - unzip awscli-bundle.zip
            - ./awscli-bundle/install -b ~/bin/aws
            - export PATH=~/bin:$PATH
            # Build the codebase + FPM
            - docker build -t badzilla-core-drupal .
            # Authenticate with ECR
            - aws configure set aws_access_key_id "${AWS_ACCESS_KEY}"
            - aws configure set aws_secret_access_key "${AWS_SECRET_KEY}"
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${CORE_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-core-drupal:latest "${CORE_DRUPAL_REPO_URL}"
            - docker push "${CORE_DRUPAL_REPO_URL}"
            # NGINX_DRUPAL_REPO_URL
            - cd docker/nginx/fargate
            - docker build -t badzilla-nginx-drupal .
            - cd -
            # Authenticate with ECR
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${NGINX_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-nginx-drupal:latest "${NGINX_DRUPAL_REPO_URL}"
            - docker push "${NGINX_DRUPAL_REPO_URL}"
            # Pre deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/pre-deploy/
            - cd -
            # Deploy script
            - ./deploy-scripts/ecs-service-deploy.sh
            # Post deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/post-deploy/
            - cd -
          services:
            - docker
          caches:
            - docker

Now let's inspect the code per section. Firstly the declaration of the Docker image to use for the pipeline.

image:
  name: drupal:9.2.6-php7.4-fpm

You'll note that I'm using the same image as I used for my development environment with one small change. In the pipeline I have abandoned the use of the ultra lightweight Alpine image - the image being used is standard Ubuntu. Why so? Firstly there is less necessity for keeping the image small since Atlassian don't charge for storage of the image, and secondly I will want to add additional Linux packages to this image for the pipeline process, and thus I want to use a package manager I am familiar with - apt.

clone:
  depth: full
 
definitions:
  services:
    docker:
      memory: 2048

Here I'm saying I want a full clone of the repository in the pipeline. This is a little unnecessary and causes an overhead. In the future I will consider a more shallow clone for greater performance. The Docker memory setting should provide enough headroom for Drupal builds.

  steps:
    - step: &build-environment
        name: Build Environment
        script:
          # Install git & unzip for composer
          - apt-get update
          - apt-get install git unzip -y
          # Install dependencies
          - export COMPOSER_ALLOW_SUPERUSER=1
          - composer install

Steps are defined in the pipeline that are invoked later in the script. This step &build-environment will build out the environment by installing the packages I will need and then running composer install to build out the codebase.

          # Copy project settings file in place
          - cp storage/settings.php web/sites/default/settings.php
          # Remove all non-essential dirs and files from top level directory
          - rm -rf assets backup databases .git
          - rm Makefile README.md docker-compose.yml composer.json composer.lock .env .csslintrc .eslintignore .eslintrc.json
          - rm .gitignore

Every Drupal site needs a settings.php file. I decided to have mine saved in the repository in the directory storage since there is no confidential data. All settings use environmental variables. Your mileage may vary here - other strategies are possible for injecting the settings.php into the build.

I then delete files and directories from the build which won't be needed in a production environment.

          # Add the bind mount VOLUME so in Fargate the nginx container can do a VOLUME_FROM and load /opt/drupal docroot
          # from the core-drupal container
          - sed -i '/^FROM.*/a VOLUME ["/opt/drupal"]' Dockerfile > /dev/null 2>&1

This snippet is a kludge to fix a difference between the local environment docker-compose and the production Fargate Task Definition. Fargate shared volumes use bind mounts which are created by initially using the VOLUME instruction in the Dockerfile and settings in the Task Definition (to be discussed in a later blog). Here I use interactive sed to inject the VOLUME into the Dockerfile

        caches:
          - composer
        artifacts:
          - '**'

Closing the build step I ensure that Docker is cached to save time next time the pipeline is invoked and the artifact (American spelling) setting tells the pipeline to save the all my work in that pipeline step for steps that follow it.

    # Automated Tests placeholder
    - step: &automated-tests
        name: Automated Tests
        script:
          - echo "Add automated tests here"
 
    # Static Analysis tests placeholder
    - step: &static-tests
        name: Static Analysis Tests
        script:
          - echo "Add static analysis here"
 
pipelines:
  # Run when code is merged / pushed into specific branches.
  branches:
    master:
      - step: *build-environment
 
      - parallel:
          - step: *automated-tests
          - step: *static-tests

Here I have defined the testing steps (both of which are currently placeholders), and then started the 'implementation' section of the code. The steps will be invoked on the master branch, initially the build and then the the two testing steps in parallel.

      - step:
          name: Build & Deploy
          deployment: AWS Prod
          script:
            # Install AWS CLI
            - apt update && apt-get install -y jq unzip python3 python-dev python3-dev build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev python-pip
            - curl "https://s3.amazonaws.com/aws-cli/awscli-bundle-1.18.200.zip" -o "awscli-bundle.zip"
            - unzip awscli-bundle.zip
            - ./awscli-bundle/install -b ~/bin/aws
            - export PATH=~/bin:$PATH

Once testing has completed successfully we commence the inline deploy step. We are going to need the AWS CLI installing here since the CLI is necessary to issue the correct sequence of commands to AWS to commit and deploy.

Note I am also installing the amazing utility jq - a JSON query language utility. This is imperative for inspecting the output of the CLI commands since their responses are sent in JSON format.

            # Build the codebase + FPM
            - docker build -t badzilla-core-drupal .
            # Authenticate with ECR
            - aws configure set aws_access_key_id "${AWS_ACCESS_KEY}"
            - aws configure set aws_secret_access_key "${AWS_SECRET_KEY}"
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${CORE_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-core-drupal:latest "${CORE_DRUPAL_REPO_URL}"
            - docker push "${CORE_DRUPAL_REPO_URL}"

Now I've got all I need installed I can build the Docker image artefacts. Firstly I build out the Drupal codebase image which also contains PHP and FPM. I then authenticate with AWS ECR where I will push the artefact once its been tagged.

            # NGINX_DRUPAL_REPO_URL
            - cd docker/nginx/fargate
            - docker build -t badzilla-nginx-drupal .
            - cd -
            # Authenticate with ECR
            - aws ecr get-login --region "${AWS_REGION}" | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS "${NGINX_DRUPAL_REPO_URL}"
            # Tag and push to the repo
            - docker tag badzilla-nginx-drupal:latest "${NGINX_DRUPAL_REPO_URL}"
            - docker push "${NGINX_DRUPAL_REPO_URL}"

I now repeat the process for the NGINX web server Docker image. Note I have the configuration for this in my codebase repo under the directory docker/nginx/fargate

            # Pre deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/pre-deploy/
            - cd -
            # Deploy script
            - ./deploy-scripts/ecs-service-deploy.sh
            # Post deploy Drush Commands
            - cd deploy-scripts
            - ./deploy-parser.sh ../webhooks/post-deploy/
            - cd -

This is the fun part! I now run the deploy shell scripts which are listed with narratives later in the blog. Firstly I run the pre-deploy commands, then the deploy commands and the post-deploy commands. All will be revealed...

          services:
            - docker
          caches:
            - docker

To use Docker commands in a pipeline, you need to list Docker under the services setting. Installing Docker in the pipeline itself and then invoking it will not work! I tried and failed! Note I am also using caching for greater performance.

Repository Environmental Variables

The pipeline and the shell scripts heavily use environmental variables that are set in the repository in Bitbucket. Their names are self-evident and listed in their entirety above.

Pre- and Post-deploy shell script

The pre- and post-deploy shell script is defined below. Currently it only works on drush commands but could be extended to include shell scripts easily enough. The idea is it scans the directory passed as a parameter and it looks for lines in any file that starts with the word drush. It then takes the runtime arguments of that drush command and converts it into a Docker cmd syntax. Once that is done the command is executed as a Fargate task using the codebase image with parameter overrides. This task runs to completion and exits. The status of the command can be captured and reported back to the pipeline. Obviously this isn't as quick as running a drush command in a terminal shell since the Fargate task needs to be provisioned which takes time. It is however a fiendish way of reusing an existing Docker image since the official Drupal Docker image already contains the drush command.
deploy-scripts/deploy-parser.sh

#!/bin/bash
 
# Parse the passed files and issue commands in accordance.
# Currently only supports drush commands.
# This can be expanded out to other shell scripts.
 
# Output directory to be parsed
echo "Directory: $1 to be parsed"
 
set_network_configuration()
{
NETWORK=$(cat <
)
echo "${NETWORK}"
}
 
 
 
set_overrides_command()
{
OVERRIDES=$(cat <
)
echo "${OVERRIDES}"
}
 
 
 
# Process each line in a selected file
parse_line()
{
    # All lines to be processed must start with the word drush to be considered
    # This grep / sed combination will turn the drush command into docker cmd syntax
    grep -e "^drush" < "${1}" | while read -r line ; do
        cmd=`echo $line | sed -r 's/[^ ][^ ]*/"&"/g' | sed -r 's/\s+/,/g' | sed -r 's/drush/vendor\/bin\/drush/'`
 
        # @BUG with quoting in dotenv and how I load into environment. Appears to add newlines making it difficult to add quotes later.
        # So already added quotes for $CORE_TASK_DEFINITION_NAME in .env file but need the unquoted version in
        # --task-definition flag so unquote here
        TASK_DEF_UNQUOTE=`echo ${CORE_TASK_DEFINITION_NAME} | sed s/\"//g`
 
        # Now execute the command to start the container with the drush cmd
        FARGATE_TASK=`aws ecs run-task \
            --cluster "${ECS_CLUSTER_NAME}" \
            --task-definition "${TASK_DEF_UNQUOTE}" \
            --network-configuration "$(set_network_configuration  ${ECS_SUBNET_1} ${ECS_SUBNET_2}  ${ECS_SECURITY_GROUP})" \
            --launch-type FARGATE \
            --platform-version '1.4.0' \
            --region "${AWS_DEFAULT_REGION}" \
            --overrides "$(set_overrides_command ${CORE_TASK_DEFINITION_NAME} ${cmd})" | jq .tasks[0].taskArn -r`
 
        if [[ $? != 0 ]]; then
            echo "Fargate Run Task Failed."
            exit $?
        elif [[ -z  "${FARGATE_TASK}" ]]; then
            echo "Unspecified Fargate Run Task Failure."
            exit 1
        else
            echo "Cmd run is ${cmd}"
        fi
 
        # Loop round to check on the status of the task. Can't run in parallel so have to wait for each task to finish
        while true; do
            FARGATE_STATUS=`aws ecs describe-tasks --cluster "${ECS_CLUSTER_NAME}" --tasks "${FARGATE_TASK}"  | jq .tasks[0].lastStatus -r`
 
            if [[ -z "${FARGATE_STATUS}" ]]; then
                echo "Unspecified Fargate Describe Task Failure."
                exit 1
            fi
 
            echo "${FARGATE_STATUS}"
 
            if [[ "${FARGATE_STATUS}" == "STOPPED" ]] || [[ "${FARGATE_STATUS}" == "DEPROVISIONING" ]]; then
                  break
            fi
 
            sleep 20
        done
 
	echo "Drush command completed"
    done
}
 
 
# Get a list of all the files in the directory and process
parse_directory()
{
    # loop though all the files in the directory
    for file in "${1}"/*; do
        base="$(basename ${file})"
        echo "${base} being parsed"
        parse_line "${file}"
    done
}
 
parse_directory "${1}"

ECS Deploy Script

The main ECS deploy shell script uses AWS CLI ecs update-service to force a new deployment. This means that when the Fargate bundles are recycling any new codebase and NGINX Docker image will be used from AWS ECR. The process can take a few minutes so I have added a loop facility which uses AWS CLI ecs describe-services to capture the current deployment status and act accordingly. Note I am using the previously mentioned jq utility to interrogate the long JSON response from the CLI and whittle it down to the status field which should contain the word "COMPLETED" or "FAILED" or "IN_PROGRESS".
deploy-scripts/ecs-service-deploy.sh

#!/bin/bash
 
 
# Use the aws cli to force deploy Fargate task running in ECS service
aws ecs update-service --cluster "${ECS_CLUSTER_NAME}" --service "${ECS_SERVICE_NAME}" --force-new-deployment --region "${AWS_DEFAULT_REGION}"
 
# Give it a few seconds to take effect
sleep 5
 
# Get the deployment status by looping over the deployment
# Try for a maximum of 20 minutes then give up. Chances are after 20 minutes of incomplete
# deployment there is a problem in the task definition
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
do
    if [[ "${i}" -eq 21 ]]; then
        echo "Timed out before could determine if deployment succeeded"
        exit 1
    fi
 
    STATUS=`aws ecs describe-services --cluster "${ECS_CLUSTER_NAME}" --services "${ECS_SERVICE_NAME}" --region "${AWS_DEFAULT_REGION}" | jq .services[0].deployments[0].rolloutState -r`
 
    # Was it successful?
    if [[ "${STATUS}" = "COMPLETED"  ]]; then
        echo "Deployment succeeded"
        exit 0
    # Anything we need to be concerned about?
    elif [[ "${STATUS}" = "FAILED"  ]]; then
        echo "Deployment failed"
        exit 2
    # If still deploying echo this
    elif [[ "${STATUS}" = "IN_PROGRESS"  ]]; then
        echo "Deployment in progress"
    else
        echo "Unknown status "${STATUS}" so quitting"
        exit 3
    fi
 
    # Wait a minute before retrying
    sleep 60
done

Ad hoc Drush Commands

The solution provides the capability to run drush during the pre- and post-deploy scripts. But what about running drush commands on an ad hoc basis from a remote laptop? This is often required by developers or devops and would be crucial for the management of a Drupal 9 website. Stay tuned for a future blog on how to achieve this!

blog terms

AWS AWS CLI devops PHP Drupal Drupal 9

Hosting Drupal 9 on AWS Fargate Part 1: Local Development

nigel — Sun, 19 Sep 2021 08:46:14 +0000

Hosting Drupal 9 on AWS Fargate Part 1: Local Development nigel Sun, 19/09/2021 - 09:46

Introduction

Welcome to a series of blogs on hosting Drupal 9 (or legacy Drupal 8) sites on AWS Fargate. Fargate can now be considered a mature product, and provides a simple serverless containerised platform for hosting web apps. It offers considerable savings over traditional Virtual servers without, if configured correctly, loss of performance. Furthermore it offers greater simplicity of operation and setup then its AWS sibling offering ECS over EC2, and again a cost saving.

The blogs in this series will cover important considerations such as the pipelines to deploy your codebase within Docker images, and their storage; set up of the Drupal S3FS module along with the CloudFront accelerator and S3 object store; auto scaling of the Fargate tasks; load testing your app to ensure you are fully optimised; and how to run your Drush commands in short living Fargate tasks.

This first blog will cover your team's local environment and how its build will ensure your path to Fargate deploy is smooth, with as much reuse of the configuration in production as possible.

Objective

The objective is to stand up a Fargate task running as a service which will be the web app, and it will persist and auto scale. Also have Fargate tasks to run which will run Drush commands, stood up on a per Drush command basis, and terminating after each Drush command completes. The diagram above shows the Docker images requires to achieve our goal.

Drupal Core + FPM PHP + Codebase

The intention is to use the official Drupal FPM Docker image using Alpine, and be as up to date as possible with the versioning. Note that the Alpine build has been chosen since it's an extremely lightweight version of Linux, based on busybox which has a very small footprint. This comes at a price - those with Ubuntu experience will have to learn new commands. For instance Alpine uses the apk package manager instead of apt-get. The web app's codebase - ie all the configuration and custom modules and themes created by the dev team - will be copied into this image.

Drupal Core + FPM PHP + Codebase + Exec command

This will be the Fargate task to run one off Drush commands. Since the codebase will already contain Drush binaries in vendor/bin/drush, it will be the same as the previously mentioned image. Notwithstanding this, it won't persist. It will use the Fargate equivalent of the Docker command to exec a particular command and then quit.

Memcached

I have elected to run individual Memcached caching in each task running in the Web App service. Memcached will be used for caching the Drupal MySQL cache tables and for page caching for anonymous users. There is an alternative strategy to this - centralise the caching using for instance the AWS ElastiCache with Memcached product. Each task in the web service would have an endpoint to the same centralised ElastiCache instance. This would be the preferred option if your website is largely serving logged in traffic, and if you are running an eCommerce site with the users' baskets stored in Memcached. Since that doesn't apply to my own site, and I would like to keep costs down too, my feeling is having Memcached as part of the Fargate task is a better fit.

nginx

A web server is needed, and I've elected for nginx although Apache would also be a good choice. My rationale for selecting nginx is I would like to add the feature of nginx serving slightly stale cached pages when the origin is returning 5xx errors. This would give a seamless switch-over when a task becomes unhealthy - during the period AWS determines that a task is returning errors and has crossed the unhealthy threshold, nginx could still serve pages which would be less than 5 minutes old. The task would then be drained and removed from the pool with no problem noticeable to the end user. This will be the subject of a blog at a later date.

Using a local Virtual Machine

Those who know me well will know I am not an advocate of using my laptop's OS for development purposes. The reason being that with multiple clients and with multiple projects in each, my host OS would quickly become a mess of different development tooling and versions which could collide and could also generate inconsistencies with other devs in the team should say different versions of bin files end up in the $PATH.

My ethos has always been do not pollute your host OS! Always use VMs for development!

I have created my own VM for Docker based implementations. My belief is if the target deploy is using Docker containers then the local development environment should match this as much as is practicable. In fact it's impossible to match the ECS/Fargate environment exactly, but it is certainly possible to use Docker with docker-compose to simulate closely what will be seen in prod.

My VM can be found in GitHub by clicking on this link. It is a very lightweight VM and can be orchestrated easily using Vagrant and Ansible. PHP and MySQL are deliberately not included in the playbook since all development work will be undertaken by using Docker containers.

All my localhost config examples in this blog will have come from using my DockerVM mentioned above. If you intend to follow this blog closely, you will need to install it, but read the instructions in the README.md carefully!

Composer

Once you've got a lightweight VM up and running (hopefully mine), and ssh'd into it, and cloned your repo, you will need to build out the dependencies using composer install. So how do we go about this when there is no composer and no PHP in the VM? There are two choices:

Use the official Docker Composer image
Use the latest official Docker Drupal image (9.2.6-php7.4-fpm-alpine.3.14 at time of writing this blog)

I chose the official Docker Composer image purely because (a) there was more documentation on it readily available, and (b) I wasn't sure at the time whether the Drupal image bundled Composer. It does, and it will be used in the Bitbucket Pipeline later so you may want to select that instead for consistency, but if you do, you'll have to change my composer commands slightly.

Ok so let's build our composer image.

$ docker run --rm --interactive --tty --volume $PWD:/app composer install --ignore-platform-reqs
Unable to find image 'composer:latest' locally
latest: Pulling from library/composer
a0d0a0d46f8b: Pull complete 
153eea49496a: Pull complete 
11efd0df1fcb: Pull complete 
b3f3214c344d: Pull complete 
9abd2f85688c: Pull complete 
83d85b95eb4c: Pull complete 
923d73ddadfa: Pull complete 
711b5c4b02a7: Pull complete 
ee08fa481788: Pull complete 
5f9f812bf61d: Pull complete 
28b3ecec72d5: Pull complete 
a5ffb14bc4f9: Pull complete 
d92aff3e5aec: Pull complete 
cd0274c84e75: Pull complete 
Digest: sha256:33d2c1d9fb7311cb8f215a8f52669894ef534292ac5d331347f929c27e317bb1
Status: Downloaded newer image for composer:latest
> DrupalProject\composer\ScriptHandler::checkComposerVersion
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
  - Downloading composer/installers (v2.0.1)
<---- SNIPPED ---->
  - Copy [web-root]/profiles/README.txt from assets/scaffold/files/profiles.README.txt
  - Copy [web-root]/themes/README.txt from assets/scaffold/files/themes.README.txt
> DrupalProject\composer\ScriptHandler::createRequiredFiles
Created a sites/default/files directory with chmod 0777

Success! Note that there is a requirement to use the composer flag --ignore-platform-reqs since it expects ext-gd locally which obviously we don't have.

docker-compose: Codebase + FPM

In preparation for creating a docker-compose file we'll need to build out an image containing the latest Drupal 9 Docker image + our codebase once composer install has been run. This will need a Dockerfile, and we can add APCu caching in the Dockerfile for greater Drupal performance. I also noted that Memcached API library needs to be added to Dockerfile since it is my intention to connect to a Memcached server for cache management in the container bundle. The Dockerfile is listed below.

FROM drupal:9.2.6-php7.4-fpm-alpine3.14
 
# Copy the artefact (on Bitbucket) or local files into the image
COPY ./ /opt/drupal/.
 
# PHP Memcached which needs the libmemcached API
# Bug in Apline - creates extensions in the wrong directory so add them in correct location
RUN apk add php7-igbinary php7-pecl-memcached libmemcached \
    && echo "extension=memcached.so" > /usr/local/etc/php/conf.d/docker-php-ext-memcached.ini \
    && echo "extension=igbinary.so" > /usr/local/etc/php/conf.d/docker-php-ext-igbinary.ini \
    && mv /usr/lib/php7/modules/memcached.so /usr/local/lib/php/extensions/no-debug-non-zts-20190902 \
    && mv /usr/lib/php7/modules/igbinary.so /usr/local/lib/php/extensions/no-debug-non-zts-20190902
 
# APCu cache
RUN apk add --update --no-cache --virtual .build-dependencies $PHPIZE_DEPS \
    && pecl install apcu \
    && docker-php-ext-enable apcu \
    && pecl clear-cache \
    && apk del .build-dependencies

docker-compose: nginx image

We will also need to build out an nginx image with the standard Drupal configuration. To do this I created a docker directory at the top level of the repo (i.e. a sibling of web directory) as such:

$ tree docker
docker
├── core-drupal
├── memcached
└── nginx
    ├── fargate
    │   ├── default.conf
    │   ├── Dockerfile
    │   └── nginx.conf
    └── local
        ├── default.conf
        ├── Dockerfile
        └── nginx.conf
 
5 directories, 6 files

You'll note that I have two sets of config files - one for AWS Fargate and one for local development. In fact the differences are minimal and will be discussed later. The local files are below
nginx.conf

user  nginx;
worker_processes  auto;
 
error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
 
 
events {
    worker_connections  1024;
}
 
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    fastcgi_buffers 16 16k;
    fastcgi_buffer_size 32k;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    #tcp_nopush     on;
 
    keepalive_timeout  65;
 
    #gzip  on;
 
    include /etc/nginx/conf.d/*.conf;
}

default.conf

  server {
    server_name badzilla-fargate.test;
    root /opt/drupal/web; ## <-- Your only path reference.
 
    location = /favicon.ico {
      log_not_found off;
      access_log off;
    }
 
    location = /robots.txt {
      allow all;
      log_not_found off;
      access_log off;
    }
 
    location ~ \..*/.*\.php$ {
      return 403;
    }
 
    location ~ ^/sites/.*/private/ {
      return 403;
    }
 
    # Block access to scripts in site files directory
    location ~ ^/sites/[^/]+/files/.*\.php$ {
      deny all;
    }
 
    # Allow "Well-Known URIs" as per RFC 5785
    location ~* ^/.well-known/ {
      allow all;
    }
 
    # Block access to "hidden" files and directories whose names begin with a
    # period. This includes directories used by version control systems such
    # as Subversion or Git to store control files.
    location ~ (^|/)\. {
      return 403;
    }
 
    location / {
      # try_files $uri @rewrite; # For Drupal <= 6
      try_files $uri /index.php?$query_string; # For Drupal >= 7
    }
 
    location @rewrite {
      #rewrite ^/(.*)$ /index.php?q=$1; # For Drupal <= 6
      rewrite ^ /index.php; # For Drupal >= 7
    }
 
    # Don't allow direct access to PHP files in the vendor directory.
    location ~ /vendor/.*\.php$ {
      deny all;
      return 404;
    }
 
    # Protect files and directories from prying eyes.
    location ~* \.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|/(\.(?!well-known).*)|Entries.*|Root|/#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$ {
      deny all;
      return 404;
    }
 
    # In Drupal 8, we must also match new paths where the '.php' appears in
    # the middle, such as update.php/selection. The rule we use is strict,
    # and only allows this pattern with the update.php front controller.
    # This allows legacy path aliases in the form of
    # blog/index.php/legacy-path to continue to route to Drupal nodes. If
    # you do not have any paths like that, then you might prefer to use a
    # laxer rule, such as:
    #   location ~ \.php(/|$) {
    # The laxer rule will continue to work if Drupal uses this new URL
    # pattern with front controllers other than update.php in a future
    # release.
    location ~ '\.php$|^/update.php' {
      fastcgi_split_path_info ^(.+?\.php)(|/.*)$;
      # Ensure the php file exists. Mitigates CVE-2019-11043
      try_files $fastcgi_script_name =404;
      # Security note: If you're running a version of PHP older than the
      # latest 5.3, you should have "cgi.fix_pathinfo = 0;" in php.ini.
      # See http://serverfault.com/q/627903/94922 for details.
      include fastcgi_params;
      # Block httpoxy attacks. See https://httpoxy.org/.
      fastcgi_param HTTP_PROXY "";
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
      fastcgi_param QUERY_STRING $query_string;
      fastcgi_intercept_errors on;
      # PHP 5 socket location.
      #fastcgi_pass unix:/var/run/php5-fpm.sock;
      # PHP 7 socket location.
#      fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
      fastcgi_pass fpm:9000;
    }
 
    # S3FS Module
    location ~ ^/s3/files/styles/ {
      try_files $uri @rewrite;
    }
 
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg)$ {
      try_files $uri @rewrite;
      expires max;
      log_not_found off;
    }
 
    # Fighting with Styles? This little gem is amazing.
    # location ~ ^/sites/.*/files/imagecache/ { # For Drupal <= 6
    #location ~ ^/sites/.*/files/styles/ { # For Drupal >= 7
    #  try_files $uri @rewrite;
    #}
 
    # Handle private files through Drupal. Private file's path can come
    # with a language prefix.
    location ~ ^(/[a-z\-]+)?/system/files/ { # For Drupal >= 7
      try_files $uri /index.php?$query_string;
    }
 
    # Enforce clean URLs
    # Removes index.php from urls like www.example.com/index.php/my-page --> www.example.com/my-page
    # Could be done with 301 for permanent or other redirect codes.
    if ($request_uri ~* "^(.*/)index\.php/(.*)") {
      return 307 $1$2;
    }
  }

Dockerfile

FROM nginx:1.21.1-alpine
COPY ./default.conf /etc/nginx/conf.d/
COPY ./nginx.conf /etc/nginx/

Some commentary is required here. You'll see in the default.conf file there is a snippet:

    # S3FS Module
    location ~ ^/s3/files/styles/ {
      try_files $uri @rewrite;
    }

This is because the assets are not served locally - they are stored in an AWS S3 bucket with the AWS CloudFront accelerator providing edge capabilities. I will provide a further blog at a later date on setting all this up but in essence, assets shouldn't be kept in individual Fargate tasks, so either an AWS S3 / CloudFront solution needs to be utilised or an AWS EFS drive.

Now let's see what the differences are between the local configuration and Fargate.

$ diff docker/nginx/local/default.conf docker/nginx/fargate/default.conf 
2c2
<     server_name badzilla-fargate.test;
---
>     server_name badzilla.co.uk www.badzilla.co.uk;
4a5,10
>     # User Agent ELB-HealthChecker sends no headers with the request so nginx by default will respond with a 400
>     # Fix with this
>     location = /health {
>         return 200;
>     }
> 
92c98
<       fastcgi_pass fpm:9000;
---
>       fastcgi_pass 127.0.0.1:9000;

Ok there are three differences. The first is obvious - I am using different server names dependent on the environment. The second difference is the AWS health checking capability that I have included for Fargate but not needed locally. I use the Drupal health_check module by the way, but the problem resides with the AWS Application Load Balancer health check feature. It sends requests to nginx without headers which nginx doesn't like - and always replies with a 400 response. This snippet fixes that problem. The third difference relates to how Fargate handles Docker networking. It is much simplified and all containers in the Fargate task use 127.0.0.1 for communication. Locally I am using a label named fpm for fastcgi_pass communication between nginx and FPM.

docker-compose: MariaDB and Memcached

The use of MariaDB is the most significant difference between the local environment and Fargate. Fargate connects to a MariaDB instance in AWS RDS, whilst the local MariaDB persists its data in the VM. I did consider using another RDS instance for local development, but that would require public access to the database which I wasn't keen to do for security reasons. In addition there is a cost implication which isn't viable for a hobby project like my blog. I am using the standard Docker images for both MariaDB and Memcached locally.

docker-compose Configuration

docker-compose

version: "3"
 
services:
  mariadb:
    image: "mariadb:${MARIADB_VERSION}"
    restart: 'always'
    container_name: mariadb-drupal
    volumes:
      - "/var/lib/mysql/data:${MARIADB_DATA_DIR}"
      - "/var/lib/mysql/logs:${MARIADB_LOG_DIR}"
      - "/var/docker/mariadb/conf:/etc/mysql"
    environment:
      MYSQL_ROOT_PASSWORD: "${MYSQL_ROOT_PASSWORD}"
      MYSQL_DATABASE: "${MYSQL_DATABASE}"
      MYSQL_USER: "${MYSQL_USER}"
      MYSQL_PASSWORD: "${MYSQL_PASSWORD}"
  drupal:
    image: core-drupal:latest
    container_name: core-drupal
    depends_on:
      - mariadb
      - memcached
    volumes:
      - ./:/opt/drupal
    restart: 'always'
    environment:
      MYSQL_DATABASE: "${MYSQL_DATABASE}"
      MYSQL_USER: "${MYSQL_USER}"
      MYSQL_PASSWORD: "${MYSQL_PASSWORD}"
      MYSQL_HOSTNAME: "${MYSQL_HOSTNAME}"
      MYSQL_NAMESPACE: "${MYSQL_NAMESPACE}"
      MYSQL_DRIVER: "${MYSQL_DRIVER}"
      MYSQL_PORT: "${MYSQL_PORT}"
      AH_SITE_ENVIRONMENT: "${AH_SITE_ENVIRONMENT}"
      S3FS_ACCESS_KEY: "${S3FS_ACCESS_KEY}"
      S3FS_SECRET_KEY: "${S3FS_SECRET_KEY}"
      S3FS_BUCKET: "${S3FS_BUCKET}"
      S3FS_AWS_REGION: "${S3FS_AWS_REGION}"
      S3FS_CLOUDFRONT_CNAME: "${S3FS_CLOUDFRONT_CNAME}"
      S3FS_USE_S3_PUBLIC: "${S3FS_USE_S3_PUBLIC}"
      S3FS_USE_CNAME: "${S3FS_USE_CNAME}"
      S3FS_USE_HTTPS: "${S3FS_USE_HTTPS}"
      S3FS_TWIG_PATH: "${S3FS_TWIG_PATH}"
 
  nginx:
    image: nginx-drupal:latest
    container_name: nginx-drupal
    links:
      - drupal:fpm
    ports:
      - "80:80"
      - "443:443"
    depends_on:
      - drupal
    restart: 'always'
    volumes:
      - ./:/opt/drupal
  memcached:
    image: memcached:alpine3.14
    restart: 'always'
    container_name: memcached-drupal
    ports:
      - "11211:11211"

Note I have all my settings held in the environment. I also use the vlucas/phpdotenv composer dependency so these environmental variables are accessible in Drupal through a dot env file.

Shortcuts

Remembering commands to build the Docker images, gain command line access to MariaDB, importing and exporting databases etc is an exercise in tedium I'd rather avoid. I have built a make file with shortcuts although command line aliases could also have been used.
Makefile

.DEFAULT_GOAL := help
 
 
help:
	@echo "See the README.md for available options"
 
exportdb: set-env
	@echo "Exporting the database"
	docker exec -i mariadb-drupal bash -c 'mysqldump -u $$MYSQL_USER -p$$MYSQL_PASSWORD $$MYSQL_DATABASE' > $(file)
 
importdb: set-env
	@echo "Importing the database"
	docker exec -i mariadb-drupal bash -c 'mysql -u $$MYSQL_USER -p$$MYSQL_PASSWORD $$MYSQL_DATABASE' < $(file)
 
core-drupal: set-env
	@echo "Building Drupal Core Docker from Official Image"
	docker build -t core-drupal .
 
composer-require:
	@echo "Composer require running"
	docker run --rm --interactive --tty --volume $$PWD:/app  --volume $${COMPOSER_HOME:-$$HOME/.composer}:/tmp composer require "$(module)"  --ignore-platform-reqs
 
build-all: set-env
	@echo "Building all non-standard docker images ready for spinning up docker-compose"
	cd docker/nginx/local && docker build -t nginx-drupal .
	docker build -t core-drupal .
 
start-web: set-env
	docker-compose up
 
stop-web:
	@echo "Stopping the web app gracefully. Please wait"
	docker-compose down
 
set-env:
	@echo "Setting the env variables"
	export $(cat .env | grep -v '^#' | grep '\S' | tr '\r' '\0' | xargs -0 -n1)

So for instance should I want to build the codebase and nginx Docker images I would simply type:

make build-all

To add a new module using composer, the syntax would be:

make composer-require module=drupal/module_name

Note the requirement to use key=value for parameter substitution in make. Similarly to import a database dump it would be:

make importdb file=mydump.sql

Not included in the make commands because of its interactive nature, to gain access to MariaDB do:

docker exec -it mariadb-drupal mysql -uusername -ppassword

Finally when you do:

make start-web

you should see output similar to the screenshot above!

Drush

It's important to have Drush available in your local environment. To achieve this, ensure that drush is a dependency requirement in your composer.json file. If not add it using my shortcut mentioned earlier. This will place drush in path /opt/drupal/vendor/bin inside the drupal-core container which obviously must be running for it to work. Exec-ing a command from the VM command line and escaping it correctly is hard work - so to save you the heartache I've created a shell function in my VM to make light work of it. If you are not using my VM, add this to the bottom of your /etc/bash.bashrc file.

drush() {
  PARAMS=$@
  CONTAINER="core-drupal"
  CMD="sh -c \"/opt/drupal/vendor/bin/drush ${PARAMS}\""
  DCMD="docker exec -it $CONTAINER $CMD"
  eval $DCMD
}

Now any Drush command is available to you (when the container is running!)

drush cr
 [success] Cache rebuild complete.

blog terms

AWS AWS CLI devops PHP Drupal Drupal 9

AWS Fargate: CI/CD Architecture for Drupal and PHP Frameworks

nigel — Sat, 04 Jan 2020 13:45:57 +0000

AWS Fargate: CI/CD Architecture for Drupal and PHP Frameworks nigel Sat, 04/01/2020 - 13:45

Introduction

AWS Fargate is a compute engine for Amazon ECS which allows the running of Docker containers without having to provision and manage the underlying EC2 servers. This makes the standing up of Docker bundles immeasurably easier, and thankfully that ease comes with very few limitations. It is often considered the Docker equivalent of AWS Lambda.

Fargate's ease of use and convenience makes it a natural fit for CI/CD architecture. Apps can be defined as Fargate tasks which are analogous to Docker Bundles, and pipeline stages can be engineered to perform standard activities such as build, automated testing, static analysis, dev playground, and deploy.

This architecture statement explains how this can be applied to the world of PHP frameworks and Drupal CMS. With a little imagination, and a few tweaks to the ecosystem and Fargate tasks, and the architecture would be valid for other high level languages and frameworks.

Pipeline Stages

Before we get into the physical AWS ecosystem, it's worth discussing what we are trying to achieve. I have identified the four main stages in the pipeline in the diagram above. There will be two entry points into the build stage: it will be either automated on a git pull request creation and approval, but also a manual invocation is possible for times when a dev or QA wants to check progress on a feature after a commit but before a PR is created. In PHP frameworks and Drupal 8+ codebases, it is customary to use the dependency management tool composer which obviates the need to check into vcs contributed plugins and modules. The build stage will also compile the front end themes and JavaScript and minify / uglify at the same - therefore production ready CSS and JS should not be committed into vcs by developers. Once the codebase has been built, it will be copied into a PHP/Apache Docker image and pushed into the AWS ECR repository for later use.

The automated testing will cover both functional BDD and unit TDD testing against the codebase image that has just been built. Writing TDD against the largely procedural Drupal 7 was onerous and challenging and therefore overlooked in the main. Thankfully Drupal 8 is OOP which makes writing back-end tests with PHPUnit much more straightforward.

The static analysis will check for code quality, vulnerability, smell, copy and paste detection, and basic lint type checking.

The deploy step can mean different things to different people. Deployment can mean pushing your built codebase to the production server, or it can mean simply creating a semantic versioning tag, and pushing the codebase artefact to an upstream repository, ready for a manual deployment to prod by Ops staff. I have opted for the latter interpretation in this instance.

CI/CD Ecosystem

The physical ecosystem is described below and shown in diagrammatic form above. Whilst on first glance this may appear quite complex, in fact it is largely self-evident with a little explanation.

Jenkins

Jenkins will be the automation tool since it is ubiquitous across the industry and there is a great deal of expertise in the market place. Jenkins has pipeline capability, and new projects should elect to use the Groovy-based declarative scripting DSL. This allows stages to run in parallel, and thus once the build stage is completed, automated testing and static analysis can be concurrent. In addition I propose the standing up of a 'persisting playground' - the built feature branch of the web app with a DNS qualified relevant url which can be shared between devs, QA, and stakeholders. This would be useful for troubleshooting, demonstrations of progress, show and tells etc., and would be torn down at close of business to reduce costs. My tutorial here shows how a friendly branch specific url can be created and propagated by AWS Route 53 within 60 seconds.

The Jenkins pipeline script is responsible for control flow only; all the actual 'doing' activities, such as building the codebase Docker image on the fly and committing it to ECR, is undertaken by shell scripts spawned from the pipeline. This means the power of AWS CLI can be leveraged to undertake all AWS specific tasks.

The flow of the pipeline will vary depending upon the digital team's needs. I have elected for the following:

On PR Creation, run build, automated testing, static analysis, persisting playground
On PR Approval, run build, automation testing (and fail the pipeline if testing fails), deploy

Obviously you may have different requirements here such as building on each feature branch commit, but the re-configuration effort will be minimal regardless. Adding conditions on the Jenkins pipeline stages is trivial.

AWS ECR

The PHP-Apache Docker image is a starting point for creating an image that will include the feature branch built by Jenkins. Detailed instructions on how to achieve this are covered in my earlier blog, and once the image is created it is committed to ECR. When a Fargate task is run, it must first be provisioned by fetching all the Docker images it needs, and having the codebase + Apache/PHP local to AWS will improve start-up.

AWS Fargate

Jenkins will run up to three concurrent AWS Fargate tasks - a task which includes all the Docker containers to run the automated tests, a task that has the static analysis containers, and the basic web app containers for the persisting playground which will contain Apache+PHP along with probably Memcached or Redis, and any other products and services required by your web app. More on the AWS Fargate task definitions later.

The automated tests and the static analysis run to conclusion and automatically die once their entry point scripts have run. The persisting playground will run Apache in the foreground to prevent termination.

AWS RDS

It is impractical to import db dumps into Fargate tasks, so all Fargate tasks will use one shared database resource. Obviously sharing a database can feasibly lead to schema inconsistencies when different schema changes are being developed on different feature branches. In Drupal these are implemented by drush updatedb and in PHP frameworks there is always specific command line tooling to perform database migrations. To mitigate against schema inconsistencies, there would need to be communication between developers, and the facility to reset the database to a known develop branch datum with a Jenkins job.

AWS S3

An export of the develop branch database should be held locally in AWS S3 to facilitate quick import into RDS as required, and updated nightly.

AWS S3 & AWS Cloudfront

File assets such as jpg and png images are normally held in /sites/{default|domain name}/files in Drupal but are not committed to vcs. These need to be synced from the host's server nightly into S3, and for greater performance the use of Cloudfront acceleration is recommended. Configuration changes will be required in the Drupal or PHP framework codebase to point to the Cloudfront endpoints. Drupal will require the use of the Storage API module or the S3 File System module.

AWS CloudWatch

Fargate integrates with CloudWatch easily, and output from each Docker container in the Fargate task definition will send stdout and stderr to a CloudWatch log group. This is imperative not just for troubleshooting, but also to determine the automated test results. By parsing the logs, a full test output can be created.

Slack Integration

It is desirable for the CI/CD pipeline to be hands-free, i.e. Jenkins login not required unless a manual pipeline invocation is necessary. To achieve this goal, Slack integration is required with job statuses being sent to dedicated Slack channels, and output from the test results also sent through Slack.

Existing Host

Of course to seed RDS and the file asset S3, we need to get database exports and assets from our existing host. Exactly how to achieve this will vary dependent upon the host. If possible, a slave Jenkins can execute remote jobs on the host to upload the db exports and file assets to RDS and S3 respectively, and that is how I have drawn my diagram. If this is not possible, a combination of scp and rsync will suffice.

Fargate Task Definitions

Fargate tasks can be defined in the AWS Console, but I've already identified that AWS CLI is the tool of choice for programmatic creation. The CLI requires the container definitions in JSON format, and for the pipeline discussed above there would be three Fargate definitions per feature branch. The first is the standard web app which would typically have the PHP/Apache container with the codebase copied into it during a docker build command. The second is the automated testing stack which includes Maven (which has the Java runtime bundled) along with Selenium grid and two nodes: Firefox and Chrome. I spoke about this Fargate task at length in an earlier blog here. The third definition is for static analysis and I am suggesting the excellent jakzal/phpqa repository for this.

The three tasks would be defined and then run within the Jenkins pipeline. It must be noted that once the tasks have been run they should be immediately deleted from AWS ECR. Storage in ECR is costed, and in the unlikely event of the tasks being required at a later date, they can always be recreated by invoking the pipeline again manually and selecting the feature branch from a drop down list.

The use of Drupal Drush

Fargate is a black box, without access to the underlying EC2 infrastructure, and therefore it is not possible to ssh into EC2 and then perform a docker exec command. Furthermore drush aliases require an ssh daemon running on the target alias to be able to execute remote commands from say a developer working on premises. So how can we run the normal set of everyday drush commands such as to apply schema changes, apply features, split the configuration, flush the caches etc?

Ok so we have PHP installed on Jenkins since we have to run composer install in the build stage. We also have an RDS instance with the database. We therefore have what we need to bootstrap Drupal - which is a requirement of drush.

Therefore all drush commands should be run on Jenkins during the build stage. These commands should be listed in a sibling directory to the codebase's docroot or web directory, in say devops_hooks or some such. As part of the Jenkins build stage, the existence of directory devops_hooks will be checked, and all shell scripts within it will be run.

blog terms

AWS AWS CLI devops PHP Drupal

Drupal 8: Move Deprecated drupal/drupal Composer Template to drupal-composer/drupal-project

nigel — Wed, 11 Dec 2019 15:52:32 +0000

Drupal 8: Move Deprecated drupal/drupal Composer Template to drupal-composer/drupal-project nigel Wed, 11/12/2019 - 15:52

Tutorial for moving an existing Drupal 8 codebase away from the deprecated drupal/drupal composer template using the utility GoComposer. This will move the project to the latest Fully Composer Managed template. This will also align the codebase with Drupal 9 reducing any effort required to ensure full Drupal 9 compatibility on day 1 of Drupal 9 release.

This solution will apply to many Drupal 8 codebases, including my own Badzilla site. When I migrated Badzilla from D6, the drupal/drupal composer template appeared to be the correct choice. Sadly experience has proven otherwise, and all Drupal 8 minor versions have caused serious headaches when updating.

So I will be writing this blog from the perspective of moving composer templates on Badzilla.

Getting Started

I will be doing this in a development environment. I am using my own BadzillaVM development environment. Back up both the database and the codebase artefact on prod anyway in preparation for the deploy later.

My first activity is to change directory in my dev environment and create a new git branch dedicated to this project.

$ git checkout -b feature/composer-template

Now let's see what our starting point by listing my existing composer.json file.

$ cd docroot
$ cat composer.json
{
    "name": "drupal/drupal",
    "description": "Drupal is an open source content management platform powering millions of websites and applications.",
    "type": "project",
    "license": "GPL-2.0+",
    "require": {
        "composer/installers": "^1.0.24",
        "cweagans/composer-patches": "~1.0",
        "wikimedia/composer-merge-plugin": "~1.4",
        "drupal/core": "8.6.2",
        "drupal/bootstrap": "^3.9",
        "drupal/memcache": "^2.0@alpha",
        "drupal/module_filter": "^3.0",
        "drupal/pathauto": "^1.0@RC",
        "drupal/devel": "^1.0@RC",
        "drupal/admin_toolbar": "^1.19",
        "drupal/config_inspector": "^1.0@beta",
        "drupal/google_analytics": "^2.1",
        "drupal/metatag": "^1.2",
        "drupal/addtoany": "^1.8",
        "drupal/migrate_tools": "^4.0@beta",
        "drupal/migrate_plus": "^4.0@beta",
        "drupal/migrate_upgrade": "^3.0",
        "drupal/migrate_manifest": "^1.5",
        "drupal/config_update": "^1.3",
        "drupal/paragraphs": "^1.1",
        "drupal/geshifilter": "^1.1",
        "drupal/xmlsitemap": "^1.0@alpha",
        "drupal/youtube": "^1.0@beta",
        "drupal/fontawesome": "^2.0",
        "drupal/config_filter": "^1.0",
        "drupal/config_split": "^1.2",
        "drupal/schema_metatag": "^1.0-rc4",
        "phpdocumentor/reflection-docblock": "^2.0",
        "drush/drush": "9.*",
        "drupal/libraries": "^3.0@alpha",
        "phing/phing": "^2.16",
        "drupal/tome": "^1.0@alpha",
        "drupal-tome/tome_drush": "dev-master",
        "drupal/search_api": "^1.10",
        "drupal/elasticsearch_connector": "^6.0-alpha1"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "platform": {
            "php": "7.0.32"
        },
        "preferred-install": "dist",
        "autoloader-suffix": "Drupal8"
    },
    "extra": {
        "_readme": [
            "By default Drupal loads the autoloader from ./vendor/autoload.php.",
            "To change the autoloader you can edit ./autoload.php.",
            "This file specifies the packages.drupal.org repository.",
            "You can read more about this composer repository at:",
            "https://www.drupal.org/node/2718229"
        ],
        "merge-plugin": {
            "include": [
                "core/composer.json"
            ],
            "recurse": false,
            "replace": false,
            "merge-extra": false
        },
        "installer-paths": {
            "core": ["type:drupal-core"],
            "modules/contrib/{$name}": ["type:drupal-module"],
            "profiles/contrib/{$name}": ["type:drupal-profile"],
            "themes/contrib/{$name}": ["type:drupal-theme"],
            "drush/contrib/{$name}": ["type:drupal-drush"],
            "modules/custom/{$name}": ["type:drupal-custom-module"],
            "themes/custom/{$name}": ["type:drupal-custom-theme"],
            "libraries/{$name}": ["type:drupal-library"]
        },
        "patches": {
            "drupal/core": {
                "rdf: Fatal error: Call to a member function url() on null": "https://www.drupal.org/files/issues/member-function-url-fix-2565247-4.patch"
            },
            "modules/contrib/module_filter": {
                "Module Filter issues notices after extend list changes": "https://www.drupal.org/files/issues/module_filter-undefined_index_recent_modules_submit-2857431-16.patch"
            }
        }  
    },
    "autoload": {
        "psr-4": {
            "Drupal\\Core\\Composer\\": "core/lib/Drupal/Core/Composer"
        }
    },
    "scripts": {
        "pre-autoload-dump": "Drupal\\Core\\Composer\\Composer::preAutoloadDump",
        "post-autoload-dump": [
          "Drupal\\Core\\Composer\\Composer::ensureHtaccess"
        ],
        "post-package-install": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",
        "post-package-update": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup"
    },
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ]
}

Ok so I'm a little behind on my Drupal updates with core currently on 8.6.2, and I'm using PHP 7.0 on both sandbox and prod which is ok for now but needs to be updated soon. I've also got a couple of patches which may be resolved already as I go through this exercise.

GoComposer

To run the GoComposer code I need to be in the top level directory of the codebase, i.e. the directory that contains my .git directory. Ensure this is the case then create a skeleton composer file for GoComposer

$ cd ..
$ composer require webkings-ca/gocomposer:dev-master
You are running composer with xdebug enabled. This has a major impact on runtime performance. See getcomposer.org/xdebug
./composer.json has been created
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 6 installs, 0 updates, 0 removals
  - Installing webkings-ca/gocomposer (dev-master 310fa76): Cloning 310fa76ed0 from cache
  - Installing symfony/polyfill-ctype (v1.13.1): Downloading (100%)         
  - Installing webmozart/assert (1.6.0): Downloading (100%)         
  - Installing webmozart/path-util (2.3.0): Loading from cache
  - Installing webflo/drupal-finder (1.2.0): Downloading (100%)         
  - Installing symfony/yaml (v3.4.36): Downloading (100%)         
symfony/yaml suggests installing symfony/console (For validating YAML files using the lint command)
Writing lock file
Generating autoload files
PHP 7.0.33-13+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Nov 28 2019 07:43:06) ( NTS )

Now I am ready to run gocomposer

$ composer gocomposer
You are running composer with xdebug enabled. This has a major impact on runtime performance. See getcomposer.org/xdebug
 
GoComposer is Initializing...
=============================
 
 
 Please Confirm: Composer.json file will be Created in /var/www/html/meedjum-composer/docroot (Recommended) (yes/no) [yes]:
 > 
 
 Please Confirm: Your Original Site will be Backed up to the following directory:  /var/www/html/meedjum-composer/docroot/backup (Recommended) (yes/no) [yes]:
 > 
 
 Please select which Settings.php file you want to use for your Environment (Recommended Version is: [0]) 
  [0] /var/www/html/meedjum-composer/docroot/sites/default/settings.php
 
 > 
 You have just selected: /var/www/html/meedjum-composer/docroot/sites/default/settings.php
 
 
 
  [ErrorException]                                               
  Use of undefined constant DRUPAL_ROOT - assumed 'DRUPAL_ROOT'  
 
gocomposer

Problem 1: DRUPAL_ROOT not defined

An exception was thrown soon after the process began - Use of undefined constant DRUPAL_ROOT - assumed 'DRUPAL_ROOT'which was thrown in my settings.local.php file on the following line:

$settings['container_yamls'][] = DRUPAL_ROOT . '/sites/default/local.services.yml';
?>

Now that's weird since it's saying that DRUPAL_ROOT isn't been set during the Drupal bootstrap process. So I changed it to $app_root which worked.

Problem 2: Removing drupal_coder

I then started the process again and get further, but was presented with

  - Removing drupal/coder (8.3.1)
    The package has modified files:
    D coder_sniffer/Drupal/Test/Arrays/ArrayUnitTest.inc
    D coder_sniffer/Drupal/Test/Arrays/ArrayUnitTest.inc.fixed
    D coder_sniffer/Drupal/Test/Arrays/ArrayUnitTest.php
    D coder_sniffer/Drupal/Test/Arrays/DisallowLongArraySyntaxUnitTest.php
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d7/DisallowLongArraySyntaxUnitTest.1.inc
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d7/disallow_long_array_d7.info
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d8/DisallowLongArraySyntaxUnitTest.2.inc
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d8/DisallowLongArraySyntaxUnitTest.2.inc.fixed
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d8/disallow_long_array_d8.info.yml
    D coder_sniffer/Drupal/Test/Classes/ClassCreateInstanceUnitTest.inc
    240 more files modified, choose "v" to view the full list
    Discard changes [y,n,v,d,?]? ?
    y - discard changes and apply the uninstall
    n - abort the uninstall and let you manually clean things up
    v - view modified files
    d - view local modifications (diff)
    ? - print help
    Discard changes [y,n,v,d,?]?

This is one of those occasions when none of the options are what I wanted, but I elected to uninstall drupal_coder

Problem 3: Config directory in the wrong place

The process continued and once it starts the database work, go for a cup of tea because it takes a lot of time. I got the following diagnostic part way

Current Step: Using Drush to update the Database... Press Enter to Continue...
------------------------------------------------------------------------------
 
 [error]  The directory <em class="placeholder">../config/syncem> does not exist. 
 [error]  The directory <em class="placeholder">../config/split/prodem> does not exist. 
 [error]  The directory <em class="placeholder">../config/split/sandboxem> does not exist.

The config directory is in the wrong place which surprised me so I elected to let the process complete and then I would investigate

GoComposer Completes

  views               make_place   post-update     Rebuild cache to allow       
                      holders_tr                   placeholder texts to be      
                      anslatable                   translatable.                
  views               remove_cor   post-update     Remove core key from views   
                      e_key                        configuration.               
 ------------------- ------------ --------------- ----------------------------- 
 
 Do you wish to run the specified pending updates? (yes/no) [yes]:
 > 
 
 
 [OK] Congrats!... You have Successfully updated your Site... The old site files and Database Sql dump are saved in the 
      backup folder                                                                                                     
 
 
 
 ! [NOTE] Your New Docroot is in the newly created /web directory... Dont forget to update you vhosts file by adding    
 !        /web to the site path...

The process finally completed, which included updated Drupal and all my modules to the latest version - you can just see in the top part of the listing above it is finishing the module updates. Now I investigated the problem with the sync directory and I was astounded to discover that the new web directory is actually a child of docroot! Say what? I assumed that docroot would be replaced by web, but instead it's created a complete subdirectory structure underneath it! That explains why the config directory is wrong in the settings.php file.

Fix Problem 3: Directory structure incorrect

GoComposer left me with a directory structure of

/var/www/html/meedjum-composer/docroot/web

So web needs moving to be a child of the the top level directory (meedjum-composer) whilst the files and directories of docroot need moving into meedjum-composer. First steps I need to clear down the top level directory.

$ pwd
/var/www/html/meedjum-composer
$ rm -r composer.*  #old stuff 
$ rm -rf vendor  #old stuff
$ rm -rf drush   #old stuff

Next copy the web directory in place and delete the obsolete docroot directory

$ cp -vaR docroot/. .
$ rm -rf docroot

Did it Work?

Before the solution can be tested, the web server docroot will need to be changed. This will vary dependent upon your web server (usually nginx or Apache2) and that server's configuration so it isn't covered here. Mine is simple because I am using the BadzillaVM so I changed my docroot setting in my Ansible playbook to point to /var/www/html/meedjum-composer/web and re-provisioned my virtual box.

I then successfully logged into my sandbox version of Badzilla and navigated to admin/reports/updates. I now have a Drupal site running core 8.8.0, and I have successfully moved away from the deprecated drupa/drupal composer template.

However the process didn't update the Admin Toolbar as you can see. So lets have a look at our new composer,json file and perform the update.

Update Admin Toolbar

The new composer.json file looks like:

{
    "name": "drupal-composer/drupal-project",
    "description": "Project template for Drupal 8 projects with composer",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "authors": [
        {
            "name": "",
            "role": ""
        }
    ],
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ],
    "require": {
        "php": ">=7.0.8",
        "composer/installers": "^1.2",
        "cweagans/composer-patches": "^1.6.5",
        "drupal/console": "^1.0.2",
        "drupal/core": "^8.6.2",
        "drupal/core-composer-scaffold": "^8.8.0",
        "drush/drush": "^9.7.1 | ^10.0.0",
        "vlucas/phpdotenv": "^4.0",
        "webflo/drupal-finder": "^1.0.0",
        "zaporylie/composer-drupal-optimizations": "^1.0",
        "drupal/token": "^1.5.0",
        "drupal/devel": "^1.2.0",
        "drupal/tome": "^1.0.0-alpha2",
        "drupal/schema_metatag": "^1.3.0",
        "drupal/addtoany": "^1.10.0",
        "drupal/ctools": "^3.0.0",
        "drupal/fontawesome": "^2.8.0",
        "drupal/config_update": "^1.5.0",
        "drupal/google_analytics": "^2.3.0",
        "drupal/search_api": "^1.10.0",
        "drupal/elasticsearch_connector": "^6.0.0-alpha1",
        "drupal/migrate_upgrade": "^3.0.0-rc5",
        "drupal/geshifilter": "^1.2.0",
        "drupal/libraries": "^3.0.0-alpha1",
        "drupal/migrate_plus": "^4.0.0",
        "drupal/config_split": "^1.4.0",
        "drupal/pathauto": "^1.3.0",
        "drupal/youtube": "^1.0.0-beta3",
        "drupal/migrate_tools": "^4.0.0",
        "drupal/admin_toolbar": "^1.24.0",
        "drupal/metatag": "^1.7.0",
        "drupal/config_inspector": "^1.0.0-beta2",
        "drupal/memcache": "^2.0.0-rc2",
        "drupal/migrate_manifest": "^1.7.0",
        "drupal/module_filter": "^3.1.0",
        "drupal/config_filter": "^1.3.0",
        "drupal/entity_reference_revisions": "^1.6.0",
        "drupal/paragraphs": "^1.3.0",
        "drupal/xmlsitemap": "^1.0.0-alpha3",
        "drupal/bootstrap": "^3.13.0"
    },
    "require-dev": {
        "drupal/core-dev": "^8.8.0"
    },
    "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "sort-packages": true
    },
    "autoload": {
        "classmap": [
            "scripts/composer/ScriptHandler.php"
        ],
        "files": [
            "load.environment.php"
        ]
    },
    "scripts": {
        "pre-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "pre-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "post-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ],
        "post-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ]
    },
    "extra": {
        "composer-exit-on-patch-failure": true,
        "patchLevel": {
            "drupal/core": "-p2"
        },
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            }
        },
        "installer-paths": {
            "web/core": [
                "type:drupal-core"
            ],
            "web/libraries/{$name}": [
                "type:drupal-library"
            ],
            "web/modules/contrib/{$name}": [
                "type:drupal-module"
            ],
            "web/profiles/contrib/{$name}": [
                "type:drupal-profile"
            ],
            "web/themes/contrib/{$name}": [
                "type:drupal-theme"
            ],
            "drush/Commands/contrib/{$name}": [
                "type:drupal-drush"
            ]
        }
    }
}

The "conflict" entry made me smile - I haven't seen that kind of entry before! But the focus is admin_toolbar and it can be seen that the reason it wasn't updated is because it has moved up a major version and the tilde on the specified version prohibited such a large jump. Let's fix this.

$ composer require drupal/admin_toolbar:2.0
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
./composer.json has been updated
> DrupalProject\composer\ScriptHandler::checkComposerVersion
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 0 installs, 1 update, 0 removals
  - Updating drupal/admin_toolbar (1.27.0 => 2.0.0): Downloading (100%)         
Package container-interop/container-interop is abandoned, you should avoid using it. Use psr/container instead.
Package phpunit/phpunit-mock-objects is abandoned, you should avoid using it. No replacement was suggested.
Writing lock file
Generating autoload files
Scaffolding files for drupal/core:
  - Copy [project-root]/.editorconfig from assets/scaffold/files/editorconfig
  - Copy [project-root]/.gitattributes from assets/scaffold/files/gitattributes
  - Copy [web-root]/.csslintrc from assets/scaffold/files/csslintrc
  - Copy [web-root]/.eslintignore from assets/scaffold/files/eslintignore
  - Copy [web-root]/.eslintrc.json from assets/scaffold/files/eslintrc.json
  - Copy [web-root]/.ht.router.php from assets/scaffold/files/ht.router.php
  - Copy [web-root]/.htaccess from assets/scaffold/files/htaccess
  - Copy [web-root]/example.gitignore from assets/scaffold/files/example.gitignore
  - Copy [web-root]/index.php from assets/scaffold/files/index.php
  - Copy [web-root]/INSTALL.txt from assets/scaffold/files/drupal.INSTALL.txt
  - Copy [web-root]/README.txt from assets/scaffold/files/drupal.README.txt
  - Copy [web-root]/robots.txt from assets/scaffold/files/robots.txt
  - Copy [web-root]/update.php from assets/scaffold/files/update.php
  - Copy [web-root]/web.config from assets/scaffold/files/web.config
  - Copy [web-root]/sites/README.txt from assets/scaffold/files/sites.README.txt
  - Copy [web-root]/sites/development.services.yml from assets/scaffold/files/development.services.yml
  - Copy [web-root]/sites/example.settings.local.php from assets/scaffold/files/example.settings.local.php
  - Copy [web-root]/sites/example.sites.php from assets/scaffold/files/example.sites.php
  - Copy [web-root]/sites/default/default.services.yml from assets/scaffold/files/default.services.yml
  - Copy [web-root]/sites/default/default.settings.php from assets/scaffold/files/default.settings.php
  - Copy [web-root]/modules/README.txt from assets/scaffold/files/modules.README.txt
  - Copy [web-root]/profiles/README.txt from assets/scaffold/files/profiles.README.txt
  - Copy [web-root]/themes/README.txt from assets/scaffold/files/themes.README.txt
> DrupalProject\composer\ScriptHandler::createRequiredFiles
PHP 7.0.33-13+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Nov 28 2019 07:43:06) ( NTS )
$ drush updb
 [success] No pending updates.
PHP 7.0.33-13+ubuntu16.04.1+deb.s

Ok so no schema changes required, and if I now point a browser at admin/reports/updates - everything looks ok as can be seen in the screenshot above.

Pin the Composer Dependencies!!!

I've always said that PHP deserves a better dependency management tool than Composer, but alas we are stuck with it. To limit the possibility (likelihood?) of disaster, you should always pin the version numbers for each Drupal project dependency. Pinning means setting a contrib module or theme or profile at a particular version so it cannot be accidentally updated on a composer update command.

So basically I will go through the composer.json file and remove all the tilde or carat symbols, and ensure the version is set to the current version on the system.

{
    "name": "drupal-composer/drupal-project",
    "description": "Project template for Drupal 8 projects with composer",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "authors": [
        {
            "name": "",
            "role": ""
        }
    ],
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ],
    "require": {
        "php": ">=7.0.8",
        "composer/installers": "^1.2",
        "cweagans/composer-patches": "^1.6.5",
        "drupal/addtoany": "1.12",
        "drupal/admin_toolbar": "2.0",
        "drupal/bootstrap": "3.21",
        "drupal/config_filter": "1.5",
        "drupal/config_inspector": "1.0",
        "drupal/config_split": "1.4",
        "drupal/config_update": "1.5",
        "drupal/console": "1.0.2",
        "drupal/core": "8.8.0",
        "drupal/core-composer-scaffold": "8.8.0",
        "drupal/ctools": "3.2",
        "drupal/entity_reference_revisions": "1.7",
        "drupal/fontawesome": "2.14",
        "drupal/geshifilter": "1.2",
        "drupal/google_analytics": "2.4",
        "drupal/libraries": "^3.0.0-alpha1",
        "drupal/memcache": "2.0",
        "drupal/metatag": "1.10",
        "drupal/module_filter": "3.1",
        "drupal/paragraphs": "1.10",
        "drupal/pathauto": "1.6",
        "drupal/schema_metatag": "1.4",
        "drupal/token": "1.5",
        "drupal/xmlsitemap": "1.0.0-alpha4",
        "drupal/youtube": "1.0",
        "drush/drush": "^9.7.1 | ^10.0.0",
        "vlucas/phpdotenv": "^4.0",
        "webflo/drupal-finder": "^1.0.0",
        "zaporylie/composer-drupal-optimizations": "^1.0"
    },
    "require-dev": {
        "drupal/core-dev": "^8.8.0"
    },
    "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "sort-packages": true
    },
    "autoload": {
        "classmap": [
            "scripts/composer/ScriptHandler.php"
        ],
        "files": [
            "load.environment.php"
        ]
    },
    "scripts": {
        "pre-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "pre-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "post-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ],
        "post-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ]
    },
    "extra": {
        "composer-exit-on-patch-failure": true,
        "patchLevel": {
            "drupal/core": "-p2"
        },
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            }
        },
        "installer-paths": {
            "web/core": [
                "type:drupal-core"
            ],
            "web/libraries/{$name}": [
                "type:drupal-library"
            ],
            "web/modules/contrib/{$name}": [
                "type:drupal-module"
            ],
            "web/profiles/contrib/{$name}": [
                "type:drupal-profile"
            ],
            "web/themes/contrib/{$name}": [
                "type:drupal-theme"
            ],
            "drush/Commands/contrib/{$name}": [
                "type:drupal-drush"
            ]
        }
    }
}

The composer.json file has now been modified and the drupal project files are pinned at particular versions for safety.

blog terms

Drupal 8 Development PHP Linux

Drupal 8 as a Static Site: Drupal 8 Contact Form Thank You Page

nigel — Sat, 04 May 2019 13:50:36 +0000

Drupal 8 as a Static Site: Drupal 8 Contact Form Thank You Page nigel Sat, 04/05/2019 - 14:50

Architecture

Our previous two tutorials on the Drupal 8 as a static site covered the Contact Form and the journey through the action to AWS Lambda to process the form and send an email. Now the final part of the contact form journey - returning back to the static site and showing a Thank You page.

In fact in this tutorial the Thank You path will actually be the same path with the contact form on it - so we return to the starting point, but we want to show a message back to the user to thank them for filling out the form. The notification back to the user will have to be JavaScript obviously since we have no backend language functionality.

If you remember the redirect back to the static site from AWS uses a HTTP code of 307. This code matches in the incoming HTTP request with the redirect. So since in our case the form was POSTed, then the redirect will also be POSTed back to the static site. This is both a blessing and a curse as we'll see.

Common wisdom says that JavaScript can neither detect whether an incoming request is POSTed or GETed, nor retrieve any of the POSTed data. That is theoretically true, but there are always workarounds of course :) :)

Adding the JavaScript

This is very similar to our earlier tutorial on how to add the Elasticsearch JavaScript client. So I won't dwell too much on this. Firstly we need to extend our template_preprocess_page function to add our thank-you JS library. This is the second section of code below, underneath the elasticsearch part

{themename}.theme - in my case the custom theme is called beezee8.

function beezee8_preprocess_page(&$variables) {

    // If we are on the search page, load the JS search client api
    // and our implementation to Elasticsearch
    if (\Drupal::routeMatch()->getRouteName() == 'search.view_node_search') {
        $variables['#attached']['library'][] = 'beezee8/elastic-library';
    }

    // If we are on the contact form page, load the JS to show thank you message
    if (\Drupal::request()->getRequestUri() == '/about') {
        $variables['#attached']['library'][] = 'beezee8/thank-you-library';
   }
}
?>

In the code above I get the current path alias and should that equal the '/about' page then I inject my JS library which is defined below.

Libraries file

The libraries file now needs a further entry for our thank you JS. Again we already created an entry for elasticsearch so I am not going to go over this in too much detail.

{themename}.libraries.yml

global-styling:
  css:
    theme:
      css/style.css: {}
elastic-library:
  js:
    js/elasticsearch-js/elasticsearch.min.js: {}
    js/beezee/beezee_elastic.js: {}
thank-you-library:
  js:
    js/beezee/beezee_thankyou.js: {}

JavaScript Thank You

Now here's the JavaScript. I did say that there is no way of knowing whether the HTTP request is GET or POST - well true, but we can deduce it. If we interrogate the referrer and it equals the current URL then we can make the assumption that the form has been POSTed, completed its round trip to AWS Lambda and back with a 307 code, and therefore we should show the thank you message. If you have a look at my code below you will see the conditional, and you will see the markup I inject into the page - it's classic Drupal 8 Bootstrap since that's how my theme is built. Your mileage here will obviously vary dependent upon your own theme.

{themename}_thankyou.js

(function ($, Drupal) {
 
    var thank_once;
 
    function BeezeeThankyou() {
        if (!thank_once) {
            thank_once = true;
 
            // If we are arriving here from lambda which passes on the referrer from here
            // then the form has been POSTed so show the thank you page
            if (document.referrer == document.URL) {
 
                // Markup to add the thankyou message
                var thanks = [
                    '',
                    '',
                    'Status message
',
                    'Thank you for contacting me. I will be in touch.',
                    '
'
                ];
 
                // Inject the markup
                $('.region-highlighted').append(thanks.join(''));
            }
        }
    }
 
    Drupal.behaviors.beezee_thankyou = {
        attach: function (context, settings) {
            BeezeeThankyou();
        }
    };
})(jQuery, Drupal);

nginx Server Configuration

If we try our form now (or at least if I do since I'm using nginx) - we will get an nginx 405 Not Allowed error. Remember I said the POST could be a blessing and a curse? The error is because by default nginx will not allow a POST to a static page. Thankfully there is a workaround. Locate the server configuration for your particular virtual host, and edit it by adding the following line into the server section.

{VM name}.conf

server {
.
.
.
      error_page  405     =200 $uri;
.
.

Form submission

Ok we should now be good to go. Submit the form and it should do the round trip from static site to AWS Lambda back to static site and an email arriving in your inbox!

An even better solution?

If you look carefully above you will see a placeholder in the thank you message. So am I saying it is possible to retrieve the POSTed data and insert the name into the placeholder? Sort of. There is a JS library called jQuery-PostCapture which can achieve this. It works by capturing the POSTed data when the form is initially submitted. This copy of the data is stored either as a cookie or in the browser's local storage. Once the trip to AWS Lambda is completed, this data can be fetched from the cookie or local storage - so the data isn't fetched from the actual POST itself.

The problem I've got with this is neither a cookie nor local storage is secure. However if you feel that the gains (being able to customise a message on the thank you page) is worth the security risk, then give it a shot!

blog terms

Static Site Drupal 8 Drupal PHP Development JavaScript

Drupal 8 as a Static Site: AWS API Gateway, Lambda and SES Form Processing

nigel — Wed, 02 Jan 2019 10:38:46 +0000

Drupal 8 as a Static Site: AWS API Gateway, Lambda and SES Form Processing nigel Wed, 02/01/2019 - 10:38

Serverless Installation

The intention is to build our AWS ecosystem using the serverless product. This will enable us to configure our AWS provisioning in a YML file which is then translated into AWS CloudFormation orchestration. In the same codebase our PHP Lambda function can be built out. It isn't my intention to provide heavy documentation here since the Contact Form architecture was defined in an earlier blog, and I have previously written many articles on the serverless / lambda / API Gateway combination of technologies to deliver HTTP content and handle form POSTs.

Notwithstanding this, I will of course document points of interest along the way so it all makes sense!

The first step is to install serverless on your system. A dependency is node, so if you don't have that then please follow tutorials elsewhere on the Internet before executing the command below to install globally serverless.

$ npm install serverless -g

We will be using Andy Raines' excellent PHP for AWS Lambda via Serverless Framework repo as our stating point. Issuing the following command will install our PHP serverless project into a directory called d8-contact-form.

$ serverless install --url https://github.com/araines/serverless-php -n d8-contact-form
Serverless: Downloading and installing "serverless-php"...
Serverless: Successfully installed "d8-contact-form"

We then cd into the directory and list.

$ cd d8-contact-form/
$ ls -las
total 48
0 drwxrwxr-x 16 501 dialout  512 Jan  2 11:14 .
0 drwxr-xr-x 14 501 dialout  448 Jan  2 11:14 ..
4 -rwxr-xr-x  1 501 dialout  660 Feb 14  2018 buildphp.sh
4 -rw-r--r--  1 501 dialout  569 Feb 14  2018 CHANGELOG.md
4 -rw-r--r--  1 501 dialout  430 Feb 14  2018 composer.json
0 drwxrwxr-x  3 501 dialout   96 Jan  2 11:14 config
4 -rw-r--r--  1 501 dialout 1011 Feb 14  2018 dockerfile.buildphp
4 -rw-r--r--  1 501 dialout   40 Feb 14  2018 .gitattributes
4 -rw-r--r--  1 501 dialout  120 Feb 14  2018 .gitignore
4 -rw-r--r--  1 501 dialout 1330 Feb 14  2018 handler.js
4 -rw-r--r--  1 501 dialout  829 Feb 14  2018 handler.php
4 -rw-r--r--  1 501 dialout 1103 Feb 14  2018 LICENSE
4 -rwxr-xr-x  1 501 dialout  133 Feb 14  2018 php
4 -rw-r--r--  1 501 dialout 3705 Feb 14  2018 README.md
4 -rw-r--r--  1 501 dialout 3008 Jan  2 11:14 serverless.yml
0 drwxrwxr-x  5 501 dialout  160 Jan  2 11:14 src

Since I don't have Git LFS installed, the PHP executable hasn't been downloaded from the repo correctly. I will download this manually, and add it to the directory. Once completed, we can see it there with:

$ ls -lash | grep " php"
 27M -rwxr-xr-x  1 501 dialout  27M Jan  2 11:22 php

Now we need to install the PHP dependencies using composer. This does have a dependency on PHP 7.1.

$ composer install -o --no-dev
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Loading composer repositories with package information
Updating dependencies
Package operations: 9 installs, 0 updates, 0 removals
  - Installing psr/log (1.1.0): Downloading (100%)         
  - Installing monolog/monolog (1.24.0): Downloading (100%)         
  - Installing symfony/polyfill-ctype (v1.10.0): Loading from cache
  - Installing symfony/filesystem (v4.2.1): Downloading (100%)         
  - Installing symfony/config (v4.2.1): Downloading (100%)         
  - Installing symfony/contracts (v1.0.2): Downloading (100%)         
  - Installing psr/container (1.0.0): Loading from cache
  - Installing symfony/dependency-injection (v4.2.1): Downloading (100%)         
  - Installing symfony/yaml (v4.2.1): Downloading (100%)         
Writing lock file
Generating optimized autoload files

Any Raines' code comes with a very basic Lambda function called hello. To prove the environment is working correctly, we should invoke it locally (no need for deploying to AWS yet) and if everything is correct, it'll look like this:

$ serverless invoke local -f hello
Got event [] []
{
    "statusCode": 200,
    "body": "Go Serverless v1.0! Your function executed successfully!"
}

In an earlier blog I replaced the references of Hello with demo. I have changed it to D8ContactForm for the contact form. By deploying the code to AWS, and submitting the contact form, you will see the same screenshot as shown at the end of the previous blog.

SES Validation of email address

To use the AWS SES service, your email address will need to be verified. This is easy - navigate to Simple Email Service and click on email addresses, then "Verify a new email address". Add the email address and a verification email message will be sent for you to click back and confirm the verification process. Once verified, you'll see the second screenshot above.

Serverless configuration

The next step is to add the SES configuration to the serverless.yml file. Add a custom area for your own variables - in this case the email address.

custom:
  SENDER_EMAIL: xxxx@xxxxxxxx
  RECIPIENT_EMAIL: xxxx@xxxxxxxx

And add the SES config to the provider area so it looks like this in its entirety:

provider:
  name: aws
  runtime: nodejs6.10
  stage: dev
  region: eu-west-1
  environment:
    SENDER: ${self:custom.SENDER_EMAIL}
    RECIPIENT: ${self:custom.RECIPIENT_EMAIL}
    DOMAIN: "*"
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "ses:SendEmail"
      Resource: "*"

Then deploy

$ sls deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...

Add AWS SDK to the codebase using composer

The SDK will make it much easier to build out the SES interface code and can be installed to the codebase in the usual composer way

$ composer require aws/aws-sdk-php
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Using version ^3.90 for aws/aws-sdk-php
./composer.json has been updated

Add SDK PHP required libraries

The AWS PHP SDK requires the filter and simplexml libraries that aren't in the PHP executable we deploy to AWS by default. Thankfully there is a way of recompiling the PHP executable and I covered this in an earlier blog. So the tl;dr is

Add the filter and simplexml libraries to dockerfile.buildphp
Make a note of the php creation date and filesize.
Make sure buildphp.sh is executable on the command line and run it.

Add the Serverless and SDK handler code

Now the AWS SDK is in the code base we can build out the functionality to integrate the SDK. Our first work is to ensure that the recipient and sender email addresses are available to the code. To achieve this, they were defined in the serverless.yml in the environment section which can be picked up in handler.php and assigned to the $event array which is made available to the handler class. Add the following lines before the call to handle

$event['sender_email'] = getenv('SENDER');
$event['recipient_email'] = getenv('RECIPIENT');
?>

Now the handler itself. A few things to note.

I have added the necessary use clauses to the AWS SDK client and the exception class
The protected $client_config holds the configuration we have - but note that further down I am getting the credentials from the CredentialProvider
The recipients are an array and so it is possible if required to add multiples here.

D8ContactFormHandler.php

namespace Raines\Serverless;

require 'vendor/autoload.php';

use Aws\Ses\SesClient;
use Aws\Exception\AwsException;

class D8ContactFormHandler implements Handler
{
    protected $client_config = [
        'region' => 'eu-west-1',
        'version' => '2010-12-01',
        'credentials.cache' => TRUE,
        'validation' => FALSE,
    ];

    /**
     * {@inheritdoc}
     */
    public function handle(array $event, Context $context)
    {
        $logger = $context->getLogger();
        // Comment out for debugging
        //$logger->notice('Got event', $event);

        // Set up AWS SDK
        $this->client_config['credentials'] = \Aws\Credentials\CredentialProvider::env();
        $SesClient = new SesClient($this->client_config);

        $sender_email = $event['sender_email'];
        $recipient_emails[] = $event['recipient_email'];

        return [
            'statusCode' => 200,
            'body' => 'Go Serverless v1.0! Your function executed successfully!',
        ];
    }
}

?>

Email generation

Now we can insert the code to actually generate the email. All of this is self evident and really doesn't need further commentary from me. It should be inserted just before the return statement.
D8ContactFormHandler.php

        // Process the submitted form.
        // *TODO* This could do with more validation.
        $fields = [];
        parse_str($event['body'], $fields);

        if (!isset($fields['name'])) $fields['name'] = '{unknown name}';
        if (!isset($fields['mail'])) $fields['mail'] = '{unknown email address}';
        if (!isset($fields['subject'][0]['value'])) $fields['subject'][0]['value'] = '{unknown subject}';
        if (!isset($fields['message'][0]['value'])) $fields['message'][0]['value'] = '{unknown message}';

        $subject = '[badzilla.co.uk website feedback] '.$fields['subject'][0]['value'];
        $plaintext_body = 'From: '.$fields['name'].'  '.$fields['mail'].
            'Subject: '.$fields['subject'][0]['value'].
            ' Message: '. $fields['message'][0]['value'];
        $html_body =  ''.$fields['subject'][0]['value'].'
'.
            ''.'From: '.$fields['name'].'  .$fields['mail'].'">'.$fields['mail'].''.'
'.
            ''.$fields['message'][0]['value'].'
';
        $char_set = 'UTF-8';

        try {
            $result = $SesClient->sendEmail([
                'Destination' => [
                    'ToAddresses' => $recipient_emails,
                ],
                'ReplyToAddresses' => [$sender_email],
                'Source' => $sender_email,
                'Message' => [
                    'Body' => [
                        'Html' => [
                            'Charset' => $char_set,
                            'Data' => $html_body,
                        ],
                        'Text' => [
                            'Charset' => $char_set,
                            'Data' => $plaintext_body,
                        ],
                    ],
                    'Subject' => [
                        'Charset' => $char_set,
                        'Data' => $subject,
                    ],
                ],
            ]);
            $messageId = $result['MessageId'];
        } catch (AwsException $e) {
            // output error message if fails
            $logger->notice('Message', $e->getMessage());
            $logger->notice('AWS Message', $e->getAwsErrorMessage());
        }
?>

Redirect to thank you page

Once we've sent the email we want to redirect back to our thank you page. This is done with a 307 code which preserves the original POSTed values so they can be used on the forwarded page. Our use will be to make the message personalised. I have decided that I would like my redirect back to my originating page, but of course you could customise it as you see fit. For my case, I use the Referer header. The code below replaces the existing return functionality.
D8 ContactFormHandler.php

        return [
            'headers' => ['Location' => $event['headers']['Referer']],
            'statusCode' => 307,
        ];
codE>?>

Debugging

During the development process you will have to do multiple deploys and doubtless you will get numerous error responses from AWS. The best way of working these issues is to use CloudWatch, and a sample screenshot is shown above. The codebase has an excellent logging facility to check variables and these are written out to CloudWatch. Have a look at:

$logger->notice('Got event', $event);
?>

This can be amended for whatever you need to interrogate, but remember the second parameter must be an array.

Git Repository

The entire codebase discussed here is at https://github.com/sanddevil/serverless-php-contact-form - if you wish to extend its functionality, please send me a pull request.

blog terms

Static Site Drupal Drupal 8 PHP Development serverless serverless-php lambda

Drupal 8 as a Static Site: Drupal 8 Contact Form

nigel — Mon, 31 Dec 2018 10:38:02 +0000

Drupal 8 as a Static Site: Drupal 8 Contact Form nigel Mon, 31/12/2018 - 10:38

Drupal Contact

Drupal core comes with a Contact form, so the first step is to check it is already enabled by navigating to /admin/modules as per the first screenshot above.

Next go to /admin/structure/contact and the default forms are listed including the Website Feedback form which is the one we will be using. See the second screenshot.

The third screenshot shows the default fields. I've made two changes here. The form preview requires a Drupal backend so has been removed, and I've disabled the 'Send copy to sender' option since that could encourage spamming.

Add contrib module Contact Block

Since I want my contact form to be below my 'about' content on my /about route, I need the form to be available as a block. This can be achieved using the contrib module Contact Block and can be installed and enabled in the normal way.

$ composer require drupal/contact_block
Using version ^1.4 for drupal/contact_block
./composer.json has been updated
Gathering patches for root package.
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.
  - Installing drupal/contact_block (1.4.0): Downloading (100%)         
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess
$ drush en contact_block
 [success] Successfully enabled: contact_block

Add contact form block to region

Adding the content form block to a block region should be familiar to most people so I won't go into this in great detail. I have placed it in the content region (first screenshot) at /admin/structure/block, configured it with the title, path restrictions so it only appears on my /about route, and ensured the correct form is to be displayed (second screenshot). The third screenshot shows I have dragged the contact form block to be the final block in the content region.

POST form to external URL

Since it is not possible in the admin UI to add an external URL for the POST action, we will have to achieve this by a few lines of code. But where are we POSTing the form anyway? This is covered in the next blog, but in a nutshell the URL is autogenerated by AWS API Gateway during the CloudFormation orchestration. I won't go into the details now how that is achieved since it can be read later in this sequence, but here is the url we'll use.

$ sls info | grep POST
  POST - https://djbbm406h0.execute-api.eu-west-2.amazonaws.com/dev

The easiest way of adding that url to the Drupal contact form would be to include it into settings.php and then use a template preprocess function to swap out the 'action' value in the form.
settings.php

$config['aws:lambda']['url'] = 'https://djbbm406h0.execute-api.eu-west-2.amazonaws.com/dev';
?>

{themename}.theme

/**
 * @param $variables
 */
function beezee8_preprocess_form(&$variables) {

  // find the contact form and add the lambda contact form action url
  if ($variables['attributes']['id'] == 'contact-message-feedback-form') {
    $variables['attributes']['action'] = \Drupal::config('aws:lambda')->get('url', FALSE);
  }
}
?>

Submission of the contact form

Now the form will be submitted to AWS Lambda for onward processing, and ultimately, a redirect back with a thank you message.

blog terms

Static Site Drupal Drupal 8 PHP Development

Drupal 8 as a Static Site: Disable Core Search

nigel — Sun, 18 Nov 2018 10:04:27 +0000

Drupal 8 as a Static Site: Disable Core Search nigel Sun, 18/11/2018 - 10:04

Since the static site will not have access to the Drupal 8 Core Search facility, it can be removed by using the command drush pm_uninstall search on the command line, or by visiting admin/modules/uninstall.

The problem with that approach is core search also has the search form that appears as a block in the navbar on every page of my website. If I uninstall core search I lose the form at the same time. It is my desire to keep on using the search form despite having an Elasticsearch backend. The proposed Elasticsearch JavaScript client will be installed on the /search route which is where the core search form action is directed to.

Okay, so feasibly I could uninstall the core search and do either:

Create a custom block with a search form that replicates (from an HTML frontend perspective) the core search form.
Copy the generated markup from the core search form, and paste it into the page template.

I don't like either approach. I would prefer to keep the core search and stub it so it doesn't actually do any backend searching. That way I will have near identical functionality for both my sandbox static and sandbox Drupal sites.

To stub the search I would have to alter the default search route and replace it with my own custom route with a controller that returns nothing.

Define the route subscriber config

The first step is to add to our custom module's services file.
badzilla_static.services.yml

  <-- previous entry snipped -->
  badzilla.route_subscriber:
      class: Drupal\badzilla_static\Routing\RouteSubscriber
      tags:
        - { name: event_subscriber }

We have now defined where our route subscriber lives. Let's create.

Route subscriber class

We need to create the directory structure and the file itself first.

$ mkdir  src/Routing
$ touch src/Routing/RouteSubscriber.php

And the file contents are:
RouteSubscriber.php

/**
* @file
* Contains \Drupal\mymodule\Routing\RouteSubscriber.
*/

namespace Drupal\badzilla_static\Routing;

use Drupal\Core\Routing\RouteSubscriberBase;
use Symfony\Component\Routing\RouteCollection;



/**
* Listens to the dynamic route events.
*/
class RouteSubscriber extends RouteSubscriberBase {

    /**
    * {@inheritdoc}
    */
    public function alterRoutes(RouteCollection $collection)
    {
        // Replace dynamically created "search.view_node_search" route's Controller with my own.
        if ($route = $collection->get('search.view_node_search')) {
            $route->setDefault('_controller', '\Drupal\badzilla_static\Controller\BadzillaStaticSearchController::view');
        }
    }
}
?>

Here I am replacing the built-in search module's controller with my own. Now let's build out my own do-nothing controller.

Custom search controller

As previously, create the file structure.

$ mkdir src/Controller
$ touch src/Controller/BadzillaStaticSearchController.php

Here's the controller in all its glory.
BadzillaStaticSearchController.php

namespace Drupal\badzilla_static\Controller;

use Drupal\search\SearchPageInterface;
use Symfony\Component\HttpFoundation\Request;
use Drupal\search\Controller\SearchController;

/**
 * Override the Route controller for search.
 */
class BadzillaStaticSearchController extends SearchController {

    /**
     * {@inheritdoc}
     */
    public function view(Request $request, SearchPageInterface $entity)
    {
        $build = [];
        $plugin = $entity->getPlugin();
        $build['#title'] = $plugin->suggestedTitle();

        $results = [];
        $build['search_results'] = [
            '#theme' => [
                'item_list__search_results__' . $plugin->getPluginId(),
                'item_list__search_results',
            ],
            '#items' => $results,
            '#empty' => [
                '#markup' => '',
            ],
            '#list_type' => 'ol',
            '#context' => [
                'plugin' => $plugin->getPluginId(),
            ],
        ];

        return $build;
    }
}
?>

The view function must return a render array, and so I have honoured the convention of the parent SearchController::view to large extent although obviously in my case there will always be no results returned. A useful stub!

Clear Rebuild and Test

Since we've added to our services yaml file we need to do the customary cache rebuild for our changes to take effect.

$ drush cr
 [success] Cache rebuild complete.

Next let's attempt a search on the dynamic and the static version of my sandbox Badzilla sites. The two screenshots above show that both the static and the dynamic version of the sandbox sites are correctly showing a blank search page. Next we have to plug in the client....

blog terms

Static Site Drupal 8 Development Drupal PHP

Drupal 8 as a Static Site: Manually Add Paths

nigel — Tue, 13 Nov 2018 09:24:34 +0000

Drupal 8 as a Static Site: Manually Add Paths nigel Tue, 13/11/2018 - 09:24

We established in my earlier blog Drupal 8 as a Static Site: Problems Identified that Tome isn't capable of expanding wildcards such as the taxonomy RSS subscription paths in Views at taxonomy/term/%/feed. This tutorial shows how to manually add paths so when the command drush tome:static is run they are created correctly in the static version of the website.

Thankfully the Tome modules makes this relatively easy although we will need to develop a custom event subscriber. The subscriber will be looking for a collect paths event which is triggered when the drush tome:static command is run. At that point we can add our missing paths. In our case it will be all the taxonomy term feed paths which the Badzilla website uses.

So the first thing to do is to create a custom module and an event subscriber within it.

Create a skeleton custom module

Let's create a custom module

$ cd docroot/modules/custom
$ mkdir badzilla_static
$ cd badzilla_static/
$ touch badzilla_static.info.yml

And populate badzilla_static.info.yml

name: Badzilla Static
description: Custom code for facilitating the generation of a static version of Badzilla
package: Badzilla
type: module
core: 8.x

Now we need the services yml file which will hold the event subscriber namespace and parameter information.

$ touch badzilla_static.services.yml

and let's populate badzilla_static.services.yml

services:
  badzilla.route_path_subscriber:
    class: Drupal\badzilla_static\EventSubscriber\RoutePathSubscriber
    arguments: ['@router.route_provider']
    tags:
      - { name: event_subscriber }

Now we need to create the RouteEventSubscriber so let's create the directory

$ mkdir -p src/EventSubscriber
$ touch src/EventSubscriber/RoutePathSubscriber.php

Event Subscriber

The event subscriber must implement the EventSubscriberInterface. Let's have a look at the code before the commentary.

namespace Drupal\badzilla_static\EventSubscriber;

use Drupal\Core\Routing\RouteProviderInterface;
use Drupal\tome\Event\CollectPathsEvent;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;
use Drupal\tome\Event\TomeEvents;



/**
 * Adds route paths to the list of paths to export.
 */
class RoutePathSubscriber implements EventSubscriberInterface
{

    /**
     * The route provider.
     *
     * @var \Drupal\Core\Routing\RouteProviderInterface
     */
    protected $routeProvider;


    /**
     * Constructs the RoutePathSubscriber object.
     *
     * @param \Drupal\Core\Routing\RouteProviderInterface $route_provider
     *   The route provider.
     */
    public function __construct(RouteProviderInterface $route_provider) {
        $this->routeProvider = $route_provider;
    }


    /**
     * Reacts to a collect paths event - adds the taxonomy feeds paths
     *
     * @param \Drupal\tome\Event\CollectPathsEvent $event
     *   The collect paths event.
     */
    public function collectPaths(CollectPathsEvent $event) {
        $terms = \Drupal::entityTypeManager()->getStorage('taxonomy_term')->loadTree('technology');
        if (is_array($terms) && count($terms)) {
            foreach($terms as $term) {
                $event->addPath(sprintf('/taxonomy/term/%d/feed/feed.rss', $term->tid));
            }
        }
    }


    /**
     * {@inheritdoc}
     */
    public static function getSubscribedEvents() {
        $events[TomeEvents::COLLECT_PATHS][] = ['collectPaths'];
        return $events;
    }
}
?>

The code is largely a copy / paste exercise from Tome's own event subscriber but the difference is in the method collectPaths(). My objective is to get the tids from the vocabulary called technology I use on my listing pages. To achieve this I load all the terms of that vocabulary, iterate through them, and inject the tids into the path /taxonomy/term/%/feed/feed.rss. Note how I have added feed.rss to the url? This is so Tome, seeing a file extension will not create the standard index.html file in the feed directory.

This solution does require a configuration change at the web server level and in the Views. See the chapter Drupal 8 as a Static Site: Add feed.rss to web server config and views

Enabling and running

Now we have the coding complete. we need to enable the module the usual way:

$ drush en badzilla_static -y
 [success] Successfully enabled: badzilla_static

Now when I run the drush tome:static I see:

Generating static HTML...
 260/260 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

as opposed to the lower value of paths before my code:

Generating static HTML...
 227/227 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

Also if I ls the directory with the terms I will now see my entries correctly:

$ ls  ../static/html/taxonomy/term/          
100  102  104  106  109  111  13  20  27  36  47  53  55  78  80  82  84  86  88  9   91  93  95  97  99
101  103  105  107  110  112  18  26  32  44  49  54  57  79  81  83  85  87  89  90  92  94  96  98

blog terms

Static Site Drupal 8 Development PHP

Launch of the BadzillaVM!

nigel — Wed, 10 Oct 2018 20:17:50 +0000

Launch of the BadzillaVM! nigel Wed, 10/10/2018 - 21:17

Welcome to the BadzillaVM

Welcome to the BadzillaVM!

The Repository can be cloned from here: https://github.com/sanddevil/badzillavm

BadzillaVM is a lightweight VM for Drupal, based heavily on Geerlingguy's DrupalVM, and just like his, built with Ansible.

BadzillaVM is significantly pared down from DrupalVM, and as such it isn't as feature-rich as DrupalVM. It does however offer something DrupalVM does not.

It offers the ability to run four different version of PHP (web) simultaneously. In addition, there is a simple mechanism to switch between PHP (cli) versions on the command line.

Currently, PHP 5.6, 7.0, 7.1 and 7.2 are supported.

This means that BadzillaVM is useful for agencies that perhaps have multiple clients running Drupal builds on differing versions of PHP. It is a breeze with BadzillaVM to configure multiple virtual hosts, each of which can be assigned any of the available versions of PHP.

BadzillaVM needs Vagrant and VirtualBox to run. You should install those two products first at their latest versions.

BadzillaVM is a simple VM with lightweight functionality. It isn't intended to support complex Drupal ecosystems. Developers seeking an advanced product set should have a look at DrupalVM instead.

BadzillaVM installs to a Ubuntu 16.04LTS virtual machine. The build contains:

nginx
PHP 5.6, 7.0, 7.1, 7.2
MariaDB Server and Client
Memcached
Drush Loader
Firewall
Java
Jenkins
Mailhog
Adminer
Pimpmylog
Composer
NodeJS
XHPRof
XDebug

This is a useful toolset for building professional standard Drupal websites

Installation Guide

1 - Install Vagrant and VirtualBox

Download and install Vagrant and VirtualBox

You can also use an alternative provider like Parallels or VMware. (Parallels Desktop 11+ requires the "Pro" or "Business" edition and the Parallels Provider, and VMware requires the paid Vagrant VMware integration plugin).

Notes:

For faster provisioning (macOS/Linux only): Install Ansible on your host machine, so Drupal VM can run the provisioning steps locally instead of inside the VM.
For stability: Because every version of VirtualBox introduces changes to networking, for the best stability, you should install Vagrant's `vbguest` plugin: `vagrant plugin install vagrant-vbguest`
NFS on Linux: If NFS is not already installed on your host, you will need to install it to use the default NFS synced folder configuration. See guides for Debian/Ubuntu, Arch, and RHEL/CentOS.
Version: Make sure you're running the latest releases of Vagrant, VirtualBox, and Ansible—as of early 2018, BadzillaVM recommends: Vagrant 2.0.x, VirtualBox 5.2.x, and Ansible 2.4.x

2 - Build the Virtual Machine

Download this project and put it wherever you want.
Create a file called `config.yml` in the directory in which you downloaded or cloned this repository.
Populate this file with your configuration (see below) This is mandatory - do not go past this step until you have added your own configuration into this file.
Type in `vagrant up`, and let Vagrant do its magic.

3 - Configuring the VM

You must configure your VM before you attempt `vagrant up` or the build will error.

A typical `config.yml` configuration will look similar to this:

---
vagrant_hostname: nigel-dev-box
 
user_vhosts_conf:
  web1:
    file_name: web1.conf 
    template_name: template-drupal8.conf.j2
    template_params:
      server_name: web1.test      
      root: /var/www/html/web1    
      php_version: 5.6
  meedjum:
    file_name: meedjum.conf 
    template_name: template-drupal8.conf.j2
    template_params:
      server_name: meedjum.test      
      root: /var/www/html/meedjum/docroot   
      php_version: 7.0
 
vagrant_synced_folders:
  - local_path: /Users/nigel/Projects/VHosts
    destination: /var/www/html
    type: nfs
    create: true

You must start the file with a line containing three hyphens.

You should then add a hostname. I've called mine nigel-dev-box.

I then define two virtual hosts, named web1 and meedjum respectively. file_name refers to the file that hold the nginx configuration. The template_name can be any defined in `roles/TroodoNmike.nginx-conf/templates`. The template_params is a list of the token substitutions which are the server_name, root, and php_version.

Finally the synchronise folders must be defined. Ensure you stick to the convention above. The values are self explanatory and they are an array so multiple values can be defined.

It is not a bad idea to ensure this configuration is saved in a vcs. To that end, I create my `config.yml` file in a sibling directory to BadzillaVM, and simlink it into the BadzillaVM directory. That way it is separate from the BadzillaVM repo, yet Vagrant will find it.

4 - Access the VM

The dashboard can be accessed by opening clicking through to the dashboard. To ssh into the VM, simply type `vagrant ssh`.

5 - Switching Versions of PHP

All four web versions of PHP run concurrently without any need for interaction. However it is necessary to manually switch between CLI versions of PHP. To do this, whilst inside the VM, simply issue the `switch-php` command:

$ switch-php
There are 4 choices for the alternative php (providing /usr/bin/php).
 
  Selection    Path             Priority   Status
------------------------------------------------------------
  0            /usr/bin/php7.2   72        auto mode
  1            /usr/bin/php5.6   56        manual mode
* 2            /usr/bin/php7.0   70        manual mode
  3            /usr/bin/php7.1   71        manual mode
  4            /usr/bin/php7.2   72        manual mode
 
Press <enter> to keep the current choice[*], or type selection number:

To help you remember which version of PHP you are currently using on the CLI, simply refer to the prompt. It will show something like:

PHP 7.0.32-1+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Oct  1 2018 11:45:35) ( NTS )
vagrant@nigel-dev-box:~$

License

This project is licensed under the MIT open source license.

Credits

Jeff Geerling's Drupal VM was both the inspiration and the starting point for this project. Huge thanks to the incredible work Jeff has done. His amazing book led me to try my own VM build. Please check his book out: Ansible for DevOps

About the Author

Badzilla (Nigel Milligan) is a London UK based Freelance Drupal Architect and Consultant and can be contacted via LinkedIn

blog terms

Drupal PHP VirtualBox BadzillaVM

PHP

Hosting Drupal 9 on AWS Fargate Part 2: Bitbucket Pipeline

Hosting Drupal 9 on AWS Fargate Part 1: Local Development

Drupal Core + FPM PHP + Codebase

Drupal Core + FPM PHP + Codebase + Exec command

Memcached

nginx

AWS Fargate: CI/CD Architecture for Drupal and PHP Frameworks

Drupal 8: Move Deprecated drupal/drupal Composer Template to drupal-composer/drupal-project

Drupal 8 as a Static Site: Drupal 8 Contact Form Thank You Page

Status message

Drupal 8 as a Static Site: AWS API Gateway, Lambda and SES Form Processing

'.$fields['subject'][0]['value'].'

'.'From: '.$fields['name'].' .$fields['mail'].'">'.$fields['mail'].''.'

Drupal 8 as a Static Site: Drupal 8 Contact Form

Drupal 8 as a Static Site: Disable Core Search

Drupal 8 as a Static Site: Manually Add Paths

Launch of the BadzillaVM!