Docker

AWS Fargate Static Analysis with PHP

nigel — Mon, 13 Jan 2020 14:28:49 +0000

AWS Fargate Static Analysis with PHP nigel Mon, 13/01/2020 - 14:28

In my earlier blog on AWS Fargate Automated Testing I spoke about creating an AWS Fargate task definition which would include a web app's codebase and the Cucumber / Gherkin / maven / Java runtime stack. This tutorial is an opportunity to look at the static analysis stage of the pipeline where code is inspected for vulnerabilities, coding standards, compilation errors, copy and paste detection amongst others. Since my background is in Drupal and PHP, my tests will be focused around the PHP language and will use the excellent Static Analysis Tools for PHP repo.

Looking at the jakzal/phpqa repo, it is clear there is a great deal of tooling available. The purpose of this tutorial is take a small sample of these tools and create a Fargate task to run them against my codebase. My somewhat arbitrary selection is:

phpcpd - Copy/paste detection
phpcs - Codesniffer that detects coding standards violations
phplint - PHP syntax error checker.

Ok - let's get started.

Create the Dockerfile and the Entrypoint

As I have previously mentioned, Fargate is a black box and it doesn't lend itself to command line invocation of the tooling. Typically locally on a laptop we would run the jakzal/phpqa Docker image like:

$  docker run -it --rm -v $(PWD):/project -w /project  jakzal/phpqa phpcpd web/core/modules/views/src
$  docker run -it --rm -v $(PWD):/project -w /project  jakzal/phpqa phpcs web/core/modules/views/src
$  docker run -it --rm -v $(PWD):/project -w /project  jakzal/phpqa phplint web/core/modules/views/src

This could obviously be placed in a shell script. Now in Fargate we don't have access to the command line and we have to create a Fargate definition up front before that can be provisioned and then run. So it's a little bit more involved.

The simplest approach is to create a Dockerfile based on the jakzal/phpqa repo, and an extrypoint script that will run each one of the tools I have identified.

The Dockerfile is defined below. Note that I have selected PHP version 7.2. This is the closest supported version of PHP to what I'm running on my production box. Think I need to upgrade that at some point..

FROM jakzal/phpqa:1.28.1-php7.2
ADD entry.sh /entry.sh
RUN /bin/bash -c 'chmod +x /entry.sh'
ENTRYPOINT ["/entry.sh"]

Next up is the entry.sh file. First thing to note is I have added the set +e flag, which is default anyway. This is included to ensure that even if I get some pernickety error messages the script won't terminate immediately, and it acts as an aide memoire that we don't want error termination so don't accidentally copy paste in a competing set -e flag.

There is then a check whether the shared ECS volume has been loaded - this is the responsibility of the codebase container, but because containers can start at varying times, I have to ensure I don't run the PHP analysis before the volume is ready.

The path to the source code to perform the analysis on is defined as /var/static_tests/web/core/modules/views/src. This has two components. The first part /var/static_tests/ is the mount point which will correspond to my codebase top level directory. The second component, web/core/modules/views/src points to the core contributed Drupal Views module. This is purely there as a reference to some arbitrary code. I am using the codebase to my own Badzilla which is a site build with no custom code and thus I have none of my own code to use in this sample.

#!/usr/bin/env sh
 
set +e
 
while [ ! -d /var/static_tests/web ]
do
  sleep 5
  echo "Waiting for volume to be mounted and files copied into position"
done
 
phpcpd /var/static_tests/web/core/modules/views/src
phpcs /var/static_tests/web/core/modules/views/src
phplint /var/static_tests/web/core/modules/views/src

Building the Docker image locally

It isn't mandatory to test everything works before constructing the Fargate task definition, but it makes sense to me to test my work locally first. If the Fargate task subsequently fails then I know the problem area will be localised to the AWS eco-system, and probably to Fargate. So I always build locally and run locally first. To build I cd into my Dockerfile / entry.sh entry and issue:

$ docker build -f Dockerfile -t static-fargate  .

To run it's

$ docker run -it --rm -v $(PWD):/var/static_tests  static-fargate

You can see above I am establishing the Docker volume which will be shared with my codebase.

The Docker Hub repo for this is at https://hub.docker.com/repository/docker/sanddevil/static-fargate and the GitHub repo for the code is at https://github.com/sanddevil/devops-static-php

Define the Fargate task

With everything working locally it's time to build the Fargate task. This is very similar to the script I created in my earlier automated testing blog. There are a few differences obviously. My container stack comprises of a combined PHP + Apache + my own codebase image, and the PHP static analysis jakzal/phpqa obviously. I use the shell script heredoc convention to define each container. You should note I have now expanded my codebase container to include two shared volumes - one for the automated tests (not discussed here but discussed previously) and the static tests.

#!/bin/bash
 
 
BRANCH="master"
REPO="php7.0-apache/${BRANCH}"
 
# Repo account and repo
ECR_ACCOUNT="xxxxxxxxxxx.dkr.ecr.eu-west-2.amazonaws.com"
ECR_REPO="${ECR_ACCOUNT}/${REPO,,}"
ECS_CLUSTER="FargatePrototype"
 
# Get the Task Role ARN for executing the task
TASK_ARN=`aws iam get-role --role-name ecsTaskExecutionRole | jq .Role.Arn -r`
 
 
# Execution ARN appears to be the same as the Task ARN
EXECUTION_ARN="${TASK_ARN}"
 
 
# Determine if the log group already exists: NOTE: "aws ecs register-task-definition" will not create the log group for you!
LOG_EXISTS=`aws logs describe-log-groups | jq ".logGroups[] | select(.logGroupName == \"/ecs/"${BRANCH}"\") | .logGroupName" -r | wc -l`
if [[ "${LOG_EXISTS}" -eq 0 ]]; then
	aws logs create-log-group --log-group-name "/ecs/${BRANCH}"
fi
 
CONTAINER_CODEBASE=$(cat <
)
 
CONTAINER_STATIC_PHP=$(cat <
)
 
 
CONTAINERS_STATIC=$(cat <
)
 
 
aws ecs register-task-definition \
  --family "STATIC-${BRANCH}" \
  --volumes "name=static_tests,host={}" "name=automated_tests,host={}" \
  --task-role-arn "$TASK_ARN" \
  --execution-role-arn "${EXECUTION_ARN}" \
  --network-mode "awsvpc" \
  --container-definitions "${CONTAINERS_STATIC}" \
  --cpu "256" \
  --memory "512" \
  --requires-compatibilities "FARGATE"

I'm not quite done yet though. Somehow I need to copy my codebase into the shared Docker volume so that the PHP static analysis container can see it. This is done in the entry.sh script of my PHP + Apache + Codebase image (not discussed here) but the entry.sh file is shown below for completeness.

#!/bin/bash
set -e
 
# Copy the automated tests into a shared area accessible to maven
cp -R /var/www/html/test  /var/automated_tests/.
 
# Copy the PHP codebase into a shared area accessible to the PHP static analysis container
cp -R /var/www/html/web  /var/static_tests/.
 
# Run Apache in the foreground to prevent the container from quitting
apache2-foreground

Here you can see the copying I do for both the automated tests and the static analysis.

Running the Fargate task

Normally I would invoke the Fargate task programmatically using AWS CLI but since that leads to a rather dry image-free blog, I will use the console in this instance. My previous blog shows how to navigate around the ECS dashboard to run the task, then you can see the task running in the first screenshot above. It will only run until the entrypoint script terminates as per the rules on how Docker works, so to capture that scrennshot I had to be quick. The second screenshot shows the CloudWatch logs for the task, and you can see there are a few copy/paste errors detected.

blog terms

devops AWS AWS CLI ECS Docker bash

AWS Fargate Automated Testing with Cucumber, Gherkin, Java, Maven, Selenium Grid Hub and Chrome and Firefox Nodes

nigel — Mon, 09 Dec 2019 08:37:50 +0000

AWS Fargate Automated Testing with Cucumber, Gherkin, Java, Maven, Selenium Grid Hub and Chrome and Firefox Nodes nigel Mon, 09/12/2019 - 08:37

Tutorial for creating an automated testing AWS Fargate task that will include Docker containers for a full testing framework:

Maven for build automation
Java Runtime to enable the execution of maven and the junit unit testing framework
Java Cucumber - a tool for BDD testing
Gherkin to define behaviours in the English language
Selenium Hub being the central loading point for the tests
Selenium Chrome - one of the nodes
Selenium Firefox - a second node
Apache / PHP - the web app and web server.

This is not a trivial exercise but by the end of the blog I will have a working automated testing environment which can be slotted into a DevOps pipeline. Please note this blog is not a tutorial on how to write Cucumber / Gherkin tests, however I will create a few sample Gherkin behaviours and their respective Java Cucumber stepdefs as a starting point. These can then be expanded out by your QA team.

Starting Point

The starting point will be the simple PHP/Apache web app I put together in my previous blog here - this is a single url website which merely shows the phpinfo() status page. But it's ideal for our automated testing because our testing framework is plenty complex already, and we can still run a few tests against this simple landing page such as whether the additional PHP extensions I included in the Dockerfile are loaded correctly.

Now we need to build out our testing tree structure.

At a sibling level to the docroot directory I created in the previous blog, create the following:

$ mkdir -p mytests/src/test/java/mytests
$ mkdir -p mytests/src/test/resources/mytests

Now it's time to create the files I need

$ cd mytests/
$ touch pom.xml
$ touch src/test/java/mytests/RunCucumberTest.java
$ touch src/test/java/mytests/Stepdefs.java
$ touch src/test/java/mytests/SetupTestDriver.java
$ touch src/test/resources/mytests/home_page.feature

This will create a tree structure like this:

$ tree
.
|-- pom.xml
`-- src
    `-- test
        |-- java
        |   `-- mytests
        |       |-- RunCucumberTest.java
        |       |-- SetupTestDriver.java
        |       `-- Stepdefs.java
        `-- resources
            `-- mytests
                `-- home_page.feature
 
6 directories, 5 files

Now we need to flesh out those files we created. Firstly the pom file which contains a list of the project's dependencies. Careful inspection will show you that I used the hellocucumber template as my starting point.
pom.xml

 version="1.0" encoding="UTF-8"?>
 xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    >4.0.0>
 
    >hellocucumber>
    >mytests>
    >1.0.0-SNAPSHOT>
    >jar>
 
    >
        >4.2.6>
        >UTF-8>
    >
 
    >
        >
            >io.cucumber>
            >cucumber-java>
            >${cucumber.version}>
            >test>
        >
 
        >
            >io.cucumber>
            >cucumber-junit>
            >${cucumber.version}>
            >test>
        >
 
        >
            >junit>
            >junit>
            >4.12>
            >test>
        >
 
    
    >
        >io.github.bonigarcia>
        >webdrivermanager>
        >3.7.1>
    >
 
	
	>
    		>org.seleniumhq.selenium>
    		>selenium-java>
    		>3.141.59>
	>
 
    
    >
        >org.seleniumhq.selenium>
        >selenium-chrome-driver>
        >3.141.59>
    >
 
    
    >
        >org.apache.maven.plugins>
        >maven-surefire-plugin>
        >2.22.2>
    > 
 
    >
        >org.slf4j>
        >slf4j-api>
        >1.5.6>
    >
 
    >
        >org.slf4j>
        >slf4j-log4j12>
        >1.5.6>
    > 
 
    >
 
    >
        >
            >
                >org.apache.maven.plugins>
                >maven-compiler-plugin>
                >3.7.0>
                >
                    >UTF-8>
                    >1.8>
                    >1.8>
                >
            >
        >
    >
>

Now my Gherkin tests themselves. These are rather simple - I am checking for whether my GD and Memecached PHP extensions have been loaded, i.e. they are listed on the front page which just shows the entire PHP configuration. Under normal circumstances Gherkin test tend to be GIVEN..WHEN..THEN constructs, but since I am not actually doing anything (e.g. clicking a button to go somewhere) I only need WHEN..THEN
home_page.feature

Feature: Are we on the Home Page?
 
 
  Scenario: On the home page
    When User is on the Home Page
    Then Message Displayed Memcached
 
    When User is on the Home Page
    Then Message Displayed GD

Next is the StepDef file where I convert the Gherkin into Cucumber Java containing the actual business logic behind the quasi-English Gherkin. I don't pretend to be an expert Java coder - I have the sum total of about 4 days experience. Therefore my code is probably sub-optimal, but it does the job and an in-house Java professional could whip it into shape. I have defined two browsers - Firefox and Chrome as per our Selenium configuration, although please note there is currently no semblance of parallel running of the tests in here. That again has been left to more talented Java practitioners than myself.
StepDefs.java

package mytests;
 
import base.SetupTestDriver;
 
import java.util.concurrent.TimeUnit;
 
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import io.github.bonigarcia.wdm.WebDriverManager;
 
import cucumber.api.java.en.Given;
import cucumber.api.java.en.Then;
import cucumber.api.java.en.When;
 
import java.net.MalformedURLException;
 
import static org.junit.Assert.*;
 
 
public class Stepdefs {
 
	public static WebDriver[] drivers;
	public static String[][] browsers;
 
    @When("User is on the Home Page")
    public void user_is_on_Home_Page() throws Throwable {
       	browsers = new String[][] {
       		{"linux", "firefox", "http://127.0.0.1", "http://127.0.0.1:4444"},
            {"linux", "chrome", "http://127.0.0.1", "http://127.0.0.1:4444"}
        };
 
		drivers = new WebDriver[browsers.length];
        for (int i = 0; i < browsers.length; i++) {
        	drivers[i] = new SetupTestDriver(browsers[i][0], browsers[i][1], browsers[i][2], browsers[i][3]).getDriver();
        }
    }
 
    @Then("Message Displayed Memcached")
    public void message_displayed_memcached() throws Throwable {
        for (int i = 0; i < drivers.length; i++) {
        	Boolean exists = !drivers[i].findElements(By.name("module_memcached")).isEmpty();
        	drivers[i].quit();
        	assertTrue(exists);
        }
    }
 
	@Then("Message Displayed GD")
	public void gd() throws Throwable {
        for (int i = 0; i < drivers.length; i++) {
			Boolean exists = !drivers[i].findElements(By.name("module_gd")).isEmpty();
        	drivers[i].quit();
        	assertTrue(exists);
        }
	}
}

The StepDef file doesn't contain any Firefox or Chrome webdriver startup configuration. This has been placed in a separate Java class.
SetupTestDriver.java

// src/main/java/base/SetupTestDriver.java
package base;
 
import org.openqa.selenium.Platform;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.firefox.FirefoxProfile;
import org.openqa.selenium.firefox.FirefoxDriverLogLevel;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.Dimension;
 
import java.net.MalformedURLException;
import java.net.URL;
import java.util.concurrent.TimeUnit;
 
public class SetupTestDriver {
    private WebDriver driver = null;
    private String browser = null;
    private String baseUrl = null;
    private String os = null;
    private String node = null;
 
    public SetupTestDriver(String os, String browser, String baseUrl, String node) throws MalformedURLException {
        this.browser = browser;
        this.os = os;
        this.baseUrl = baseUrl;
        this.node = node;
 
        Platform platform = Platform.fromString(os.toUpperCase());
        if(browser.equalsIgnoreCase("chrome")) {
            ChromeOptions chromeOptions = new ChromeOptions();
            chromeOptions.addArguments("--headless");
            chromeOptions.setCapability("platform", platform);
            this.driver = new RemoteWebDriver(new URL(node + "/wd/hub"), chromeOptions);
        } else if (browser.equalsIgnoreCase("firefox")) {
            FirefoxOptions firefoxOptions = new FirefoxOptions();
            firefoxOptions.setProfile(new FirefoxProfile());
            firefoxOptions.setCapability("platform", platform);
            firefoxOptions.setHeadless(true);
            this.driver = new RemoteWebDriver(new URL(node + "/wd/hub"), firefoxOptions);
        }
 
        this.driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
        // Maximizing the screen doesn't appear to work.
        this.driver.manage().window().setSize(new Dimension(1600,900));
        this.driver.get(baseUrl);
 
    }
 
    public String getOs() {
        return this.os;
    }
 
    public String getBrowser() {
        return this.browser;
    }
 
    public String getBaseUrl() {
        return this.baseUrl;
    }
 
    public String getNode() {
        return this.node;
    }
 
    public WebDriver getDriver() {
        return this.driver;
    }
}

Finally there's some code to perform the glue between Gherkin and Cucumber.
RunCucumberTest.java

package mytests;
 
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
 
@RunWith(Cucumber.class)
@CucumberOptions(plugin = {"pretty"})
public class RunCucumberTest {
}

Build the Docker Codebase + PHP/Apache

In my previous blog here I showed how to add the codebase to the php:7.2-apache image. The codebase has now changed, and I now have two top level sibling directories docroot and mytests - which contains the automated test code I've just added. There is a change we need to make to the Dockerfile in this revision, and the inclusion of an entrypoint script called entry.sh. The entry.sh file will copy the test code suite into a shared Fargate non-persistent drive so that the Maven container can access the codebase and run the tests against it. Once the copy of files has been completed, Apache must be run in foreground mode to prevent the container from quitting. Below we have the new version of the Dockerfile, and the entry.sh file. The image needs to be rebuilt, but since that was covered in the previous blog, it is omitted here.

Dockerfile

FROM php:7.2-apache
RUN apt-get update && apt-get install -y libmemcached-dev zlib1g-dev \
    libfreetype6-dev \
    libjpeg62-turbo-dev \
    libpng-dev \
    libwebp-dev \
    && pecl install memcached \
    && docker-php-ext-enable memcached \
    && docker-php-ext-configure gd --with-gd --with-webp-dir --with-jpeg-dir \
       --with-png-dir --with-zlib-dir --with-freetype-dir \
    && docker-php-ext-install gd
COPY --chown=www-data:www-data ./ /var/www/html/.
COPY entry.sh /usr/local/bin/.
ENV AH_SITE_ENVIRONMENT devops
ENV APACHE_DOCUMENT_ROOT /var/www/html/docroot
RUN sed -ri -e 's!/var/www/html!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's!/var/www/!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf
RUN /bin/bash -c 'chmod +x /usr/local/bin/entry.sh'

entry.sh

#!/bin/bash
set -e
 
# Copy the automated tests into a shared areaaccessible to maven
cp -R /var/www/html/mytests  /var/automated_tests/.
 
# Run Apache in the foreground to prevent the container from quitting
apache2-foreground

Maven:3.6.2-jdk-13

The Java runtime is bundled with the Maven image if the correct tag is selected. I have opted for maven:3.6.2-jdk-13 which is up to date and supported. This requires a small amount of customisation, so again a Dockerfile will be needed. In this instance I want to put a wait loop in the entrypoint script to ensure that there's no attempt to run the tests before the non-persistent shared drive containing the test suite has been mounted by the d8codebase container. At the end I exec the mvn clean install command to run the tests.

The easiest way of achieving this is to edit the mvn-entrypoint.sh script which is in /usrlocal/bin. I copied out of the image to edit and the Dockerfile copies it back into place.

Using my previous blog I built this, tagged and pushed to ECR using the image name d8maven. The instructions aren't copied here for brevity reasons - please refer to my previous blog if you need to.

Dockerfile

FROM maven:3.6.2-jdk-13
COPY mvn-entrypoint.sh /usr/local/bin/mvn-entrypoint.sh
RUN /bin/bash -c 'chmod +x /usr/local/bin/mvn-entrypoint.sh'

mvn-entrypoint.sh

#! /bin/sh -eu
 
# Copy files from /usr/share/maven/ref into ${MAVEN_CONFIG}
# So the initial ~/.m2 is set with expected content.
# Don't override, as this is just a reference setup
 
copy_reference_files() {
  local log="$MAVEN_CONFIG/copy_reference_file.log"
  local ref="/usr/share/maven/ref"
 
  if mkdir -p "${MAVEN_CONFIG}/repository" && touch "${log}" > /dev/null 2>&1 ; then
      cd "${ref}"
      local reflink=""
      if cp --help 2>&1 | grep -q reflink ; then
          reflink="--reflink=auto"
      fi
      if [ -n "$(find "${MAVEN_CONFIG}/repository" -maxdepth 0 -type d -empty 2>/dev/null)" ] ; then
          # destination is empty...
          echo "--- Copying all files to ${MAVEN_CONFIG} at $(date)" >> "${log}"
          cp -rv ${reflink} . "${MAVEN_CONFIG}" >> "${log}"
      else
          # destination is non-empty, copy file-by-file
          echo "--- Copying individual files to ${MAVEN_CONFIG} at $(date)" >> "${log}"
          find . -type f -exec sh -eu -c '
              log="${1}"
              shift
              reflink="${1}"
              shift
              for f in "$@" ; do
                  if [ ! -e "${MAVEN_CONFIG}/${f}" ] || [ -e "${f}.override" ] ; then
                      mkdir -p "${MAVEN_CONFIG}/$(dirname "${f}")"
                      cp -rv ${reflink} "${f}" "${MAVEN_CONFIG}/${f}" >> "${log}"
                  fi
              done
          ' _ "${log}" "${reflink}" {} +
      fi
      echo >> "${log}"
  else
    echo "Can not write to ${log}. Wrong volume permissions? Carrying on ..."
  fi
}
 
# Wait until we see the test files that are being copied from the PHP/Apache/Codebase container to a shared volume.
while [ ! -d /var/automated_tests/mytests ]
do
  sleep 2
done
 
owd="$(pwd)"
copy_reference_files
unset MAVEN_CONFIG
 
cd "${owd}"
unset owd
 
exec mvn -f /var/automated_tests/mytests -q clean install

Define the Fargate Task

I now have all parts I need apart from a shell script to actually define the Fargate task which will contain all the config for the containers. I have made extensive use of the heredoc convention in the shell script - one per container definition - which makes it easier to read, and also means when I come to make up other tasks (such as static analysis, and persistent playground) I can simple swap out those containers I don't need.

There are a few comments here to note.

You cannot create a Fargate Task without the IAM ecsTaskExecutionRole, but that will not be created for you. It is however created when you manually create your first Fargate task in the AWS console. Of course ecsTaskExecutionRole can be created programmatically too - and will be the subject of a block at another time.
The Fargate task definition will not create your CloudFront log group for you! Therefore I do a check in the script to see if it already exists, and if not then create.
I have defined a mount point to where I copy the test suite. This is because the test suite is in the code repository which is copied into the PHP/Apache container, but is needed by the Maven container also.
The Selenium Chrome browser is using a non-standard port 5556. This is because it would clash with Firefox otherwise which also defaults to port 5555. You must remember to use an environmental variable NODE_PORT to change the port as well as the hostPort and containerPort.
I have defaulted the task's compute to extremely high cpu and memory. They should be reduced using trial and error to avoid incurring high compute fees.

#!/bin/bash
 
# Rewrite branch name so no feature/ hotfix/ forward slash
BRANCH="selenium-testing"
REPO="d8codebase"
 
# Repo account and repo
ECR_ACCOUNT="XXXXXXXXXXX.dkr.ecr.eu-west-2.amazonaws.com"
ECR_REPO="${ECR_ACCOUNT}/${REPO,,}"
ECS_CLUSTER="default"
 
# Get the Task Role ARN for executing the task
TASK_ARN=`aws iam get-role --role-name ecsTaskExecutionRole | jq .Role.Arn -r`
 
 
# Execution ARN appears to be the same as the Task ARN
EXECUTION_ARN="${TASK_ARN}"
 
 
# Determine if the log group already exists: NOTE: "aws ecs register-task-definition" will not create the log group for you!
LOG_EXISTS=`aws logs describe-log-groups | jq ".logGroups[] | select(.logGroupName == \"/ecs/"${BRANCH}"\") | .logGroupName" -r | wc -l`
if [[ "${LOG_EXISTS}" -eq 0 ]]; then
	aws logs create-log-group --log-group-name "/ecs/${BRANCH}"
fi
 
CONTAINER_CODEBASE=$(cat <
)
 
CONTAINER_SELENIUM_HUB=$(cat <
)
 
CONTAINER_SELENIUM_FIREFOX=$(cat <
)
 
CONTAINER_SELENIUM_CHROME=$(cat <
)
 
CONTAINER_MAVEN=$(cat <
)
 
 
CONTAINERS_AUTOMATIC=$(cat <
)
 
 
aws ecs register-task-definition \
  --family "AUTOMATION-${BRANCH}" \
  --volumes "name=automated_tests,host={}" \
  --task-role-arn "$TASK_ARN" \
  --execution-role-arn "${EXECUTION_ARN}" \
  --network-mode "awsvpc" \
  --container-definitions "${CONTAINERS_AUTOMATIC}" \
  --cpu "4096" \
  --memory "30720" \
  --requires-compatibilities "FARGATE"

Running the Fargate Task

Once the script is executed, the task will be defined and ready to run. This can be run by creating a new shell script (or extending the one above) or by using the console. Since this blog is becoming too code focused, I'll document the running of tasks programmatically in another blog later. For now you should be able to see your task in the console and run it. See screenshot 1 above.

Screenshot 2 show that the task has reached RUNNING state. If you are quick you can click on the task identifier and be taken to the more detailed screen shown in image 3 which shows all the containers that are running.

Once one of the Docker images completes its activity, a signal will be sent to all the other containers to stop. In the task described in this blog, the first task to complete is the running of the automated tests. Once the tests complete, then all other containers will stop. See image 4.

Did the Tests Actually Succeed?

A valid question! The output of the Maven container is written to a CloudWatch log stream which can be parsed and echoed back to the screen, or sent to a Slack channel (more on this in a later blog). For now I have created a sequence of commands using pipes which includes the open source awslogs utility.

$ aws logs describe-log-streams --log-group-name "/ecs/selenium-testing" --log-stream-name-prefix "ecs/Maven" | jq '.logStreams|=sort_by(-.creationTime)|.logStreams[0].logeamName' -r | xargs -I{} awslogs get "/ecs/selenium-testing" {} -s1d | cut -d ' ' -f 1,2 --complement
mvn
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running mytests.RunCucumberTest
Dec 10, 2019 3:43:46 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Dec 10, 2019 3:43:47 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Dec 10, 2019 3:43:49 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Dec 10, 2019 3:43:50 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Feature: Are we on the Home Page?
  Scenario: On the home page         # mytests/home_page.feature:4
    When User is on the Home Page    # Stepdefs.user_is_on_Home_Page()
    Then Message Displayed Memcached # Stepdefs.message_displayed_memcached()
    When User is on the Home Page    # Stepdefs.user_is_on_Home_Page()
    Then Message Displayed GD        # Stepdefs.gd()
1 Scenarios (1 passed)
4 Steps (4 passed)
0m6.835s
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.055 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Troubleshooting Selenium Grid

Getting the Selenium stack working can be fiddly, although the CloudWatch logs are a very good way of diagnosing problems. If you want to satisy yourself that Selenium Hub and its two nodes (Firefox and Chrome) are actually there and available, there is a way but you'll have to work fast on the console since the Fargate tasks completes and shuts down once the tests are complete.

When the Fargate task transitions from PENDING to RUNNING, click on the task identifier. Copy the public IP address into your clipboard - you'll need it shortly. Now click on the ENI (Elastic Network Interface) link which will launch a new tab. Then click on the security group. Now you need to create a new rule. By default port 4444 is blocked on the public IP address - and to see Selenium working I need that to be open.

Edit the rules, create a new custom TCP/IP rule, add 4444 as the port and select anywhere for the CIDR, then save. See screenshot 1 above.

You now instantly have an open port 4444. Open a new browser tab, then paste the IP address you copied a few steps ago, then append /grid/console to it and enter. You should see your Grid + Node configuration as per the second screenshot above.

blog terms

devops ECS AWS AWS CLI Development Docker bash

Programmatically Create AWS ECR Repository and Commit Docker Image to ECR

nigel — Sat, 07 Dec 2019 07:56:15 +0000

Programmatically Create AWS ECR Repository and Commit Docker Image to ECR nigel Sat, 07/12/2019 - 07:56

This is a tutorial for using AWS CLI to programmatically create an AWS ECR repository and then commit a Docker image to that repository. This is the first step in a series of tutorials for creating an automated testing framework in AWS Fargate which can be one step of a DevOps pipeline.

As an example I am going to use the Docker php:7.2-apache image and to make it a little more complex, add a few more PHP extensions using a new Dockerfile so that my new image will be Drupal 8 compliant.

Let's get started.

Pull Base Image and Define Dockerfile

The base php:7.2-apache image needs to be pulled from Docker Hub. Issue the following command.

$ docker pull php:7.2-apache
7.2-apache: Pulling from library/php
000eee12ec04: Pull complete 
8ae4f9fcfeea: Pull complete 
60f22fbbd07a: Pull complete 
ccc7a63ad75f: Pull complete 
a2427b8dd6e7: Pull complete 
91cac3b30184: Pull complete 
d6e40015fc10: Pull complete 
240e21c03bb4: Pull complete 
504858e1e4aa: Pull complete 
9a0523b2d73f: Pull complete 
e3acb84829f4: Pull complete 
28a372733f87: Pull complete 
62ee66cdb80a: Pull complete 
c3f368ddc7aa: Pull complete 
Digest: sha256:602bef9acc8d54527ae5b7ee0f16e5cdb6ab81364310f5123b93b95e357f608d
Status: Downloaded newer image for php:7.2-apache
docker.io/library/php:7.2-apache
$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
php                 7.2-apache          fec8a1aaffb8        31 hours ago        410MB
lambci/lambda       go1.x               8d962cde7e27        3 months ago        707MB
sls-docker          latest              8d962cde7e27        3 months ago        707MB

Yes the docker image is there. Now I need to create a directory for the Dockerfile and a sample index.php file which will prove our image is working.

$ mkdir automated && cd automated

By default the Docker php image uses /var/www/html as Apache's docroot. However, I'm going to alter this to something more standard for Drupal web apps - a directory called docroot will be created and will contain a simple index.php containing a call to phpinfo();

$ mkdir docroot && echo " > docroot/index.php

The Dockerfile should now be created and populated. There a number of activities that need to be performed on it.

Reference the base image with a FROM instruction
Install the libraries that support memcached and graphics manipulation
Enable memcached PHP extension
Configure the graphics PHP extension
Install the graphics PHP extension
Copy the files in the current directory and below into the Docker image at the /var/www/html ensuring the permissions are correct for the apache user and group
Set the environment variable AH_SITE_ENVIRONMENT to devops - this is an Acquia Cloud environment variable and only of any significance for those using Acquia as their host - or need a setting in my case to denote that devops is not running on Acquia
Change the default location for the docroot - that means setting an environment variable and using the sed editor to change the configuration in the sites files

Dockerfile

FROM php:7.2-apache
RUN apt-get update && apt-get install -y libmemcached-dev zlib1g-dev \
    libfreetype6-dev \
    libjpeg62-turbo-dev \
    libpng-dev \
    libwebp-dev \
    && pecl install memcached \
    && docker-php-ext-enable memcached \
    && docker-php-ext-configure gd --with-gd --with-webp-dir --with-jpeg-dir \
       --with-png-dir --with-zlib-dir --with-freetype-dir \
    && docker-php-ext-install gd
COPY --chown=www-data:www-data ./ /var/www/html/.
ENV AH_SITE_ENVIRONMENT devops
ENV APACHE_DOCUMENT_ROOT /var/www/html/docroot
RUN sed -ri -e 's!/var/www/html!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's!/var/www/!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf

Build the Image, Run the Container

Once the Dockerfile is complete, it is built with

$ docker build -t d8codebase .

and to confirm it's built correctly:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
d8codebase          latest              c2193f9778a6        33 minutes ago      439MB
<none>              <none>              2a71964685e0        2 hours ago         432MB
php                 7.2-apache          fec8a1aaffb8        35 hours ago        410MB
lambci/lambda       go1.x               8d962cde7e27        3 months ago        707MB
sls-docker          latest              8d962cde7e27        3 months ago        707MB

Great it's there. Now to spin up the container.

$ docker run -d -p 8080:80 d8codebase

Note I am mapping port 8080 to the container's port 80 which probably won't be pertinent to you. Long story I will try to make short. I write my blogs on my Macbook on a local network in the 192.168 range assigned by my ISP's hub. But I develop and test all the practical work in my blogs in a VM in the IP 10. range. Routing between 192.168 and 10. is difficult without port forwarding - so I port forward 127.0.0.1 8080 to my VM's 10.0.2.15 8080, so when I run my Docker Apache container I must remember incoming is on port 8080 and map that against the container's port 80. Hope that makes sense!

Now check it's running:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
9a6dc2fb88ff        d8codebase          "docker-php-entrypoi…"   44 minutes ago      Up 44 minutes       0.0.0.0:8080->80/tcp   agitated_feistel

Yes, and we can see from the screenshot above that it's picking up the memcached and the gd graphics extensions. Success.

Programmatically Create the AWS ECR Repository

This is quite easy and I've put it into a shell script that requires one parameter - the repo name to be created. The work is done by the aws ecr create-repository call. The rest of the code is simply checking for error conditions. The screenshot above was taken on the AWS console after the shell script below was successfully run with a parameter of d8codebase.

#!/bin/bash
 
#Programmatically create an ECR repo
 
# Check whether we have the args
if [[ -z $@ ]]; then
	echo "usage: [repo_name]"
	exit 1
fi
 
# Create the repo
REPO_NAME=`aws ecr create-repository --repository-name "$1" | jq .repository.repositoryName -r`
 
# Check it was created ok. We already get a quality stderr msg from aws cli so just quit on error.
# Similarly, if already exists it will tell us
if [[ "${REPO_NAME}" != $1 ]]; then
	exit 2
fi
 
echo "Successfully created ${REPO_NAME}"

Now the script to automatically login to Docker and push the image to ECR. Note that AWS ECR does require you to use the docker login command, and therefore provides a AWS CLI ECR command get-login which returns a token than can be piped into the docker login command. The token retrieved from ECR has a large payload, and it is top and tailed with other metadata beyond just the password. Therefore that needs to be removed, and if you look closely at the script below, you can see that I pipe the entire returned payload through sed with a regex, and then the result is piped onwards to docker login. Hopefully the rest is quite self explanatory - and you can see the screenshot above that the push has worked and the image can now be included in ECS / Fargate task definitions.

#!/bin/bash
 
#Programmatically push to an ECR repo
 
# Check whether we have the args
if [[ -z $@ ]]; then
	echo "usage: [repo_name]"
	exit 1
fi
 
REPO_NAME=$1
ECR_ACCOUNT="XXXXXXXXXX.dkr.ecr.eu-west-2.amazonaws.com"
ECR_REPO="${ECR_ACCOUNT}/$1"
 
# AWS Authentication - since we need to use docker push
# The sed regex will top and tail the additional meta data sent by the aws login request.
aws ecr get-login --region eu-west-2  | sed -e 's/^.*-p \(.*\)\s\-\e.*$/\1/' |  docker login --password-stdin -u AWS ${ECR_ACCOUNT}
 
# Get the image id of the Docker build. If there is a "latest" use that else get the first without latest
TAG=`docker images | grep -w "${REPO_NAME}" | grep latest | awk '{ print $3; }'`
if [[ -z ${TAG} ]]; then
        TAG=`docker images | grep -w "${REPO_NAME}" | awk '{ print $3; }'`
fi
 
docker tag "$TAG" "${ECR_REPO}"
docker push "${ECR_REPO}"

blog terms

AWS AWS CLI Docker devops Fargate ECS Development bash

Create Arbitrary Subdomains for AWS Fargate Tasks using AWS CLI

nigel — Fri, 01 Nov 2019 15:03:02 +0000

Create Arbitrary Subdomains for AWS Fargate Tasks using AWS CLI nigel Fri, 01/11/2019 - 15:03

AWS Fargate is a relatively new product offering the capability of running Docker bundles as an ECS compute service on EC2 without the necessity for the user to have to orchestrate the underlying EC2 infrastructure. It is perfect for DevOps, and was the technology I chose as DevOps Architect at my current client. The client wanted standard CI/CD capabilities such as spinning up containerised copies of their website environment per feature branch to perform static analysis, automated testing, and also significantly, a persisting playground that QA or development personnel could use for manual testing and troubleshooting a feature (or hotfix) branch they are currently working on.

This is similar to the offering of both Platform.sh and Amazee Labs hosting solutions - developers can spin up their feature branches. Alas my client is locked into another host yet needed on-the-fly feature branch Docker bundles.

AWS Fargate tasks can achieve this, but out of the box it isn't possible to create public domain registration via ECS Service Discovery. The Fargate task is issued a bare IPv4 address which will obviously work but our requirement is to have a subdomain created per branch that would be of the format {hotfix|feature}--{jira ticket}-{suitable branch name}.{top level domain}. So an example would be:

feature--doi-746-my-whizzy-new-feature.clientdomain.dev

So whilst we can't create a subdomain like that natively, we can do it by retrieving the IP address from the ECS Fargate task, then add that IP to a public Route 53 Hosted Zone. Furthermore, since the Fargate task is run via a Jenkins pipeline job that uses shell scripts populated with AWS CLI, the good news is we can indeed complete the registration of subdomain also using the AWS CLI *and* have the DNS propagate within 60 seconds! Amazing! Let's dive into how we achieve this.

Requirements + Assumptions

The solution will be built using the standard httpd Apache2 server sample Fargate task described on the AWS blog Tutorial: Creating a Cluster with a Fargate Task Using the AWS CLI. I added the task definition into the console, although it is also possible create a shell script and use AWS CLI. I have used the default cluster for brevity's sake, and the AWS account default VPC. I have also transferred a spare domain I had, saasidate.com, into Route 53 before I started the blog. This will be my tld and all the subdomains will be added off it. It's important that the ownership of the domain is transferred into Route 53 since otherwise subdomains will have authority issues if the parent domain is still registered to other domain providers such as GoDaddy or 123-Reg.

The solution I am putting together will use AWS CLI which by default returns JSON structured data. It is absolutely essential to use the Linux command line utility jq for parsing the JSON. Documentation on jq is readily available on the net.

Steps

The solution will need the following steps:

Get the user's default VPC (or create a custom VPC if preferred. We'll need this at a few points in the code.
Get a subnet from the VPC. Three subnets are created for use with the default VPC. I pick the first which is arbitrary.
Get a security group. A new security group is created per task when using the console but that is unnecessary for the shell scripts we will be building. So in our script, check whether our 'standard' security group has been created. If so use it, if not then create a security group with an ingress port of 80 and a CIDR of 0.0.0.0/0 which allows access worldwide on TCP port 80.
Run the Fargate task and loop / sleep until the task has a status of RUNNING.
Get the network interface id of the task and interrogate it for the IP address.
Get the hosted zone for the parent domain.
Add a record set containing the feature branch subdomain to the parent domain's hosted zone.
Create a new hosted zone using the feature branch.

Ok - let's crack on and put this into AWS CLI shell scripts.

Get the VPC

I'm using the default VPC so I can filter the list I get back from AWS using the IsDefault attribute.

VPC=`aws ec2 describe-vpcs | jq '.Vpcs[] | select(.IsDefault == true) | .VpcId' -r`

Note the use of the jq select statement. This is a familiar construct and will be used throughout this blog. Also note the -r flag - this removes the double quotes from the response.

Get the First Subnet in the Default VPC

Here I am using jq again but piping the results through head to get the first subnet. There wasn't an obvious jq query to do this natively within jq.

# Get the first subnet of the VPC
SUBNET=`aws ec2 describe-subnets | jq --arg VPC "$VPC" '.Subnets[] | select(.VpcId == $VPC) | .SubnetId' -r | head -n 1`

Get the Security Group

Here I check whether we already have a dedicated security group which gives us worldwide access to port 80. I search first for a unique security group description and if it doesn't exist, then create it.

# Check if our security group for port 80 webserver already exists. Search for our funky unique description
SECURITY=`aws ec2 describe-security-groups | jq ' .SecurityGroups[] | select (.Description == "Ingress Port 80 Anywhere") | .GroupId' -r`
 
# Let's inspect what we got back.
if [[ $? -ne 0 ]]; then
	echo "describe-security-groups failed with code $?"
	exit $?
fi
 
# If we didn't get a security group then create
if [[ -z "$SECURITY" ]]; then
	SECURITY=`aws ec2 create-security-group \
		--description "Ingress Port 80 Anywhere" \
		--group-name "Fargate Webserver Port 80" \
		--vpc-id "$VPC" | jq .GroupId -r `
 
 
	aws ec2 authorize-security-group-ingress \
		--group-id "$SECURITY" \
		--ip-permissions "FromPort=80,ToPort=80,IpProtocol=TCP,IpRanges=[{CidrIp=0.0.0.0/0}]"
fi

Run the Fargate Task

Next I run the Fargate task. This is the sample Fargate task covered in the AWS documentation and uses the httpd docker image. When I created it in the console I called it "first-run-task-definition" - for the life of me I can't remember why I chose such a crazy name! Note I am piping the result through jq as per normal and then onwards through sed to get the task id without all the arn information that precedes it. The sed command looks for the final forward slash in the task arn and crops anything before it.

# Run the task and parse the task ARN for polling later to determine its status
TASK=`aws ecs run-task \
	--cluster "default" \
	--task-definition "first-run-task-definition" \
	--network-configuration "awsvpcConfiguration={subnets=[${SUBNET}],securityGroups=[${SECURITY}],assignPublicIp=ENABLED}" \
	--launch-type FARGATE  | jq .tasks[0].taskArn -r | sed "s/.*\///" `
 
# Let's inspect what we got back. There should be a task identifier.
if [[ $? -ne 0 ]]; then
	echo "task-run failed with code $?"
	exit $?
fi
 
if [[ -z "$TASK" ]]; then
	echo "task-run could not create task"
	exit 1
fi

Loop and Wait for Fargate Task to get to RUNNING Status

Next I have to wait for the task to acquire RUNNING status, However I can't wait for ever, and thus I have wrapped a shell script rule into a timeout mechanism. I have allowed for 5 minutes for the task to get to RUNNING status and that should be more than enough.

# Loop around until we see a RUNNING status or timeout after 10 x 30 second sleeps
for i in 1 2 3 4 5 6 7 8 9 10 11
do
	if [[ $i -eq 11 ]]; then
		echo "Timed out before could establish task is running"
		exit 1
	fi
 
	STATUS=`aws ecs describe-tasks --cluster="default" --tasks $TASK  | jq .tasks[0].lastStatus -r`
 
	# Did it exit unexpectedly?
	if [[ $? -ne 0 ]]; then
		echo "describe-tasks failed with code $?"
		exit $?
	fi
 
	if [[ "$STATUS" = "RUNNING"  ]]; then
		break
	fi
 
	sleep 30
done

Obtain the Network Interface Id

The ENI is available in the Fargate task's description so I needed to parse this on my quest to get the public IP.

# Get the network Interface id
ENI=`aws ecs describe-tasks --cluster="default" --tasks $TASK  | jq '.tasks[0].attachments[0].details[] | select(.name == "networkInterfaceId") | .value' -r `
 
# Let's inspect what we got back. There should be a network interface id.
if [[ $? -ne 0 ]]; then
	echo "describe-tasks failed with code $?"
	exit $?
fi
 
if [[ -z "$ENI" ]]; then
	echo "describe-tasks could not establish eni"
	exit 1
fi

Get the Public IP Address

With the ENI it is possible to parse the the descriptions of all the network interfaces for the public IP address.

PUBLIC_IP=`aws ec2 describe-network-interfaces | jq --arg ENI "$ENI" '.NetworkInterfaces[] | select(.NetworkInterfaceId == $ENI) | .PrivateIpAddresses[0].Association.PublicIp' -r`
 
# Did we get an IP address?
if [[ $? -ne 0 ]]; then
	echo "describe-network-interfaces failed with code $?"
	exit $?
fi
 
if [[ -z "$PUBLIC_IP" ]]; then
	echo "describe-network-interfaces could not retrieve public IP address"
	exit 1
fi

Get the Parent Domain's Hosted Zone Id

The shell script needs to know about the parent domain and the subdomain name (which in my case is a feature branch name) - so those need to be retrieved from runtime arguments. This will go at the top of the shell script.

if [[ -z $1 ]] || [[ -z $2 ]]; then
	echo "usage: parent_domain_name feature_branch_name"
	exit 1
fi
 
BRANCH=$2.$1
PARENT=$1.

Then continuing at the end of the codebase, the following gets the parent domain

PARENT_ZONE=`aws route53 list-hosted-zones-by-name | jq --arg PARENT "$PARENT" '.HostedZones[] | select(.Name == $PARENT) | .Id' -r | sed "s/.*\///"`

Add the Feature Branch Name and Public IP Address to the Parent Hosted Zone

Now I created a new record set for the parent domain's hosted zone. This uses the AWS CLI command change-resource-record-sets which allows for insert, upsert and delete. I needed a DNS A record adding. The easiest way of providing the runtime parameters is via a JSON structure and that is created in shell script heredoc format to simplify shell script escaping.

BATCH=$(cat <
)
 
 
# Use the hosted zone of the tld
aws route53 change-resource-record-sets \
	--hosted-zone-id $PARENT_ZONE \
	--change-batch "$BATCH"

Create the Feature Branch Hosted Zone

The final AWS CLI call is to use create-hosted-zone, and here I needed to provide the feature branch name and an arbitrary unique caller reference for which I used a timestamp.

timestamp=$(date +%s)
 
# Now create a new hosted zone of the feature branch
aws route53 create-hosted-zone \
	--name "$BRANCH" \
	--caller-reference "$timestamp"

The Complete Script

The complete script is listed below. Note that for brevity and easy reading I have cut a few coding standards. There is repetition in the error condition checking. This should be rewritten with calls to shell script functions in a separate file.

#!/bin/bash
 
# Script to assign a public IP address issued by a Fargate task to a subdomain
 
if [[ -z $1 ]] || [[ -z $2 ]]; then
	echo "usage: parent_domain_name feature_branch_name"
	exit 1
fi
 
BRANCH=$2.$1
PARENT=$1.
 
 
# Get the default VPC.
VPC=`aws ec2 describe-vpcs | jq '.Vpcs[] | select(.IsDefault == true) | .VpcId' -r`
 
# Get the first subnet of the VPC
SUBNET=`aws ec2 describe-subnets | jq --arg VPC "$VPC" '.Subnets[] | select(.VpcId == $VPC) | .SubnetId' -r | head -n 1`
 
# Check if our security group for port 80 webserver already exists. Search for our funky unique description
SECURITY=`aws ec2 describe-security-groups | jq ' .SecurityGroups[] | select (.Description == "Ingress Port 80 Anywhere") | .GroupId' -r`
 
# Let's inspect what we got back.
if [[ $? -ne 0 ]]; then
	echo "describe-security-groups failed with code $?"
	exit $?
fi
 
# If we didn't get a security group then create
if [[ "$SECURITY" = "" ]]; then
	SECURITY=`aws ec2 create-security-group \
		--description "Ingress Port 80 Anywhere" \
		--group-name "Fargate Webserver Port 80" \
		--vpc-id "$VPC" | jq .GroupId -r `
 
 
	aws ec2 authorize-security-group-ingress \
		--group-id "$SECURITY" \
		--ip-permissions "FromPort=80,ToPort=80,IpProtocol=TCP,IpRanges=[{CidrIp=0.0.0.0/0}]"
fi
 
# Run the task and parse the task ARN for polling later to determine its status
TASK=`aws ecs run-task \
	--cluster "default" \
	--task-definition "first-run-task-definition" \
	--network-configuration "awsvpcConfiguration={subnets=[${SUBNET}],securityGroups=[${SECURITY}],assignPublicIp=ENABLED}" \
	--launch-type FARGATE  | jq .tasks[0].taskArn -r | sed "s/.*\///" `
 
# Let's inspect what we got back. There should be a task identifier.
if [[ $? -ne 0 ]]; then
	echo "task-run failed with code $?"
	exit $?
fi
 
if [[ -z "$TASK" ]]; then
	echo "task-run could not create task"
	exit 1
fi
 
# Loop around until we see a RUNNING status or timeout after 10 x 30 second sleeps
for i in 1 2 3 4 5 6 7 8 9 10 11
do
	if [[ $i -eq 11 ]]; then
		echo "Timed out before could establish task is running"
		exit 1
	fi
 
	STATUS=`aws ecs describe-tasks --cluster="default" --tasks $TASK  | jq .tasks[0].lastStatus -r`
 
	# Did it exit unexpectedly?
	if [[ $? -ne 0 ]]; then
		echo "describe-tasks failed with code $?"
		exit $?
	fi
 
	if [[ "$STATUS" = "RUNNING"  ]]; then
		break
	fi
 
	sleep 30
done
 
 
# Get the network Interface id
ENI=`aws ecs describe-tasks --cluster="default" --tasks $TASK  | jq '.tasks[0].attachments[0].details[] | select(.name == "networkInterfaceId") | .value' -r `
 
# Let's inspect what we got back. There should be a network interface id.
if [[ $? -ne 0 ]]; then
	echo "describe-tasks failed with code $?"
	exit $?
fi
 
if [[ -z "$ENI" ]]; then
	echo "describe-tasks could not establish eni"
	exit 1
fi
 
 
# Get the Public IP Address
PUBLIC_IP=`aws ec2 describe-network-interfaces | jq --arg ENI "$ENI" '.NetworkInterfaces[] | select(.NetworkInterfaceId == $ENI) | .PrivateIpAddresses[0].Association.PublicIp' -r`
 
# Did we get an IP address?
if [[ $? -ne 0 ]]; then
	echo "describe-network-interfaces failed with code $?"
	exit $?
fi
 
if [[ -z "$PUBLIC_IP" ]]; then
	echo "describe-network-interfaces could not retrieve public IP address"
	exit 1
fi
 
BATCH=$(cat <
)
 
PARENT_ZONE=`aws route53 list-hosted-zones-by-name | jq --arg PARENT "$PARENT" '.HostedZones[] | select(.Name == $PARENT) | .Id' -r | sed "s/.*\///"`
 
aws route53 change-resource-record-sets \
	--hosted-zone-id $PARENT_ZONE \
	--change-batch "$BATCH"
 
 
timestamp=$(date +%s)
 
# Now create a new hosted zone of the feature branch
aws route53 create-hosted-zone \
	--name "$BRANCH" \
	--caller-reference "$timestamp"

Invoking the Script

The script requires two parameters as previous discussed. Here's my example for the blog:

$ ./feature-branch.sh saasidate.com feature--doi-746-my-whizzy-new-feature

Currently the output of the script is the response from the AWS CLI route53 create-hosted-zone call, and here you can detect the full domain name.

Checking the Outcome

Obviously the first check is to copy and paste the url of the feature branch into a browser and see if it loads. An example of this is the blog heading image at the top of the page.

Also the AWS console is the place to check everything is as required. Navigate to ECS -> default -> Tasks and you should see a screenshot similar to the first one immediately above. This is the task we invoked from inside our shell script. Now navigate to Route 53 -> Hosted Zones -> {domain name} and you can see the feature branch and its IP address.

Steps left to do

New hosted zones will normally propagate in 60 seconds in Route 53 - but it would be good if the script above polls Route 53 every 10 seconds or so and reports back the subdomain url and the IP address once the propagation has completed. This should be surfaced on the command line as a minimum, but perhaps a notification on Slack would be even better. Both options are trivial.

Whilst the idea is that the Fargate tasks should persist, they shouldn't be around forever. Therefore there needs to be clear up activities to remove the feature subdomain from the parent domain's hosted zone, and the subdomain hosted zone. Also there obviously needs a step to stop the Fargate task.

Caveats

There are caveats to this solution - but for my use case, it's one of those rare occurrences in life that the caveats don't apply to me.

Firstly, this solution will only work with a Fargate task and not an ECS service running a Fargate task. Tasks should be run as services for production environments since services give you great things like replication, and health percentages against number of running tasks. There is therefore no scaling, no load balancing, no DDOS protection in what I'm offering here.

Fargate tasks run in isolation are perfect for spun up short living environments such as playgrounds for devs and QAs to run manual tests against or troubleshoot development issues - which is exactly my use case.

blog terms

AWS AWS CLI Docker devops Fargate ECS Development Route 53 bash

Drupal 8 and DRUD Tech's ddev Local Development Environment

nigel — Sat, 10 Mar 2018 21:12:50 +0000

Drupal 8 and DRUD Tech's ddev Local Development Environment nigel Sat, 10/03/2018 - 21:12

My favourite presentation at DrupalCamp London 2018 was Turbocharge Your Agency given by DRUD Tech co-founder Kevin Bridges and Jeffrey A. "jam" McGuire. They talked at length about the container solution ddev which enables developers to manage their local environments and thus optimise their time and in addition save their configuration under source control. Furthermore the demonstration showed that it was possible to flip between different configurations with different Drupal versions easily and quickly. The solution found here uses Docker and is wrapped by the lightning fast Golang.

I've been thinking for a while about moving from my existing hand crafted Virtual Machine towards Docker. My VM's configuration isn't under source control management, and has built up over the years into quite a complex beast. I have Jenkins, phing, composer, Drupal console, Mail Catcher, Compass, node, npm, Python, nginx, Apache, MariaDB and MongoDB that I can remember off the top of my head. It's becoming a difficult proposition to keep it all working. When I need to run a different version of PHP I manually switch simlinks around. Oh for something easier.

So I elected to investigate whether Drud Tech's offering could possibly help out. The demonstration given by their guys at DrupalCamp London 2018 showed the ease of moving between D7 and D8 repos. However they were fresh builds and for the tool to be of value to me, it would have to work from pre-existing codebases.

Installing a VirtualBox VM

This was purely a Proof of Concept so as a consequence I needed to keep anything I did in a ring-fenced environment. Therefore a VirtualBox Ubuntu 16.04LTS image was used as a starting point. I won't provide screenshots of the entire build process since I have previous blogs to cover this.

Once the build has completed and I have logged in I issue a ifconfig command to determine the IP address which is 10.0.2.15 (second screenshot). This will need to be used with the network configuration.

When I was building the VM I kept the default network configuration which is NAT. I have moved away from using Bridged Adapter. A bridged adapter on the face of it is very convenient - it automatically gets an IP address via DHCP from my ISP provided router at home, in the 192.168.0.0 range. However this falls foul at client sites with security policies prohibiting unknown MAC Addresses, and Internet cafes that don't use the same IP range - it simply won't work.

The solution is to use NAT and use the port forwarding facility. Navigate to Settings -> Network -> Advanced -> Port Forwarding and add values for ssh as per the third screenshot above.

I can now ssh into the ddev VM.

$ ssh -p 2222 nigel@127.0.0.1
The authenticity of host '[127.0.0.1]:2222 ([127.0.0.1]:2222)' can't be established.
ECDSA key fingerprint is SHA256:VRNxWpagSl2DGz7dmgRjBqdDdadXITdPPkItrMR9k38.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
nigel@127.0.0.1's password: 
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-87-generic x86_64)
 
 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
 
137 packages can be updated.
69 updates are security updates.
 
 
Last login: Mon Mar  5 21:50:35 2018
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

Install Docker

Don't be tempted to install the Docker version that is bundled in the Ubuntu default repos. Follow the instructions on the Docker site for installing the CE version of Docker from their private repo.

nigel@ddev:~$ sudo apt-get update
[sudo] password for nigel: 
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease
Reading package lists... Done

Install packages to allow apt to use a repository over HTTPS

nigel@ddev:~$ sudo apt-get install \
>     apt-transport-https \
>     ca-certificates \
>     curl \
>     software-properties-common
Reading package lists... Done
Building dependency tree       
Reading state information... Done
apt-transport-https is already the newest version (1.2.25).
ca-certificates is already the newest version (20170717~16.04.1).
curl is already the newest version (7.47.0-1ubuntu2.6).
software-properties-common is already the newest version (0.96.20.7).
0 to upgrade, 0 to newly install, 0 to remove and 2 not to upgrade.

That's good. Nothing to do there. Now let's add Docker’s official GPG key:

nigel@ddev:~$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK

Now I need to verify that we have the key with the fingerprint 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88, by searching for the last 8 characters of the fingerprint.

nigel@ddev:~$ sudo apt-key fingerprint 0EBFCD88
pub   4096R/0EBFCD88 2017-02-22
      Key fingerprint = 9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid                  Docker Release (CE deb) <docker@docker.com>
sub   4096R/F273FCD8 2017-02-22

All good. Next the following command to set up the stable repository.

nigel@ddev:~$ sudo add-apt-repository \
>    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
>    $(lsb_release -cs) \
>    stable"

Update apt again

nigel@ddev:~$ sudo apt-get update
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 https://download.docker.com/linux/ubuntu xenial InRelease [65.8 kB]
Get:6 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages [3,329 B]
Fetched 69.1 kB in 0s (140 kB/s)
Reading package lists... Done

Again we are looking good - we can see it's picked up the Docker repo ok. Let's install the latest version of Docker.

nigel@ddev:~$ sudo apt-get update

Let's check it's working..

nigel@ddev:~$ sudo docker run hello-world
[sudo] password for nigel: 
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
ca4f61b1923c: Pull complete 
Digest: sha256:083de497cff944f969d8499ab94f07134c50bcf5e6b9559b27182d3fa80ce3f7
Status: Downloaded newer image for hello-world:latest
 
Hello from Docker!
This message shows that your installation appears to be working correctly.

Success! Now we need docker-compose installing. Again, don't be tempted to install from the standard Ubuntu repos, because if you do with Ubuntu 16.04LTS you'll end up with docker-compose version 1.8.0 which does not meet the ddev requirements. So do it the proper way!

nigel@ddev:~$ sudo curl -L https://github.com/docker/compose/releases/download/1.19.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   617    0   617    0     0    795      0 --:--:-- --:--:-- --:--:--   795
100 8288k  100 8288k    0     0   355k      0  0:00:23  0:00:23 --:--:--  196k

Now make it executable

nigel@ddev:~$ sudo chmod +x /usr/local/bin/docker-compose

Let's see the version

nigel@ddev:~$ docker-compose --version
docker-compose version 1.19.0, build 9e633ef

Neat! We are almost there. Now we need to make sure Docker starts every time we boot the machine.

nigel@ddev:~/projects/meedjum$ sudo systemctl enable docker
Synchronizing state of docker.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable docker

Before I am done here, I need to add myself to the Docker group so I can execute Docker commands (or rather ddev can on my behalf)

nigel@ddev:~/projects/meedjum$ sudo usermod -aG docker $(whoami)

Rebooting will now serve two purposes - check that the above command works which will start Docker, and make sure my new group status has been picked up.

nigel@ddev:~/projects/meedjum$ sudo su
root@ddev:/home/nigel/projects/meedjum# init 6

Installing ddev

There are two ways of installing ddev - by downloading from the repository's releases - or use the install script in the ddev documentation. I opted for the latter.

nigel@ddev:~$ curl https://raw.githubusercontent.com/drud/ddev/master/install_ddev.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2545  100  2545    0     0   5821      0 --:--:-- --:--:-- --:--:--  5823
ddev_linux.v0.15.0.tar.gz: OK
Download verified. Ready to place ddev in your /usr/local/bin.
Running "sudo mv /tmp/ddev /usr/local/bin/" Please enter your password if prompted.
[sudo] password for nigel: 
Bash completion for ddev was not installed. You may manually install /tmp/ddev_bash_completion.sh in your bash_completions.d directory.
ddev is now installed. Run "ddev" to verify your installation and see usage.

Installed - now check our installation

nigel@ddev:~$ ddev
This Command Line Interface (CLI) gives you the ability to interact with the ddev to create a development environment.
Usage:
ddev [command]
Available Commands:
auth-pantheon Provide a machine token for the global pantheon auth.
config Create or modify a ddev project configuration in the current directory
describe Get a detailed description of a running ddev project.
exec Execute a shell command in the container for a service. Uses the web service by default.
help Help about any command
hostname Manage your hostfile entries.
import-db Import the database of an existing project to the dev environment.
import-files Import the uploaded files directory of an existing project to the default public upload directory of your project.
list List projects
logs Get the logs from your running services.
pull Import files and database using a configured provider plugin.
remove Remove the development environment for a project.
restart Restart the development environment for a project.
ssh Starts a shell session in the container for a service. Uses web service by default.
start Start a ddev project.
stop Stop the development environment for a project.
version print ddev version and component versions
Flags:
-h, --help help for ddev
-j, --json-output If true, user-oriented output will be in JSON format.
Use "ddev [command] --help" for more information about a command.

We have it!

Getting Started with an Existing Drupal 8 Project

This blog has been built using Drupal 8 - so it made sense to use this for my first project. Let's create a directory for my projects and git clone. At this point I had already created a new .ssh key for my ddev VM and added it to my GitHub profile.

nigel@ddev:~$ mkdir projects
nigel@ddev:~$ cd projects/
nigel@ddev:~/projects$ git clone git@github.com:sanddevil/meedjum.git
Cloning into 'meedjum'...

Now let's cd into the project and config ddev for our Drupal 8 through its interactive shell.

nigel@ddev:~/projects/meedjum$ ddev config
Creating a new ddev project config in the current directory (/home/nigel/projects/meedjum) 
Once completed, your configuration will be written to /home/nigel/projects/meedjum/.ddev/config.yaml
 
Project name (meedjum): 
 
The docroot is the directory from which your site is served. This is a relative path from your project root (/home/nigel/projects/meedjum) 
You may leave this value blank if your site files are in the project root 
Docroot Location (docroot): 
Found a php codebase at /home/nigel/projects/meedjum/docroot. 
Project Type [php, drupal6, drupal7, drupal8, wordpress, typo3, backdrop] (php): drupal8
Configuration complete. You may now run 'ddev start'.

ddev is suggesting I should run start, so why not?

nigel@ddev:~/projects/meedjum$ ddev start
Network ddev_default created 
Starting environment for meedjum... 
ddev needs to add an entry to your hostfile.
It will require root privileges via the sudo command, so you may be required
to enter your password for sudo. ddev is about to issue the command: 
    sudo /usr/local/bin/ddev hostname meedjum.ddev.local 127.0.0.1 
Please enter your password if prompted. 
Running Command  Command=sudo /usr/local/bin/ddev hostname meedjum.ddev.local 127.0.0.1
[sudo] password for nigel: 
Pulling db (drud/mariadb-local:v0.8.0)... 
Creating ddev-meedjum-db ... done
Pulling dba (drud/phpmyadmin:v0.2.0)... 
Creating ddev-meedjum-web ... done
Creating ddev-meedjum-dba ...  
Creating ddev-meedjum-web ...  
 
Network ddev_default is external, skipping 
Pulling ddev-router (drud/ddev-router:v0.5.0)... 
Creating ddev-router ... done
 
Successfully started meedjum 
Your project can be reached at http://meedjum.ddev.local and https://meedjum.ddev.local

Installing Composer

Drupal 8 requires composer and that needs installing on our clean system. There are plenty of tutorials online on how to achieve this, so I'll whizz through this quickly.

nigel@ddev:~$ sudo apt-get update
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:5 https://download.docker.com/linux/ubuntu xenial InRelease
Reading package lists... Done

Install the dependencies

nigel@ddev:~$ sudo apt-get install curl php-cli php-mbstring git unzip
Reading package lists... Done

Download composer

nigel@ddev:~$ curl -sS https://getcomposer.org/installer -o composer-setup.php

Install it globally

nigel@ddev:~$ sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
All settings correct for using Composer
Downloading...
 
Composer (version 1.6.3) successfully installed to: /usr/local/bin/composer
Use it: php /usr/local/bin/composer

Now check it's been installed correctly

nigel@ddev:~$ composer
   ______
  / ____/___  ____ ___  ____  ____  ________  _____
 / /   / __ \/ __ `__ \/ __ \/ __ \/ ___/ _ \/ ___/
/ /___/ /_/ / / / / / / /_/ / /_/ (__  )  __/ /
\____/\____/_/ /_/ /_/ .___/\____/____/\___/_/
                    /_/
Composer version 1.6.3 2018-01-31 16:28:17

Building Out the Drupal 8 Project

The ddev start command creates a new subdirectory called .ddev in the project's root directory. Within the directory we have config.yaml and docker-compose.yaml. The config file contains the configuration of our ddev project such as the version of PHP, webpage url, type of project (drupal8) amongst other items. The docker yaml file contains the container configuration.

Interaction with the ddev project can be accomplished in two ways, either with

ddev exec command

or by spawning a shell with

ddev ssh

and using exit to terminate the interactive session.

Our first requirement is to get dependencies into our D8 project since it is vendor-ised and uses composer as its dependency management.

Ok my Drupal core was on 8.4.4 and 8.4.5 and then 8.5 had been released so it needed updating, and in my case would update to 8.4.5 initially

nigel@ddev:~/projects/meedjum/docroot$ composer update --with-dependencies
Gathering patches for root package.
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 51 installs, 0 updates, 0 removals
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.

Now since there were schema changes, those need to be applied.

nigel@ddev:~/projects/meedjum/docroot$ ddev exec drush updatedb
Performing node_update_8401                                          [ok] 
Cache rebuild complete.                                              [ok] 
Finished performing updates.                                         [ok] 
The following updates are pending:
 
node module : 
  8401 -   Run a node access rebuild, if required. 
 
Do you wish to run all pending updates? (y/n): y

Settings.php

This is where things started to go awry. I already have a fully constructed settings.php and settings.local.php. The settings.php file contains my Memcached settings. However the memcache(d) PHP libraries aren't shipped in the ddev product. Obviously by that fact ddev doesn't support Memcached.

When I attempted a drush status command I was inundated with errors.

nigel@ddev:~/projects/meedjum/docroot/sites/default$ vi settings.local.php
nigel@ddev:~/projects/meedjum/docroot/sites/default$ ddev exec drush status
Drupal\memcache\MemcacheException: No Memcache extension found in /var/www/html/docroot/modules/contrib/memcache/src/DrupalMemcacheFactory.php on line 162 #0 /var/www/html/docroot/modules/contrib/memcache/src/DrupalMemcacheFactory.php(62): Drupal\memcache\DrupalMemcacheFactory->initialize()

As a consequence I commented out the Memcache settings

// Memcache
//Memcache configuration
//$settings['memcache']['servers'] = ['127.0.0.1:11211' => 'default'];
//$settings['memcache']['bins'] = ['default' => 'default'];
//$settings['memcache']['key_prefix'] = 'badzillad8';
//$settings['cache']['default'] = 'cache.backend.memcache';
?>

Importing the Database

The next problem is ddev looks for the db credentials in settings.php whereas I have my credentials in my settings.local.php file. I changed the credentials in the settings.local.php file and I copied my database file to the /tmp directory.

nigel@ddev:~/projects/meedjum/docroot/sites/default$ ddev import-db  --extract-path /tmp
Provide the path to the database you wish to import. 
Import path: /tmp/db-2018-03-10.sql.gz
Failed to import database for meedjum: failed to write settings file for meedjum: Failed to get Drupal settings file path: settings files already exist and are being managed by the user

More errors! The import failed. I thought it could be because it required the credentials in settings.php but that didn't work either. So I decided to use Drush instead:

nigel@ddev:~/projects/meedjum/docroot/sites/default$ ddev exec drush sql-cli < /tmp/db-2018-03-10.sql

HTTP Port Forwarding

Just like our earlier forwarding of the ssh port so we can log into the VM, we also have to forward http port 80 traffic. To do this, we go back to the virtual machine network configuration, and set up a redirect from port 9090 (somewhat arbitrary) on the host to port 80 on the guest. See the first screenshot. This will take us to the the default website on the VM. We need to explicitly tell a browser which site it is - to do that we add an entry on the machine where we will be using our browser - which in my case will be my MacBook.

So I have added the entry to /etc/hosts of the working ddev domain name and the localhost IP address

127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost
 
127.0.0.1 meedjum.ddev.local

Now we need to reboot our VM and then login back in and issue a ddev start on our project. We can also check whether everything is good with a docker ps command.

root@ddev:/home/nigel# init 6
Connection to 127.0.0.1 closed by remote host.
Connection to 127.0.0.1 closed.
Nigels-MacBook-Pro:tmp nigel$ ssh -p 2222 nigel@127.0.0.1
nigel@ddev:~$ cd projects/meedjum/
nigel@ddev:~/projects/meedjum$ ddev start
Starting environment for meedjum... 
nigel@ddev:~/projects/meedjum$ docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS                             PORTS                                                                                      NAMES
5fd9a60cb90c        drud/ddev-router:v0.5.0           "/app/docker-entrypo…"   10 seconds ago      Up 10 seconds (healthy)            0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8025->8025/tcp, 0.0.0.0:8036->8036/tcp   ddev-router
cea77b7cbc6b        drud/nginx-php-fpm-local:v1.1.0   "/start.sh"              7 hours ago         Up 11 seconds (healthy)            443/tcp, 0.0.0.0:32771->80/tcp, 0.0.0.0:32770->8025/tcp                                    ddev-meedjum-web
2dc08e116820        drud/phpmyadmin:v0.2.0            "/run.sh phpmyadmin"     7 hours ago         Up 11 seconds (health: starting)   0.0.0.0:32769->80/tcp                                                                      ddev-meedjum-dba
f239a52ae806        drud/mariadb-local:v0.8.0         "/docker-entrypoint.…"   7 hours ago         Up 12 seconds (healthy)            0.0.0.0:32768->3306/tcp                                                                    ddev-meedjum-db

We can now point a web browser at the url we typed into our /etc/hosts file - see third image above. Ok - so we are close but not quite there yet! Looks like this could be the old trusted host pattern issue since it doesn't know about my domain meedjum.ddev.test.

Add ddev Domain to Trusted Hosts in Settings.php

...or rather in my case my settings.local.php file. I added the ddev domain name to the trusted hosts pattern array

$settings['trusted_host_patterns'] = [
  'meedjum.test',
  'meedjum.ddev.local',
];
?>

Copying the Assets

Adding the trusted host entry now means the web site loads successfully but of course the assets are missing and need copying down from my prod site. ddev provides a command for this:

nigel@ddev:~/projects/meedjum/docroot$ ddev import-files --src=/tmp/files.tar.gz
Successfully imported files for meedjum

100% my fault, but my file structure ended up as sites/default/files/files because I'd tarballed my prod files directory within the tarball, instead of just its contents. So I moved everything back one directory.

nigel@ddev:~/projects/meedjum/docroot$ mv sites/default/files/files/* sites/default/files/.
nigel@ddev:~/projects/meedjum/docroot$ mv sites/default/files/files/.htaccess sites/default/files/.
nigel@ddev:~/projects/meedjum/docroot$ rmdir sites/default/files/files

Completed Project

Ok so in the interim I fixed the missing Font Awesome library - not something I'd noticed before, but obviously it's been gitignored and it isn't in the composer.json file. Anyhoo the site is now fully functional and you can see from the URL bar I need to append the port :9090 as per my previous port forwarding fix.

Verdict

Playing with ddev was an interesting experience, not without challenges, but ultimately rewarding. Here are some of the points I made as I went along.

The ddev exec command is run as root. This is a problem because it basically means using composer with it - such as ddev exec composer update - would trash the file permissions of my codebase. Ownerships would become root, and that would lead to problems down the line. Composer should never be run as root. So the net effect is composer needs to be installed on the host system. Ideally I wanted a totally un-poluted filesystem without installing anything on the host beyond ddev. It also means since composer has a dependency on PHP, that needs to be installed, and now we have a system with different PHP CLI and FPM versions.
I couldn't get the ddev import-db command to work. A minor inconvenience and of course it is easy to use alternatives with the command line or drush.
Now I've got a working system - what next? I have a welter of phing targets to deploy my code, usually invoked by Jenkins. I could add phing at project level but how would I get Jenkins docker-ised up and running across all projects? I sense that could be tricky.
I can see ddev's value. I would place it in a small agency with both legacy D6 and D7 support projects alongside new D8 and WP builds. I can't see it as an enterprise solution - there is too much missing such as Memcache and the previously mentioned phing and Jenkins.

But all in all I really enjoyed my Saturday watching the football whilst playing with ddev and writing this blog - despite my teams losing!

blog terms

Drupal Drupal 8 Drupal 7 Drupal 6 devops Docker VirtualBox

Recompiling PHP for serverless-php and Lambda

nigel — Sun, 05 Nov 2017 15:54:48 +0000

Recompiling PHP for serverless-php and Lambda nigel Sun, 05/11/2017 - 15:54

The serverless-php git repo contains a pre-compiled PHP executable which is uploaded to AWS Lambda as part of the FaaS deploy. However, and deliberately to keep it lightweight, many PHP extensions are not included in this PHP image. I thought I may get through my development without having to rebuild this image, but alas not! I decided to use the Twig template engine and that has a dependency on the hash() function. I realised something was afoot when my execution failed.

Cloudwatch error diagnostic

The error was discovered by inspecting my Lambda function run log - you may be able to see able there is a uncaught exception leading to a fatal error.

Fatal error: Uncaught Error: Call to undefined function hash() in /var/task/vendor/twig/twig/lib/Twig/Environment.php:272"

I wasn't aware that hash() required a separate PHP extension, so I checked this on my local copy of PHP 7.1 on my Macbook with

$ php -m | grep hash
hash
$

Aha! So nothing else for it, we need to add this to the compiled PHP image.

Recompilation of PHP image

Thankfully the serverless-php repo contains two files to aid the recompilation of the PHP image - buildphp.sh which is a shell script to run the process, and dockerfile.buildphp which is the configuration for a docker container in which the actual recompilation will occur.

The dockerfile contains the list of PHP extensions to include - so I added hash to the list

RUN ./configure \
    --enable-static=yes \
    --enable-shared=no \
    --disable-all \
    --enable-json \
    --enable-libxml \
    --enable-mbstring \
    --enable-phar \
    --enable-soap \
    --enable-xml \
    --enable-hash \
    --with-curl \
    --with-gd \
    --with-zlib \
    --with-openssl \
    --without-pear \
    --enable-ctype

Now we need to check the PHP filesize and run the build script - most of the output of this has been cropped since it was copious.

$ ls -las php
49144 -rwxr-xr-x@  1 nigel  staff  25161497  2 Nov 14:03 php
$ chmod +x buildphp.sh 
$ ./buildphp.sh 
Build complete.
Don't forget to run 'make test'.
 
 ---> 7b602edcddb5
Removing intermediate container 6c3cd6936a6d
Successfully built 7b602edcddb5
Successfully tagged php-build:latest
2bdff18d52adad839b6cbdba9ee55e3263d5399326175827398f757cc9c74c44
$

Ok so let's check what we've got:

$ ls -las php
50736 -rwxr-xr-x  1 nigel  staff  25976432  4 Nov 16:34 php

A different size which is encouraging! Now let's deploy and run:

$ sls deploy

Results

If we now run our function from the API Gateway or point a web browser on our assigned URL we get a 200 success! Yey!

blog terms

PHP serverless-php serverless lambda Docker devops