Drupal 8

Drupal 8: Move Deprecated drupal/drupal Composer Template to drupal-composer/drupal-project

nigel — Wed, 11 Dec 2019 15:52:32 +0000

Drupal 8: Move Deprecated drupal/drupal Composer Template to drupal-composer/drupal-project nigel Wed, 11/12/2019 - 15:52

Tutorial for moving an existing Drupal 8 codebase away from the deprecated drupal/drupal composer template using the utility GoComposer. This will move the project to the latest Fully Composer Managed template. This will also align the codebase with Drupal 9 reducing any effort required to ensure full Drupal 9 compatibility on day 1 of Drupal 9 release.

This solution will apply to many Drupal 8 codebases, including my own Badzilla site. When I migrated Badzilla from D6, the drupal/drupal composer template appeared to be the correct choice. Sadly experience has proven otherwise, and all Drupal 8 minor versions have caused serious headaches when updating.

So I will be writing this blog from the perspective of moving composer templates on Badzilla.

Getting Started

I will be doing this in a development environment. I am using my own BadzillaVM development environment. Back up both the database and the codebase artefact on prod anyway in preparation for the deploy later.

My first activity is to change directory in my dev environment and create a new git branch dedicated to this project.

$ git checkout -b feature/composer-template

Now let's see what our starting point by listing my existing composer.json file.

$ cd docroot
$ cat composer.json
{
    "name": "drupal/drupal",
    "description": "Drupal is an open source content management platform powering millions of websites and applications.",
    "type": "project",
    "license": "GPL-2.0+",
    "require": {
        "composer/installers": "^1.0.24",
        "cweagans/composer-patches": "~1.0",
        "wikimedia/composer-merge-plugin": "~1.4",
        "drupal/core": "8.6.2",
        "drupal/bootstrap": "^3.9",
        "drupal/memcache": "^2.0@alpha",
        "drupal/module_filter": "^3.0",
        "drupal/pathauto": "^1.0@RC",
        "drupal/devel": "^1.0@RC",
        "drupal/admin_toolbar": "^1.19",
        "drupal/config_inspector": "^1.0@beta",
        "drupal/google_analytics": "^2.1",
        "drupal/metatag": "^1.2",
        "drupal/addtoany": "^1.8",
        "drupal/migrate_tools": "^4.0@beta",
        "drupal/migrate_plus": "^4.0@beta",
        "drupal/migrate_upgrade": "^3.0",
        "drupal/migrate_manifest": "^1.5",
        "drupal/config_update": "^1.3",
        "drupal/paragraphs": "^1.1",
        "drupal/geshifilter": "^1.1",
        "drupal/xmlsitemap": "^1.0@alpha",
        "drupal/youtube": "^1.0@beta",
        "drupal/fontawesome": "^2.0",
        "drupal/config_filter": "^1.0",
        "drupal/config_split": "^1.2",
        "drupal/schema_metatag": "^1.0-rc4",
        "phpdocumentor/reflection-docblock": "^2.0",
        "drush/drush": "9.*",
        "drupal/libraries": "^3.0@alpha",
        "phing/phing": "^2.16",
        "drupal/tome": "^1.0@alpha",
        "drupal-tome/tome_drush": "dev-master",
        "drupal/search_api": "^1.10",
        "drupal/elasticsearch_connector": "^6.0-alpha1"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "platform": {
            "php": "7.0.32"
        },
        "preferred-install": "dist",
        "autoloader-suffix": "Drupal8"
    },
    "extra": {
        "_readme": [
            "By default Drupal loads the autoloader from ./vendor/autoload.php.",
            "To change the autoloader you can edit ./autoload.php.",
            "This file specifies the packages.drupal.org repository.",
            "You can read more about this composer repository at:",
            "https://www.drupal.org/node/2718229"
        ],
        "merge-plugin": {
            "include": [
                "core/composer.json"
            ],
            "recurse": false,
            "replace": false,
            "merge-extra": false
        },
        "installer-paths": {
            "core": ["type:drupal-core"],
            "modules/contrib/{$name}": ["type:drupal-module"],
            "profiles/contrib/{$name}": ["type:drupal-profile"],
            "themes/contrib/{$name}": ["type:drupal-theme"],
            "drush/contrib/{$name}": ["type:drupal-drush"],
            "modules/custom/{$name}": ["type:drupal-custom-module"],
            "themes/custom/{$name}": ["type:drupal-custom-theme"],
            "libraries/{$name}": ["type:drupal-library"]
        },
        "patches": {
            "drupal/core": {
                "rdf: Fatal error: Call to a member function url() on null": "https://www.drupal.org/files/issues/member-function-url-fix-2565247-4.patch"
            },
            "modules/contrib/module_filter": {
                "Module Filter issues notices after extend list changes": "https://www.drupal.org/files/issues/module_filter-undefined_index_recent_modules_submit-2857431-16.patch"
            }
        }  
    },
    "autoload": {
        "psr-4": {
            "Drupal\\Core\\Composer\\": "core/lib/Drupal/Core/Composer"
        }
    },
    "scripts": {
        "pre-autoload-dump": "Drupal\\Core\\Composer\\Composer::preAutoloadDump",
        "post-autoload-dump": [
          "Drupal\\Core\\Composer\\Composer::ensureHtaccess"
        ],
        "post-package-install": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",
        "post-package-update": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup"
    },
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ]
}

Ok so I'm a little behind on my Drupal updates with core currently on 8.6.2, and I'm using PHP 7.0 on both sandbox and prod which is ok for now but needs to be updated soon. I've also got a couple of patches which may be resolved already as I go through this exercise.

GoComposer

To run the GoComposer code I need to be in the top level directory of the codebase, i.e. the directory that contains my .git directory. Ensure this is the case then create a skeleton composer file for GoComposer

$ cd ..
$ composer require webkings-ca/gocomposer:dev-master
You are running composer with xdebug enabled. This has a major impact on runtime performance. See getcomposer.org/xdebug
./composer.json has been created
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 6 installs, 0 updates, 0 removals
  - Installing webkings-ca/gocomposer (dev-master 310fa76): Cloning 310fa76ed0 from cache
  - Installing symfony/polyfill-ctype (v1.13.1): Downloading (100%)         
  - Installing webmozart/assert (1.6.0): Downloading (100%)         
  - Installing webmozart/path-util (2.3.0): Loading from cache
  - Installing webflo/drupal-finder (1.2.0): Downloading (100%)         
  - Installing symfony/yaml (v3.4.36): Downloading (100%)         
symfony/yaml suggests installing symfony/console (For validating YAML files using the lint command)
Writing lock file
Generating autoload files
PHP 7.0.33-13+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Nov 28 2019 07:43:06) ( NTS )

Now I am ready to run gocomposer

$ composer gocomposer
You are running composer with xdebug enabled. This has a major impact on runtime performance. See getcomposer.org/xdebug
 
GoComposer is Initializing...
=============================
 
 
 Please Confirm: Composer.json file will be Created in /var/www/html/meedjum-composer/docroot (Recommended) (yes/no) [yes]:
 > 
 
 Please Confirm: Your Original Site will be Backed up to the following directory:  /var/www/html/meedjum-composer/docroot/backup (Recommended) (yes/no) [yes]:
 > 
 
 Please select which Settings.php file you want to use for your Environment (Recommended Version is: [0]) 
  [0] /var/www/html/meedjum-composer/docroot/sites/default/settings.php
 
 > 
 You have just selected: /var/www/html/meedjum-composer/docroot/sites/default/settings.php
 
 
 
  [ErrorException]                                               
  Use of undefined constant DRUPAL_ROOT - assumed 'DRUPAL_ROOT'  
 
gocomposer

Problem 1: DRUPAL_ROOT not defined

An exception was thrown soon after the process began - Use of undefined constant DRUPAL_ROOT - assumed 'DRUPAL_ROOT'which was thrown in my settings.local.php file on the following line:

$settings['container_yamls'][] = DRUPAL_ROOT . '/sites/default/local.services.yml';
?>

Now that's weird since it's saying that DRUPAL_ROOT isn't been set during the Drupal bootstrap process. So I changed it to $app_root which worked.

Problem 2: Removing drupal_coder

I then started the process again and get further, but was presented with

  - Removing drupal/coder (8.3.1)
    The package has modified files:
    D coder_sniffer/Drupal/Test/Arrays/ArrayUnitTest.inc
    D coder_sniffer/Drupal/Test/Arrays/ArrayUnitTest.inc.fixed
    D coder_sniffer/Drupal/Test/Arrays/ArrayUnitTest.php
    D coder_sniffer/Drupal/Test/Arrays/DisallowLongArraySyntaxUnitTest.php
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d7/DisallowLongArraySyntaxUnitTest.1.inc
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d7/disallow_long_array_d7.info
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d8/DisallowLongArraySyntaxUnitTest.2.inc
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d8/DisallowLongArraySyntaxUnitTest.2.inc.fixed
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d8/disallow_long_array_d8.info.yml
    D coder_sniffer/Drupal/Test/Classes/ClassCreateInstanceUnitTest.inc
    240 more files modified, choose "v" to view the full list
    Discard changes [y,n,v,d,?]? ?
    y - discard changes and apply the uninstall
    n - abort the uninstall and let you manually clean things up
    v - view modified files
    d - view local modifications (diff)
    ? - print help
    Discard changes [y,n,v,d,?]?

This is one of those occasions when none of the options are what I wanted, but I elected to uninstall drupal_coder

Problem 3: Config directory in the wrong place

The process continued and once it starts the database work, go for a cup of tea because it takes a lot of time. I got the following diagnostic part way

Current Step: Using Drush to update the Database... Press Enter to Continue...
------------------------------------------------------------------------------
 
 [error]  The directory <em class="placeholder">../config/syncem> does not exist. 
 [error]  The directory <em class="placeholder">../config/split/prodem> does not exist. 
 [error]  The directory <em class="placeholder">../config/split/sandboxem> does not exist.

The config directory is in the wrong place which surprised me so I elected to let the process complete and then I would investigate

GoComposer Completes

  views               make_place   post-update     Rebuild cache to allow       
                      holders_tr                   placeholder texts to be      
                      anslatable                   translatable.                
  views               remove_cor   post-update     Remove core key from views   
                      e_key                        configuration.               
 ------------------- ------------ --------------- ----------------------------- 
 
 Do you wish to run the specified pending updates? (yes/no) [yes]:
 > 
 
 
 [OK] Congrats!... You have Successfully updated your Site... The old site files and Database Sql dump are saved in the 
      backup folder                                                                                                     
 
 
 
 ! [NOTE] Your New Docroot is in the newly created /web directory... Dont forget to update you vhosts file by adding    
 !        /web to the site path...

The process finally completed, which included updated Drupal and all my modules to the latest version - you can just see in the top part of the listing above it is finishing the module updates. Now I investigated the problem with the sync directory and I was astounded to discover that the new web directory is actually a child of docroot! Say what? I assumed that docroot would be replaced by web, but instead it's created a complete subdirectory structure underneath it! That explains why the config directory is wrong in the settings.php file.

Fix Problem 3: Directory structure incorrect

GoComposer left me with a directory structure of

/var/www/html/meedjum-composer/docroot/web

So web needs moving to be a child of the the top level directory (meedjum-composer) whilst the files and directories of docroot need moving into meedjum-composer. First steps I need to clear down the top level directory.

$ pwd
/var/www/html/meedjum-composer
$ rm -r composer.*  #old stuff 
$ rm -rf vendor  #old stuff
$ rm -rf drush   #old stuff

Next copy the web directory in place and delete the obsolete docroot directory

$ cp -vaR docroot/. .
$ rm -rf docroot

Did it Work?

Before the solution can be tested, the web server docroot will need to be changed. This will vary dependent upon your web server (usually nginx or Apache2) and that server's configuration so it isn't covered here. Mine is simple because I am using the BadzillaVM so I changed my docroot setting in my Ansible playbook to point to /var/www/html/meedjum-composer/web and re-provisioned my virtual box.

I then successfully logged into my sandbox version of Badzilla and navigated to admin/reports/updates. I now have a Drupal site running core 8.8.0, and I have successfully moved away from the deprecated drupa/drupal composer template.

However the process didn't update the Admin Toolbar as you can see. So lets have a look at our new composer,json file and perform the update.

Update Admin Toolbar

The new composer.json file looks like:

{
    "name": "drupal-composer/drupal-project",
    "description": "Project template for Drupal 8 projects with composer",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "authors": [
        {
            "name": "",
            "role": ""
        }
    ],
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ],
    "require": {
        "php": ">=7.0.8",
        "composer/installers": "^1.2",
        "cweagans/composer-patches": "^1.6.5",
        "drupal/console": "^1.0.2",
        "drupal/core": "^8.6.2",
        "drupal/core-composer-scaffold": "^8.8.0",
        "drush/drush": "^9.7.1 | ^10.0.0",
        "vlucas/phpdotenv": "^4.0",
        "webflo/drupal-finder": "^1.0.0",
        "zaporylie/composer-drupal-optimizations": "^1.0",
        "drupal/token": "^1.5.0",
        "drupal/devel": "^1.2.0",
        "drupal/tome": "^1.0.0-alpha2",
        "drupal/schema_metatag": "^1.3.0",
        "drupal/addtoany": "^1.10.0",
        "drupal/ctools": "^3.0.0",
        "drupal/fontawesome": "^2.8.0",
        "drupal/config_update": "^1.5.0",
        "drupal/google_analytics": "^2.3.0",
        "drupal/search_api": "^1.10.0",
        "drupal/elasticsearch_connector": "^6.0.0-alpha1",
        "drupal/migrate_upgrade": "^3.0.0-rc5",
        "drupal/geshifilter": "^1.2.0",
        "drupal/libraries": "^3.0.0-alpha1",
        "drupal/migrate_plus": "^4.0.0",
        "drupal/config_split": "^1.4.0",
        "drupal/pathauto": "^1.3.0",
        "drupal/youtube": "^1.0.0-beta3",
        "drupal/migrate_tools": "^4.0.0",
        "drupal/admin_toolbar": "^1.24.0",
        "drupal/metatag": "^1.7.0",
        "drupal/config_inspector": "^1.0.0-beta2",
        "drupal/memcache": "^2.0.0-rc2",
        "drupal/migrate_manifest": "^1.7.0",
        "drupal/module_filter": "^3.1.0",
        "drupal/config_filter": "^1.3.0",
        "drupal/entity_reference_revisions": "^1.6.0",
        "drupal/paragraphs": "^1.3.0",
        "drupal/xmlsitemap": "^1.0.0-alpha3",
        "drupal/bootstrap": "^3.13.0"
    },
    "require-dev": {
        "drupal/core-dev": "^8.8.0"
    },
    "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "sort-packages": true
    },
    "autoload": {
        "classmap": [
            "scripts/composer/ScriptHandler.php"
        ],
        "files": [
            "load.environment.php"
        ]
    },
    "scripts": {
        "pre-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "pre-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "post-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ],
        "post-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ]
    },
    "extra": {
        "composer-exit-on-patch-failure": true,
        "patchLevel": {
            "drupal/core": "-p2"
        },
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            }
        },
        "installer-paths": {
            "web/core": [
                "type:drupal-core"
            ],
            "web/libraries/{$name}": [
                "type:drupal-library"
            ],
            "web/modules/contrib/{$name}": [
                "type:drupal-module"
            ],
            "web/profiles/contrib/{$name}": [
                "type:drupal-profile"
            ],
            "web/themes/contrib/{$name}": [
                "type:drupal-theme"
            ],
            "drush/Commands/contrib/{$name}": [
                "type:drupal-drush"
            ]
        }
    }
}

The "conflict" entry made me smile - I haven't seen that kind of entry before! But the focus is admin_toolbar and it can be seen that the reason it wasn't updated is because it has moved up a major version and the tilde on the specified version prohibited such a large jump. Let's fix this.

$ composer require drupal/admin_toolbar:2.0
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
./composer.json has been updated
> DrupalProject\composer\ScriptHandler::checkComposerVersion
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 0 installs, 1 update, 0 removals
  - Updating drupal/admin_toolbar (1.27.0 => 2.0.0): Downloading (100%)         
Package container-interop/container-interop is abandoned, you should avoid using it. Use psr/container instead.
Package phpunit/phpunit-mock-objects is abandoned, you should avoid using it. No replacement was suggested.
Writing lock file
Generating autoload files
Scaffolding files for drupal/core:
  - Copy [project-root]/.editorconfig from assets/scaffold/files/editorconfig
  - Copy [project-root]/.gitattributes from assets/scaffold/files/gitattributes
  - Copy [web-root]/.csslintrc from assets/scaffold/files/csslintrc
  - Copy [web-root]/.eslintignore from assets/scaffold/files/eslintignore
  - Copy [web-root]/.eslintrc.json from assets/scaffold/files/eslintrc.json
  - Copy [web-root]/.ht.router.php from assets/scaffold/files/ht.router.php
  - Copy [web-root]/.htaccess from assets/scaffold/files/htaccess
  - Copy [web-root]/example.gitignore from assets/scaffold/files/example.gitignore
  - Copy [web-root]/index.php from assets/scaffold/files/index.php
  - Copy [web-root]/INSTALL.txt from assets/scaffold/files/drupal.INSTALL.txt
  - Copy [web-root]/README.txt from assets/scaffold/files/drupal.README.txt
  - Copy [web-root]/robots.txt from assets/scaffold/files/robots.txt
  - Copy [web-root]/update.php from assets/scaffold/files/update.php
  - Copy [web-root]/web.config from assets/scaffold/files/web.config
  - Copy [web-root]/sites/README.txt from assets/scaffold/files/sites.README.txt
  - Copy [web-root]/sites/development.services.yml from assets/scaffold/files/development.services.yml
  - Copy [web-root]/sites/example.settings.local.php from assets/scaffold/files/example.settings.local.php
  - Copy [web-root]/sites/example.sites.php from assets/scaffold/files/example.sites.php
  - Copy [web-root]/sites/default/default.services.yml from assets/scaffold/files/default.services.yml
  - Copy [web-root]/sites/default/default.settings.php from assets/scaffold/files/default.settings.php
  - Copy [web-root]/modules/README.txt from assets/scaffold/files/modules.README.txt
  - Copy [web-root]/profiles/README.txt from assets/scaffold/files/profiles.README.txt
  - Copy [web-root]/themes/README.txt from assets/scaffold/files/themes.README.txt
> DrupalProject\composer\ScriptHandler::createRequiredFiles
PHP 7.0.33-13+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Nov 28 2019 07:43:06) ( NTS )
$ drush updb
 [success] No pending updates.
PHP 7.0.33-13+ubuntu16.04.1+deb.s

Ok so no schema changes required, and if I now point a browser at admin/reports/updates - everything looks ok as can be seen in the screenshot above.

Pin the Composer Dependencies!!!

I've always said that PHP deserves a better dependency management tool than Composer, but alas we are stuck with it. To limit the possibility (likelihood?) of disaster, you should always pin the version numbers for each Drupal project dependency. Pinning means setting a contrib module or theme or profile at a particular version so it cannot be accidentally updated on a composer update command.

So basically I will go through the composer.json file and remove all the tilde or carat symbols, and ensure the version is set to the current version on the system.

{
    "name": "drupal-composer/drupal-project",
    "description": "Project template for Drupal 8 projects with composer",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "authors": [
        {
            "name": "",
            "role": ""
        }
    ],
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ],
    "require": {
        "php": ">=7.0.8",
        "composer/installers": "^1.2",
        "cweagans/composer-patches": "^1.6.5",
        "drupal/addtoany": "1.12",
        "drupal/admin_toolbar": "2.0",
        "drupal/bootstrap": "3.21",
        "drupal/config_filter": "1.5",
        "drupal/config_inspector": "1.0",
        "drupal/config_split": "1.4",
        "drupal/config_update": "1.5",
        "drupal/console": "1.0.2",
        "drupal/core": "8.8.0",
        "drupal/core-composer-scaffold": "8.8.0",
        "drupal/ctools": "3.2",
        "drupal/entity_reference_revisions": "1.7",
        "drupal/fontawesome": "2.14",
        "drupal/geshifilter": "1.2",
        "drupal/google_analytics": "2.4",
        "drupal/libraries": "^3.0.0-alpha1",
        "drupal/memcache": "2.0",
        "drupal/metatag": "1.10",
        "drupal/module_filter": "3.1",
        "drupal/paragraphs": "1.10",
        "drupal/pathauto": "1.6",
        "drupal/schema_metatag": "1.4",
        "drupal/token": "1.5",
        "drupal/xmlsitemap": "1.0.0-alpha4",
        "drupal/youtube": "1.0",
        "drush/drush": "^9.7.1 | ^10.0.0",
        "vlucas/phpdotenv": "^4.0",
        "webflo/drupal-finder": "^1.0.0",
        "zaporylie/composer-drupal-optimizations": "^1.0"
    },
    "require-dev": {
        "drupal/core-dev": "^8.8.0"
    },
    "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "sort-packages": true
    },
    "autoload": {
        "classmap": [
            "scripts/composer/ScriptHandler.php"
        ],
        "files": [
            "load.environment.php"
        ]
    },
    "scripts": {
        "pre-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "pre-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "post-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ],
        "post-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ]
    },
    "extra": {
        "composer-exit-on-patch-failure": true,
        "patchLevel": {
            "drupal/core": "-p2"
        },
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            }
        },
        "installer-paths": {
            "web/core": [
                "type:drupal-core"
            ],
            "web/libraries/{$name}": [
                "type:drupal-library"
            ],
            "web/modules/contrib/{$name}": [
                "type:drupal-module"
            ],
            "web/profiles/contrib/{$name}": [
                "type:drupal-profile"
            ],
            "web/themes/contrib/{$name}": [
                "type:drupal-theme"
            ],
            "drush/Commands/contrib/{$name}": [
                "type:drupal-drush"
            ]
        }
    }
}

The composer.json file has now been modified and the drupal project files are pinned at particular versions for safety.

blog terms

Drupal 8 Development PHP Linux

Drupal 8 as a Static Site: Drupal 8 Contact Form Thank You Page

nigel — Sat, 04 May 2019 13:50:36 +0000

Drupal 8 as a Static Site: Drupal 8 Contact Form Thank You Page nigel Sat, 04/05/2019 - 14:50

Architecture

Our previous two tutorials on the Drupal 8 as a static site covered the Contact Form and the journey through the action to AWS Lambda to process the form and send an email. Now the final part of the contact form journey - returning back to the static site and showing a Thank You page.

In fact in this tutorial the Thank You path will actually be the same path with the contact form on it - so we return to the starting point, but we want to show a message back to the user to thank them for filling out the form. The notification back to the user will have to be JavaScript obviously since we have no backend language functionality.

If you remember the redirect back to the static site from AWS uses a HTTP code of 307. This code matches in the incoming HTTP request with the redirect. So since in our case the form was POSTed, then the redirect will also be POSTed back to the static site. This is both a blessing and a curse as we'll see.

Common wisdom says that JavaScript can neither detect whether an incoming request is POSTed or GETed, nor retrieve any of the POSTed data. That is theoretically true, but there are always workarounds of course :) :)

Adding the JavaScript

This is very similar to our earlier tutorial on how to add the Elasticsearch JavaScript client. So I won't dwell too much on this. Firstly we need to extend our template_preprocess_page function to add our thank-you JS library. This is the second section of code below, underneath the elasticsearch part

{themename}.theme - in my case the custom theme is called beezee8.

function beezee8_preprocess_page(&$variables) {

    // If we are on the search page, load the JS search client api
    // and our implementation to Elasticsearch
    if (\Drupal::routeMatch()->getRouteName() == 'search.view_node_search') {
        $variables['#attached']['library'][] = 'beezee8/elastic-library';
    }

    // If we are on the contact form page, load the JS to show thank you message
    if (\Drupal::request()->getRequestUri() == '/about') {
        $variables['#attached']['library'][] = 'beezee8/thank-you-library';
   }
}
?>

In the code above I get the current path alias and should that equal the '/about' page then I inject my JS library which is defined below.

Libraries file

The libraries file now needs a further entry for our thank you JS. Again we already created an entry for elasticsearch so I am not going to go over this in too much detail.

{themename}.libraries.yml

global-styling:
  css:
    theme:
      css/style.css: {}
elastic-library:
  js:
    js/elasticsearch-js/elasticsearch.min.js: {}
    js/beezee/beezee_elastic.js: {}
thank-you-library:
  js:
    js/beezee/beezee_thankyou.js: {}

JavaScript Thank You

Now here's the JavaScript. I did say that there is no way of knowing whether the HTTP request is GET or POST - well true, but we can deduce it. If we interrogate the referrer and it equals the current URL then we can make the assumption that the form has been POSTed, completed its round trip to AWS Lambda and back with a 307 code, and therefore we should show the thank you message. If you have a look at my code below you will see the conditional, and you will see the markup I inject into the page - it's classic Drupal 8 Bootstrap since that's how my theme is built. Your mileage here will obviously vary dependent upon your own theme.

{themename}_thankyou.js

(function ($, Drupal) {
 
    var thank_once;
 
    function BeezeeThankyou() {
        if (!thank_once) {
            thank_once = true;
 
            // If we are arriving here from lambda which passes on the referrer from here
            // then the form has been POSTed so show the thank you page
            if (document.referrer == document.URL) {
 
                // Markup to add the thankyou message
                var thanks = [
                    '',
                    '',
                    'Status message
',
                    'Thank you for contacting me. I will be in touch.',
                    '
'
                ];
 
                // Inject the markup
                $('.region-highlighted').append(thanks.join(''));
            }
        }
    }
 
    Drupal.behaviors.beezee_thankyou = {
        attach: function (context, settings) {
            BeezeeThankyou();
        }
    };
})(jQuery, Drupal);

nginx Server Configuration

If we try our form now (or at least if I do since I'm using nginx) - we will get an nginx 405 Not Allowed error. Remember I said the POST could be a blessing and a curse? The error is because by default nginx will not allow a POST to a static page. Thankfully there is a workaround. Locate the server configuration for your particular virtual host, and edit it by adding the following line into the server section.

{VM name}.conf

server {
.
.
.
      error_page  405     =200 $uri;
.
.

Form submission

Ok we should now be good to go. Submit the form and it should do the round trip from static site to AWS Lambda back to static site and an email arriving in your inbox!

An even better solution?

If you look carefully above you will see a placeholder in the thank you message. So am I saying it is possible to retrieve the POSTed data and insert the name into the placeholder? Sort of. There is a JS library called jQuery-PostCapture which can achieve this. It works by capturing the POSTed data when the form is initially submitted. This copy of the data is stored either as a cookie or in the browser's local storage. Once the trip to AWS Lambda is completed, this data can be fetched from the cookie or local storage - so the data isn't fetched from the actual POST itself.

The problem I've got with this is neither a cookie nor local storage is secure. However if you feel that the gains (being able to customise a message on the thank you page) is worth the security risk, then give it a shot!

blog terms

Static Site Drupal 8 Drupal PHP Development JavaScript

Drupal 8 as a Static Site: AWS API Gateway, Lambda and SES Form Processing

nigel — Wed, 02 Jan 2019 10:38:46 +0000

Drupal 8 as a Static Site: AWS API Gateway, Lambda and SES Form Processing nigel Wed, 02/01/2019 - 10:38

Serverless Installation

The intention is to build our AWS ecosystem using the serverless product. This will enable us to configure our AWS provisioning in a YML file which is then translated into AWS CloudFormation orchestration. In the same codebase our PHP Lambda function can be built out. It isn't my intention to provide heavy documentation here since the Contact Form architecture was defined in an earlier blog, and I have previously written many articles on the serverless / lambda / API Gateway combination of technologies to deliver HTTP content and handle form POSTs.

Notwithstanding this, I will of course document points of interest along the way so it all makes sense!

The first step is to install serverless on your system. A dependency is node, so if you don't have that then please follow tutorials elsewhere on the Internet before executing the command below to install globally serverless.

$ npm install serverless -g

We will be using Andy Raines' excellent PHP for AWS Lambda via Serverless Framework repo as our stating point. Issuing the following command will install our PHP serverless project into a directory called d8-contact-form.

$ serverless install --url https://github.com/araines/serverless-php -n d8-contact-form
Serverless: Downloading and installing "serverless-php"...
Serverless: Successfully installed "d8-contact-form"

We then cd into the directory and list.

$ cd d8-contact-form/
$ ls -las
total 48
0 drwxrwxr-x 16 501 dialout  512 Jan  2 11:14 .
0 drwxr-xr-x 14 501 dialout  448 Jan  2 11:14 ..
4 -rwxr-xr-x  1 501 dialout  660 Feb 14  2018 buildphp.sh
4 -rw-r--r--  1 501 dialout  569 Feb 14  2018 CHANGELOG.md
4 -rw-r--r--  1 501 dialout  430 Feb 14  2018 composer.json
0 drwxrwxr-x  3 501 dialout   96 Jan  2 11:14 config
4 -rw-r--r--  1 501 dialout 1011 Feb 14  2018 dockerfile.buildphp
4 -rw-r--r--  1 501 dialout   40 Feb 14  2018 .gitattributes
4 -rw-r--r--  1 501 dialout  120 Feb 14  2018 .gitignore
4 -rw-r--r--  1 501 dialout 1330 Feb 14  2018 handler.js
4 -rw-r--r--  1 501 dialout  829 Feb 14  2018 handler.php
4 -rw-r--r--  1 501 dialout 1103 Feb 14  2018 LICENSE
4 -rwxr-xr-x  1 501 dialout  133 Feb 14  2018 php
4 -rw-r--r--  1 501 dialout 3705 Feb 14  2018 README.md
4 -rw-r--r--  1 501 dialout 3008 Jan  2 11:14 serverless.yml
0 drwxrwxr-x  5 501 dialout  160 Jan  2 11:14 src

Since I don't have Git LFS installed, the PHP executable hasn't been downloaded from the repo correctly. I will download this manually, and add it to the directory. Once completed, we can see it there with:

$ ls -lash | grep " php"
 27M -rwxr-xr-x  1 501 dialout  27M Jan  2 11:22 php

Now we need to install the PHP dependencies using composer. This does have a dependency on PHP 7.1.

$ composer install -o --no-dev
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Loading composer repositories with package information
Updating dependencies
Package operations: 9 installs, 0 updates, 0 removals
  - Installing psr/log (1.1.0): Downloading (100%)         
  - Installing monolog/monolog (1.24.0): Downloading (100%)         
  - Installing symfony/polyfill-ctype (v1.10.0): Loading from cache
  - Installing symfony/filesystem (v4.2.1): Downloading (100%)         
  - Installing symfony/config (v4.2.1): Downloading (100%)         
  - Installing symfony/contracts (v1.0.2): Downloading (100%)         
  - Installing psr/container (1.0.0): Loading from cache
  - Installing symfony/dependency-injection (v4.2.1): Downloading (100%)         
  - Installing symfony/yaml (v4.2.1): Downloading (100%)         
Writing lock file
Generating optimized autoload files

Any Raines' code comes with a very basic Lambda function called hello. To prove the environment is working correctly, we should invoke it locally (no need for deploying to AWS yet) and if everything is correct, it'll look like this:

$ serverless invoke local -f hello
Got event [] []
{
    "statusCode": 200,
    "body": "Go Serverless v1.0! Your function executed successfully!"
}

In an earlier blog I replaced the references of Hello with demo. I have changed it to D8ContactForm for the contact form. By deploying the code to AWS, and submitting the contact form, you will see the same screenshot as shown at the end of the previous blog.

SES Validation of email address

To use the AWS SES service, your email address will need to be verified. This is easy - navigate to Simple Email Service and click on email addresses, then "Verify a new email address". Add the email address and a verification email message will be sent for you to click back and confirm the verification process. Once verified, you'll see the second screenshot above.

Serverless configuration

The next step is to add the SES configuration to the serverless.yml file. Add a custom area for your own variables - in this case the email address.

custom:
  SENDER_EMAIL: xxxx@xxxxxxxx
  RECIPIENT_EMAIL: xxxx@xxxxxxxx

And add the SES config to the provider area so it looks like this in its entirety:

provider:
  name: aws
  runtime: nodejs6.10
  stage: dev
  region: eu-west-1
  environment:
    SENDER: ${self:custom.SENDER_EMAIL}
    RECIPIENT: ${self:custom.RECIPIENT_EMAIL}
    DOMAIN: "*"
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "ses:SendEmail"
      Resource: "*"

Then deploy

$ sls deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...

Add AWS SDK to the codebase using composer

The SDK will make it much easier to build out the SES interface code and can be installed to the codebase in the usual composer way

$ composer require aws/aws-sdk-php
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Using version ^3.90 for aws/aws-sdk-php
./composer.json has been updated

Add SDK PHP required libraries

The AWS PHP SDK requires the filter and simplexml libraries that aren't in the PHP executable we deploy to AWS by default. Thankfully there is a way of recompiling the PHP executable and I covered this in an earlier blog. So the tl;dr is

Add the filter and simplexml libraries to dockerfile.buildphp
Make a note of the php creation date and filesize.
Make sure buildphp.sh is executable on the command line and run it.

Add the Serverless and SDK handler code

Now the AWS SDK is in the code base we can build out the functionality to integrate the SDK. Our first work is to ensure that the recipient and sender email addresses are available to the code. To achieve this, they were defined in the serverless.yml in the environment section which can be picked up in handler.php and assigned to the $event array which is made available to the handler class. Add the following lines before the call to handle

$event['sender_email'] = getenv('SENDER');
$event['recipient_email'] = getenv('RECIPIENT');
?>

Now the handler itself. A few things to note.

I have added the necessary use clauses to the AWS SDK client and the exception class
The protected $client_config holds the configuration we have - but note that further down I am getting the credentials from the CredentialProvider
The recipients are an array and so it is possible if required to add multiples here.

D8ContactFormHandler.php

namespace Raines\Serverless;

require 'vendor/autoload.php';

use Aws\Ses\SesClient;
use Aws\Exception\AwsException;

class D8ContactFormHandler implements Handler
{
    protected $client_config = [
        'region' => 'eu-west-1',
        'version' => '2010-12-01',
        'credentials.cache' => TRUE,
        'validation' => FALSE,
    ];

    /**
     * {@inheritdoc}
     */
    public function handle(array $event, Context $context)
    {
        $logger = $context->getLogger();
        // Comment out for debugging
        //$logger->notice('Got event', $event);

        // Set up AWS SDK
        $this->client_config['credentials'] = \Aws\Credentials\CredentialProvider::env();
        $SesClient = new SesClient($this->client_config);

        $sender_email = $event['sender_email'];
        $recipient_emails[] = $event['recipient_email'];

        return [
            'statusCode' => 200,
            'body' => 'Go Serverless v1.0! Your function executed successfully!',
        ];
    }
}

?>

Email generation

Now we can insert the code to actually generate the email. All of this is self evident and really doesn't need further commentary from me. It should be inserted just before the return statement.
D8ContactFormHandler.php

        // Process the submitted form.
        // *TODO* This could do with more validation.
        $fields = [];
        parse_str($event['body'], $fields);

        if (!isset($fields['name'])) $fields['name'] = '{unknown name}';
        if (!isset($fields['mail'])) $fields['mail'] = '{unknown email address}';
        if (!isset($fields['subject'][0]['value'])) $fields['subject'][0]['value'] = '{unknown subject}';
        if (!isset($fields['message'][0]['value'])) $fields['message'][0]['value'] = '{unknown message}';

        $subject = '[badzilla.co.uk website feedback] '.$fields['subject'][0]['value'];
        $plaintext_body = 'From: '.$fields['name'].'  '.$fields['mail'].
            'Subject: '.$fields['subject'][0]['value'].
            ' Message: '. $fields['message'][0]['value'];
        $html_body =  ''.$fields['subject'][0]['value'].'
'.
            ''.'From: '.$fields['name'].'  .$fields['mail'].'">'.$fields['mail'].''.'
'.
            ''.$fields['message'][0]['value'].'
';
        $char_set = 'UTF-8';

        try {
            $result = $SesClient->sendEmail([
                'Destination' => [
                    'ToAddresses' => $recipient_emails,
                ],
                'ReplyToAddresses' => [$sender_email],
                'Source' => $sender_email,
                'Message' => [
                    'Body' => [
                        'Html' => [
                            'Charset' => $char_set,
                            'Data' => $html_body,
                        ],
                        'Text' => [
                            'Charset' => $char_set,
                            'Data' => $plaintext_body,
                        ],
                    ],
                    'Subject' => [
                        'Charset' => $char_set,
                        'Data' => $subject,
                    ],
                ],
            ]);
            $messageId = $result['MessageId'];
        } catch (AwsException $e) {
            // output error message if fails
            $logger->notice('Message', $e->getMessage());
            $logger->notice('AWS Message', $e->getAwsErrorMessage());
        }
?>

Redirect to thank you page

Once we've sent the email we want to redirect back to our thank you page. This is done with a 307 code which preserves the original POSTed values so they can be used on the forwarded page. Our use will be to make the message personalised. I have decided that I would like my redirect back to my originating page, but of course you could customise it as you see fit. For my case, I use the Referer header. The code below replaces the existing return functionality.
D8 ContactFormHandler.php

        return [
            'headers' => ['Location' => $event['headers']['Referer']],
            'statusCode' => 307,
        ];
codE>?>

Debugging

During the development process you will have to do multiple deploys and doubtless you will get numerous error responses from AWS. The best way of working these issues is to use CloudWatch, and a sample screenshot is shown above. The codebase has an excellent logging facility to check variables and these are written out to CloudWatch. Have a look at:

$logger->notice('Got event', $event);
?>

This can be amended for whatever you need to interrogate, but remember the second parameter must be an array.

Git Repository

The entire codebase discussed here is at https://github.com/sanddevil/serverless-php-contact-form - if you wish to extend its functionality, please send me a pull request.

blog terms

Static Site Drupal Drupal 8 PHP Development serverless serverless-php lambda

Drupal 8 as a Static Site: Drupal 8 Contact Form

nigel — Mon, 31 Dec 2018 10:38:02 +0000

Drupal 8 as a Static Site: Drupal 8 Contact Form nigel Mon, 31/12/2018 - 10:38

Drupal Contact

Drupal core comes with a Contact form, so the first step is to check it is already enabled by navigating to /admin/modules as per the first screenshot above.

Next go to /admin/structure/contact and the default forms are listed including the Website Feedback form which is the one we will be using. See the second screenshot.

The third screenshot shows the default fields. I've made two changes here. The form preview requires a Drupal backend so has been removed, and I've disabled the 'Send copy to sender' option since that could encourage spamming.

Add contrib module Contact Block

Since I want my contact form to be below my 'about' content on my /about route, I need the form to be available as a block. This can be achieved using the contrib module Contact Block and can be installed and enabled in the normal way.

$ composer require drupal/contact_block
Using version ^1.4 for drupal/contact_block
./composer.json has been updated
Gathering patches for root package.
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.
  - Installing drupal/contact_block (1.4.0): Downloading (100%)         
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess
$ drush en contact_block
 [success] Successfully enabled: contact_block

Add contact form block to region

Adding the content form block to a block region should be familiar to most people so I won't go into this in great detail. I have placed it in the content region (first screenshot) at /admin/structure/block, configured it with the title, path restrictions so it only appears on my /about route, and ensured the correct form is to be displayed (second screenshot). The third screenshot shows I have dragged the contact form block to be the final block in the content region.

POST form to external URL

Since it is not possible in the admin UI to add an external URL for the POST action, we will have to achieve this by a few lines of code. But where are we POSTing the form anyway? This is covered in the next blog, but in a nutshell the URL is autogenerated by AWS API Gateway during the CloudFormation orchestration. I won't go into the details now how that is achieved since it can be read later in this sequence, but here is the url we'll use.

$ sls info | grep POST
  POST - https://djbbm406h0.execute-api.eu-west-2.amazonaws.com/dev

The easiest way of adding that url to the Drupal contact form would be to include it into settings.php and then use a template preprocess function to swap out the 'action' value in the form.
settings.php

$config['aws:lambda']['url'] = 'https://djbbm406h0.execute-api.eu-west-2.amazonaws.com/dev';
?>

{themename}.theme

/**
 * @param $variables
 */
function beezee8_preprocess_form(&$variables) {

  // find the contact form and add the lambda contact form action url
  if ($variables['attributes']['id'] == 'contact-message-feedback-form') {
    $variables['attributes']['action'] = \Drupal::config('aws:lambda')->get('url', FALSE);
  }
}
?>

Submission of the contact form

Now the form will be submitted to AWS Lambda for onward processing, and ultimately, a redirect back with a thank you message.

blog terms

Static Site Drupal Drupal 8 PHP Development

Drupal 8 as a Static Site: Contact Form Architecture

nigel — Wed, 26 Dec 2018 11:06:18 +0000

Drupal 8 as a Static Site: Contact Form Architecture nigel Wed, 26/12/2018 - 11:06

As I have already mentioned in this sequence of blogs, the intention is to use AWS products for my static site contact form. It's worthwhile explaining the proposed architecture and steps involved to implement such a solution. On the face of it, it may appear somewhat convoluted, but AWS provide a number of complementary products to achieve what we want, and the configuration can be saved in code so it is easy to reproduce.

Architecture

The above diagram shows the architecture of our contact form, and the flow through the form ecosystem. Let's have a look at the components.

Webpage with Contact Form

Drupal comes with a contact form in core, and therefore this is fairly straightforward, albeit with a couple of hurdles to jump. Firstly I would like to place my contact form on my About Me page underneath my existing bio. To achieve this I need the form to be a block which will require the installation of a contrib module. In addition I want my form's action to POST the form data to an external URL which isn't supported in the core form - so that will need a solution too.

AWS API Gateway

The form will be submitted for processing to AWS since we don't have a backend on our own static site - hence why the form needs an external URL for its action parameter.

We will use AWS API Gateway which is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. This will give us access to AWS compute backend services to process our POSTed form.

AWS Lambda Function

AWS Lambda is a FaaS (Function as a Service) compute service. It is excellent at running a snippet of code on demand once it's been triggered, for instance in our case by the contact form being POSTed.

PHP isn't a supported Lambda language, but there are workarounds for this - as explained in my sequence of blogs here. Using these workarounds, we will develop a PHP app to process the form, and in addition we will need the AWS PHP SDK to interface with AWS SES (Simple Email Service).

The PHP code will reside in Amazon's S3 object storage, and we will use the serverless product along with the aws console to manage the deploy process.

AWS SES

AWS SES is a very cost effective method of sending marketing, notification and transactional emails. For the amount of usage a blog contact form would see, it's highly unlikely to go above the AWS free tier threshold. It is the perfect tool for sending the contact form contents onto my personal email account.

Inbox

The objective is the form contents should arrive into my Gmail inbox for me to read and action if necessary.

Thank you page

The last activity of the Lambda function is to redirect the submitted form back to the website's thank you page which will have to be constructed as part of this exercise. The redirect will be a 307 to preserve the POSTed parameters that can be parsed and injected into the thank you message with a few lines of JavaScript.

blog terms

Static Site Drupal Drupal 8 lambda

Drupal 8 as a Static Site: JavaScript Search Client

nigel — Sat, 15 Dec 2018 10:19:04 +0000

Drupal 8 as a Static Site: JavaScript Search Client nigel Sat, 15/12/2018 - 10:19

We now need to plug the Elasticsearch query that was built in the previous blog into a JavaScript client that has been loaded on the search page as described in another earlier blog. So a few lines of JavaScript will be required here, and it will need to work on my three environments:

Sandbox Drupal site connecting to local Elasticsearch server.
Sandbox static site connecting to local Elasticsearch server.
Production static site connecting to production Elasticsearch server.

The JavaScript will use the elasticsearch.js API library which can be downloaded from here. We need to make our theme aware of our JavaScript files and we only wanted them loading on the node search page route. Our JavaScript file will be called beezee_elastic.js, which is the name of my theme (beezee8) and elastic obviously.

Add the JavaScript to the Beezee theme

Firstly we need an entry in our theme's info.yml file under the Libraries heading.
beezee8.info.yml

libraries:
  - 'beezee8/elastic-library'

Next we need to append to the libraries yml file the location of our JavaScript files - the Elastic API library and our own personal script.
beezee8.libraries.yml

elastic-library:
  js:
    js/elasticsearch-js/elasticsearch.min.js: {}
    js/beezee/beezee_elastic.js: {}

Obviously we then have to copy our JS files into the filesystem at those locations - for now I will have a placeholder empty file for beezee_elastic.js which we will flesh out later.

Load the JavaScript files on the search page only

There are a few ways of doing this, but my favourite is to use a preprocessor function in the {themename}.theme file. We can use hook_preprocess_page for this.
beezee8.theme

/**
 * @param $variables
 */
function beezee8_preprocess_page(&$variables) {

    // If we are on the search page, load the JS search client api
    // and our implementation to Elasticsearch
    if (\Drupal::routeMatch()->getRouteName() == 'search.view_node_search') {
        $variables['#attached']['library'][] = 'beezee8/elastic-library';
    }
}
?>

The JavaScript search client

Ok so finally we are ready to build out our front end search client. Here we go!
beezee_elastic.js

(function ($, Drupal) {
 
    var elastic_once;
 
    function BeezeeElastic() {
        if (!elastic_once) {
            elastic_once = true;
 
            // Get the params from the url
            var pageURL = window.location.search.substring(1);
            var URLVariables = pageURL.split('&');
            var keys;
            i = 0;
            while (i < URLVariables.length) {
                var ParameterName = URLVariables[i].split('=');
                if (ParameterName[0] == 'keys') {
                    keys = ParameterName[1];
                }
                i++;
            }
 
            // Did we get anything from the URL? If not bail, otherwise do the search.
            // We shouldn't get to here so don't show a message to user.
            if (!keys) {
                console.log('Invalid keys');
                return;
            }
 
            // Add the search key to the h1 selector
            $("h1").append(Drupal.t(" for ") + keys);
 
            // Endpoint defined during instantiation of API Client
            var client = new elasticsearch.Client({
                host: 'http://meedjum.test:9200/' /*,*/
                /* log: 'trace' */
            });
 
            client.search({
                index: 'elasticsearch_index_badzilla_meedjum_staticcontent',
                body: {
                    query: {
                        function_score: {
                            query: {
                                query_string: {
                                    query: keys,
                                    fields: [
                                        "title",
                                        "*body",
                                        "term"
                                    ],
                                    default_operator: "OR"
                                }
                            },
                            functions: [
                                {
                                    linear: {
                                        "created": {
                                            origin: "now",
                                            offset: "365d",
                                            scale: "1460d",
                                            decay: 0.5
                                        }
                                    }
                                }
                            ]
                        }
                    },
                    highlight: {
                        number_of_fragments: 1,
                        pre_tags: [""],
                        post_tags: [""],
                        fragment_size: 400,
                        no_match_size: 400,
                        phrase_limit: 1,
                        fields: {
                            "*body": {},
                            "title": {},
                            "term": {}
                        }
                    },
                    size: 10
                }
            }, response);
 
            /*
             Process the response from Elastic
             */
            function response(err, resp, status) {
 
                // Are we logged in? If so our jQuery calls will be slightly different due to additional DOM stuff
                // for non anonymous
                // These are our targets for injecting the content on the page. Markup is Bootstrap 3
                var h1_find;
                var h2_find;
                if ($(".user-logged-in").length) {
                    h1_find = "h1 + nav:last-child";
                    h2_find = "h1 + nav + h2:last-child";
                } else {
                    h1_find = "h1";
                    h2_find = "h1 + h2:last-child";
                }
 
                var hits = resp.hits.hits;
 
                if (resp.hits.total == 0) {
                    $(h1_find).after("" + Drupal.t("Your search yielded no results.") + "
");
                }
                else {
                    $(h1_find).after("" + Drupal.t("Search results") + "
");
 
                    var inject = "";
                    $.each(hits, function (key, value) {
 
                        // Url, title and Body
                        var prefix = "";
                        var suffix = "
";
                        var title = "\"" + value._source.url[0] + "\">" + value.highlight.title[0] + "";
 
                        // do we have a book body or a paragraph body?
                        var body;
                        if (typeof value.highlight.blog_body != 'undefined') {
                            body = value.highlight.blog_body[0];
                        } else {
                            body = value.highlight.body;
                        }
                        body = "" + body + "
";
 
                        inject += prefix + title + body + suffix;
 
                        // Add the tag
                        if (typeof value.highlight.term != 'undefined') {
                            inject += "" + Drupal.t("Tag:") + " " + value.highlight.term + "
";
                        }
 
                        // Author and date
                        var gb_obj = new Date(value._source.created[0] * 1000);
                        var gb_date = ('0' + gb_obj.getDate()).slice(-2) +
                            "/" + ('0' + (gb_obj.getMonth() + 1)).slice(-2) +
                            "/" + gb_obj.getFullYear() +
                            " " + "-" + " " +
                            gb_obj.getHours() + ":" +
                            gb_obj.getMinutes();
 
                        inject += "" + value._source.author[0] + " " + "-" + " " + gb_date;
 
                    });
                    inject += "
";
                    $(h2_find).after(inject);
                }
            }
        }
    }
 
 
    Drupal.behaviors.beezee_elastic = {
        attach: function (context, settings) {
            BeezeeElastic();
        }
    };
})(jQuery, Drupal);

Code commentary

I'd like to think the code is self-explanatory, but here are a few words for additional clarification.

The first block of code is processing the URL params. We are expecting the ?keys={search_phrase}. The search phrase needs to be injected into the Elasticsearch query later.

The client object is returned from me instantiating the Elasticsearch API. I needed a parameter of the Elasticsearch endpoint which is currently hardcoded. It will work for my two sandbox environments (Drupal and static) but not for production. I will have to come up with a solution to rewrite the endpoint during the deployment process!

The client.search call is our query created earlier, and I have added the search term from the URL. The Elasticsearch API provides two mechanisms for this call - it can return a promise or invoke a callback. I opted for the latter.

The response function is the callback and it handles the Elastic response. My first task is to set the selectors where I will inject the results - note that the DOM is different whether I am logged into my sandbox Drupal or not - so my injection anchor points are different dependent upon this circumstance.

I add the title, body text, tag, author and published date to my output, concatenating them together, and then inject them into the content area of the screen around ordered list tags.

The next result is very similar visually to the default Drupal core search with the addition of my blog tags, although of course it can be styled differently if wanted.

The results

The script works well on both the Drupal and static sites in sandbox. The screenshot above shows a search on the word nginx. At the moment I haven't built in a paginator, but that could come along later.

blog terms

Static Site Drupal Drupal 8 JavaScript

Drupal 8 as a Static Site: Elasticsearch Query

nigel — Wed, 28 Nov 2018 14:35:57 +0000

Drupal 8 as a Static Site: Elasticsearch Query nigel Wed, 28/11/2018 - 14:35

The most fundamental part of the search is the query - this will determine which documents will be retrieved after a search. It must obviously reflect the search phrase input into the search form, and the results returned should be presented in an order of most relevant first.

Elasticsearch is excellent at this - but it's crucial that the query reflects exactly what I want to achieve and it searches the correct fields. No two sites are the same and therefore no two search requirements are identical. I have decided that I want to search on the following:

Blog title
Blog body text (a custom defined Paragraph type)
Body text (only used with the book content type and not my regular blogs)
Technology vocabulary which I use to tag my blog

Each of these will be assigned a score during the search process and the results delivered back in high to low score order.

Initial basic query

Unless you re an expert in Elastsearch queries (I am definitely not that), you will need a useful tool with automatic syntax suggestions, and lo and behold Kibana is excellent at this. My first stab at the query is shown below, and it behaved pretty much to my expectations

GET /myindex/_search
{
  "query": {
        "query_string": {
           "query": "Search phrase here",
           "fields": [
              "title",
              "*body",
              "term"
              ],
            "default_operator": "or"
            }
  },
  "highlight": {
        "number_of_fragments" : 1,
        "pre_tags": [ "" ],
        "post_tags": [ "" ],
        "fragment_size": 400,
        "no_match_size": 400,
        "phrase_limit": 1,        
        "fields" : {
            "*body": {},
            "title": {},
            "term": {}
        }
  },
  "size": 10
}

Some of this is self-explanatory, some less so. Let's cherry pick our explanations.
*body - I have two fields which contain the word body and these can be searched using a wildcard.
highlight - This is where I specify what I actually want to display back to the end user. The following parameters defined this.
number_of_fragments - The maximum number of fragments to return.
pre_tags and post_tags - When the search term is found, we can specify what tags we want before and after. I chose strong instead of the default em
fragment_size - How many characters the snipped of the find will contain as a maximum, although it could be less depending upon how Elasticsearch's paragraph and line break algorithm interprets the result
no_match_size - As above if there are no matches in a particular field. I still want to show the snippet so the user has got some context.
phrase_limit - Controls the number of matching phrases in a document that are considered.

The major shortcoming of this query is it shows no favouritism towards the date the blog is published. Whether it was published yesterday or ten years ago, the score will be same.

Adding a date decay to favour more recent blogs

Technical blogs are ephemeral and transitory. They tend to have a shelf life when they are very popular, but then their popularity will wane as time goes by. This can be due to a number of factors - such as the popularity of the subject of the blog, the technology written about evolving thus obsoleting the origin blog, and the subject of the blog becoming commonplace or covered more extensively in official documentation.

Therefore we need a mechanism to favour newer blogs in our search that will be more relevant to our users. Elasticsearch has the decay feature, and it supports three different mathematical formula. There is gauss, linear and exp. I had a think here - I could use the default Gaussian approach and with some subtle parameter settings I think it would have done a good job. I could also have used Exponential, with the relatively shallow curve to present the passing of time. In the end I decided I liked a linear representation.

My rationale is this: A blog will be totally relevant for about a year, then it will decay at a linear rate for a number of years, then it will hit the bottom. Sheer speculation drove me to use a total decay period of eight years although I concede this is somewhat arbitrary and probably sub-optimal.

The syntax for representing this is a little arcane. Here we go:

"function_score": {
  "functions": [
    {
      "linear": {
        "created": {
          "origin": "now",
          "offset": "365d", 
          "scale": "1460d",
          "decay": 0.5
          }
        }
      }
    ]
  }

Here is the commentary:
created - This is the name of the date field for the date the content was published
origin - The start point for our time span, which will always be anchored at the current date.
offset - This is the initial flat spot at the top of the graph above. I am saying here - all documents in the first year will be scored equally from a date created perspective.
scale - Defines the distance from origin + offset at which the computed score will equal decay parameter.
decay - The decay parameter defines how documents are scored at the distance given at scale. So if you divide decay into scale you get the figure of 8 years.

The complete query

Adding the decay to the initial query means that they multiplied together (inferred by default).

GET /myindex/_search
{
 
  "query": {
    "function_score": {
      "query": {
        "query_string": {
          "query": "Search phrase here",
             "fields": [
                "title",
                "*body",
                "term"   
             ],
             "default_operator": "OR"
        }
      },
      "functions": [
        {
          "linear": {
            "created": {
              "origin": "now",
              "offset": "365d", 
              "scale": "1460d",
              "decay": 0.5
            }
          }
        }
      ]
    }
  },
    "highlight": {
    "number_of_fragments" : 1,
    "pre_tags": [ "" ],
    "post_tags": [ "" ],
    "fragment_size": 400,
    "no_match_size": 400,
    "phrase_limit": 1,        
    "fields" : {
      "*body": {},
      "title": {},
      "term": {}
    }
  },
  "size": 10
}

blog terms

Static Site Drupal Drupal 8

Drupal 8 as a Static Site: Disable Core Search

nigel — Sun, 18 Nov 2018 10:04:27 +0000

Drupal 8 as a Static Site: Disable Core Search nigel Sun, 18/11/2018 - 10:04

Since the static site will not have access to the Drupal 8 Core Search facility, it can be removed by using the command drush pm_uninstall search on the command line, or by visiting admin/modules/uninstall.

The problem with that approach is core search also has the search form that appears as a block in the navbar on every page of my website. If I uninstall core search I lose the form at the same time. It is my desire to keep on using the search form despite having an Elasticsearch backend. The proposed Elasticsearch JavaScript client will be installed on the /search route which is where the core search form action is directed to.

Okay, so feasibly I could uninstall the core search and do either:

Create a custom block with a search form that replicates (from an HTML frontend perspective) the core search form.
Copy the generated markup from the core search form, and paste it into the page template.

I don't like either approach. I would prefer to keep the core search and stub it so it doesn't actually do any backend searching. That way I will have near identical functionality for both my sandbox static and sandbox Drupal sites.

To stub the search I would have to alter the default search route and replace it with my own custom route with a controller that returns nothing.

Define the route subscriber config

The first step is to add to our custom module's services file.
badzilla_static.services.yml

  <-- previous entry snipped -->
  badzilla.route_subscriber:
      class: Drupal\badzilla_static\Routing\RouteSubscriber
      tags:
        - { name: event_subscriber }

We have now defined where our route subscriber lives. Let's create.

Route subscriber class

We need to create the directory structure and the file itself first.

$ mkdir  src/Routing
$ touch src/Routing/RouteSubscriber.php

And the file contents are:
RouteSubscriber.php

/**
* @file
* Contains \Drupal\mymodule\Routing\RouteSubscriber.
*/

namespace Drupal\badzilla_static\Routing;

use Drupal\Core\Routing\RouteSubscriberBase;
use Symfony\Component\Routing\RouteCollection;



/**
* Listens to the dynamic route events.
*/
class RouteSubscriber extends RouteSubscriberBase {

    /**
    * {@inheritdoc}
    */
    public function alterRoutes(RouteCollection $collection)
    {
        // Replace dynamically created "search.view_node_search" route's Controller with my own.
        if ($route = $collection->get('search.view_node_search')) {
            $route->setDefault('_controller', '\Drupal\badzilla_static\Controller\BadzillaStaticSearchController::view');
        }
    }
}
?>

Here I am replacing the built-in search module's controller with my own. Now let's build out my own do-nothing controller.

Custom search controller

As previously, create the file structure.

$ mkdir src/Controller
$ touch src/Controller/BadzillaStaticSearchController.php

Here's the controller in all its glory.
BadzillaStaticSearchController.php

namespace Drupal\badzilla_static\Controller;

use Drupal\search\SearchPageInterface;
use Symfony\Component\HttpFoundation\Request;
use Drupal\search\Controller\SearchController;

/**
 * Override the Route controller for search.
 */
class BadzillaStaticSearchController extends SearchController {

    /**
     * {@inheritdoc}
     */
    public function view(Request $request, SearchPageInterface $entity)
    {
        $build = [];
        $plugin = $entity->getPlugin();
        $build['#title'] = $plugin->suggestedTitle();

        $results = [];
        $build['search_results'] = [
            '#theme' => [
                'item_list__search_results__' . $plugin->getPluginId(),
                'item_list__search_results',
            ],
            '#items' => $results,
            '#empty' => [
                '#markup' => '',
            ],
            '#list_type' => 'ol',
            '#context' => [
                'plugin' => $plugin->getPluginId(),
            ],
        ];

        return $build;
    }
}
?>

The view function must return a render array, and so I have honoured the convention of the parent SearchController::view to large extent although obviously in my case there will always be no results returned. A useful stub!

Clear Rebuild and Test

Since we've added to our services yaml file we need to do the customary cache rebuild for our changes to take effect.

$ drush cr
 [success] Cache rebuild complete.

Next let's attempt a search on the dynamic and the static version of my sandbox Badzilla sites. The two screenshots above show that both the static and the dynamic version of the sandbox sites are correctly showing a blank search page. Next we have to plug in the client....

blog terms

Static Site Drupal 8 Development Drupal PHP

Drupal 8 as a Static Site: Search Architecture and Drupal Configuration

nigel — Thu, 15 Nov 2018 14:21:18 +0000

Drupal 8 as a Static Site: Search Architecture and Drupal Configuration nigel Thu, 15/11/2018 - 14:21

As we've already discussed in earlier blogs, any successful odyssey towards a static version of the Badzilla blog using the Drupal module Tome will depend upon having a good replacement search facility in place. The built-in Drupal search won't work since there will be no back-end to process the search request. There are many alternative ways of achieving a viable search solution on a static site, and I've listed a few in my introductory page Drupal 8 as a Static Site. Ideally I would like a solution that will work on both my dynamic, 'Drupal' version of the site (i.e. where I create my content) and the Tome generated static version of the site (i.e. the version viewable by the public).

To my mind, the obvious answer is Elasticsearch combined with a JavaScript client. I would have opted for Google Site Search but it was closed down in April 2018 and replaced with Custom Search Engine (CSE). CSE offers similar capabilities to Site Search but carries ads. That automatically deselects itself from consideration.

Elasticsearch Architecture

My proposed architecture is simplicity itself. I have a local sandboxed Virtual Machine with my development environment installed on it. For reference, as well as blatant self-promotion, I use the BadzillaVM which is a Ubuntu image with a raft of utilities and tools pre-installed, including Elasticsearch.

The architectural solution is to create an Elasticsearch index on my sandbox. To that end I'll use the Drupal Elasticsearch connector and the Search API modules. Every time content is added on to my local version of Drupal, the index will be updated automatically.

I will then trigger an automated build process to generate the static pages and to dump the Elasticsearch index using elasticdump. This will then be deployed to badzilla.co.uk - the static html into the traditional /var/www/html structure on the prod server, and the elastic index imported into the Elasticsearch server I will have on prod. Clearly locking down the Elasticsearch instance on prod will be crucial - and we'll cover that later.

Ok - so that's the theory. Let's see how easy (or otherwise) it is in practice.

Drupal configuration - Add and enable the modules

Let's make a start. Firstly we need to add and enable the two modules we'll be using - the Elasticsearch connector and the Search API.

$ composer require drupal/search_api
<-snipped->
$ composer require drupal/elasticsearch_connector:^6.0-alpha1 
<-snipped->
$ drush en search_api elasticsearch_connector -y

Drupal configuration - Add an Elasticsearch cluster

Navigate to admin/config/search/elasticsearch-connector/cluster/add to create a cluster. I have called mine the arbitrary BadzillaStatic and note I have pointed it to my local Elasticsearch instance on my sandbox with a url of http://localhost:9200

Drupal configuration - add a Search API server

Now we have to add a Search API server - this is achieved by navigating to admin/config/search/search-api/add-server. I have added the name DrupalStatic, and I also checked that the cluster and the backend were selected correctly. Once I submitted this form, I got an encouraging notification page suggesting everything went well.

Drupal Configuration: Add index

Navigate to admin/config/search/search-api/add-index to create a search index. Whilst the screenshot will be small, it is quite easy and intuitive to populate. Add an index name - I went for StaticContent. Then I ticked the 'Content' checkbox since that is all I want for my search. Further down I selected my content types. Many in my list are artefacts from my original D6 blog and not required any more. I checked English language, made sure the radio button for the search API server DrupalStatic was selected, and also made sure the Enabled checkbox is ticked. Finally I checked 'Index items immediately'. This is imperative for me since I don't want to wait for a cron run once I've create or updated content in my sandbox.

Next fields have to be added to the index. This is where we need to be careful. For instance what I want for the 'authored by' field is not the default value - the uid - but the user name instead. so when picking the fields, it is necessary to expand the entries to ensure no ids are picked instead of the actual value. Don't forget we won't have views or any other backend preprocessor at our disposal and therefore we need real values and not primary keys.

Next is the field management screen - this gives a good overview of what you have selected, and providing the fields are fulltext they can be boosted to increase their search precedence. I have done this for title, terms and body text although I reserve the right to loop back and change these values! Also note that in most instances I have changed the machine names to something more applicable. This will pay dividends later when we are dealing with data fetched by the client in the app's frontend.

By clicking on the Processors tab there is the opportunity to fine tune the index. I won't go into these in any great detail since they are purely personal preference and the supporting text on the Processors page is self-explanatory. I elected for Entity Status, HTML Filter and Transliteration.

Finally we are ready for indexing. This can be achieved through cron or by using UI and indexing the lot in one go by setting the parameters at the bottom of the screen. Note this doesn't mean the state of the index will be reflected on the server - double check everything has completed by referring to the ringed message in the screenshot above.

Checking the search works

If you are using Kibana, or even better Kibana in my BadzillaVM, then you can use this excellent tool to check the search is working. Point a browser to {your_ip}:5601 and then navigate to Dev Tools and apply the following query:

GET /elasticsearch_index_badzilla_meedjum_staticcontent/_search
{
  "query": {
    "match_all": {}
  },
  "size": 5
}

and as per the screenshot above, you should see the results output. Note that your index in the query will be different.

blog terms

Static Site Drupal Drupal 8

Drupal 8 as a Static Site: Add feed.rss to web server config and views

nigel — Tue, 13 Nov 2018 14:16:59 +0000

Drupal 8 as a Static Site: Add feed.rss to web server config and views nigel Tue, 13/11/2018 - 14:16

Fix Views RSS feeds paths

My objective here is to ensure that my existing /taxonomy/term/%/feed and /planet/feed links still work correctly so I won't lose any RSS feed subscribers to my website. As I have mentioned previously, Tome will automatically create an index.html page when there is no file extension on a path. So it's best to add extensions then! Navigate to /admin/structure/views and edit Taxonomy term then click the feed display. Click on the path and edit according to my first screenshot above and save. I also have a Drupal Planet feed which acts on my frontpage but with tag filters, so I repeated those steps and added a feed.rss path to that too.

We aren't done yet though - my web server config needs to be changed too so that the RSS pages can be served up correctly.

nginx configuration

# Default server configuration
#
server {
        listen 80 default_server;
        listen [::]:80 default_server;
 
        # SSL configuration
 
        root {{ path_to_your_docroot }};
 
        server_name {{ your_server_name }};
 
        location ~ (/planet/feed|/taxonomy/term/\d+/feed) {
                index feed.rss;
        }
 
        location / {
 
                index index.html;
 
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }
 
        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        location ~ /\.ht {
               deny all;
        }
}

Note that I have added a new location region in the configuration specifically for the feed.rss files. Here we are saying if the path is either /planet/rss or /taxonomy/feed/%/feed then ensure that feed.rss files are served and not index.html files.

Apache configuration

<VirtualHost *:80>
# change this to 443 is you are using ssl
 
  <Directory "{{ path_to_your_docroot }}/planet/feed">
	DirectoryIndex feed.rss
  Directory>
  <Directory ~ "{{ path_to_your_docroot }}/taxonomy/term/[0-9]+/feed">
	DirectoryIndex feed.rss
  Directory>
  ServerName {{ your_server_name }}
  DocumentRoot "{{ path_to_your_docroot }}"
  DirectoryIndex index.html
 
 
  # Comment out the block below for SSL
  #SSLEngine on
  #SSLCertificateFile "/path/to/www.example.com.cert"
  #SSLCertificateKeyFile "/path/to/www.example.com.key"
 
  <Directory "{{ path_to_your_docroot }}">
      AllowOverride All
  Directory>
 
  # Logs
  Errorlog ${APACHE_LOG_DIR}/static.error.log
  CustomLog ${APACHE_LOG_DIR}/static.access.log combined
 
VirtualHost>

The Apache configuration uses two new Directory directives. The second one contains a wildcard tilde and a regex to ensure that the path /taxonomy/term/%/feed is limited to numerics only. However I had another configuration problem - my Apache server was unaware of the .xml and .rss filename extensions, so wouldn't serve the content up. This was fixed by adding a filetype to the Apache configuration.
mods-available/mime.conf

AddType application/xml .xml .rss

Success!

The picture above provides the evidence the fixes worked!

blog terms

Static Site Drupal 8 Drupal

Drupal 8

Drupal 8: Move Deprecated drupal/drupal Composer Template to drupal-composer/drupal-project

Drupal 8 as a Static Site: Drupal 8 Contact Form Thank You Page

Status message

Drupal 8 as a Static Site: AWS API Gateway, Lambda and SES Form Processing

'.$fields['subject'][0]['value'].'

'.'From: '.$fields['name'].' .$fields['mail'].'">'.$fields['mail'].''.'

Drupal 8 as a Static Site: Drupal 8 Contact Form

Drupal 8 as a Static Site: Contact Form Architecture

Drupal 8 as a Static Site: JavaScript Search Client

" + Drupal.t("Your search yielded no results.") + "

" + Drupal.t("Search results") + "

"; var suffix = "

Drupal 8 as a Static Site: Elasticsearch Query

Drupal 8 as a Static Site: Disable Core Search

Drupal 8 as a Static Site: Search Architecture and Drupal Configuration

Drupal 8 as a Static Site: Add feed.rss to web server config and views