Linux

Drupal 8: Move Deprecated drupal/drupal Composer Template to drupal-composer/drupal-project

nigel — Wed, 11 Dec 2019 15:52:32 +0000

Drupal 8: Move Deprecated drupal/drupal Composer Template to drupal-composer/drupal-project nigel Wed, 11/12/2019 - 15:52

Tutorial for moving an existing Drupal 8 codebase away from the deprecated drupal/drupal composer template using the utility GoComposer. This will move the project to the latest Fully Composer Managed template. This will also align the codebase with Drupal 9 reducing any effort required to ensure full Drupal 9 compatibility on day 1 of Drupal 9 release.

This solution will apply to many Drupal 8 codebases, including my own Badzilla site. When I migrated Badzilla from D6, the drupal/drupal composer template appeared to be the correct choice. Sadly experience has proven otherwise, and all Drupal 8 minor versions have caused serious headaches when updating.

So I will be writing this blog from the perspective of moving composer templates on Badzilla.

Getting Started

I will be doing this in a development environment. I am using my own BadzillaVM development environment. Back up both the database and the codebase artefact on prod anyway in preparation for the deploy later.

My first activity is to change directory in my dev environment and create a new git branch dedicated to this project.

$ git checkout -b feature/composer-template

Now let's see what our starting point by listing my existing composer.json file.

$ cd docroot
$ cat composer.json
{
    "name": "drupal/drupal",
    "description": "Drupal is an open source content management platform powering millions of websites and applications.",
    "type": "project",
    "license": "GPL-2.0+",
    "require": {
        "composer/installers": "^1.0.24",
        "cweagans/composer-patches": "~1.0",
        "wikimedia/composer-merge-plugin": "~1.4",
        "drupal/core": "8.6.2",
        "drupal/bootstrap": "^3.9",
        "drupal/memcache": "^2.0@alpha",
        "drupal/module_filter": "^3.0",
        "drupal/pathauto": "^1.0@RC",
        "drupal/devel": "^1.0@RC",
        "drupal/admin_toolbar": "^1.19",
        "drupal/config_inspector": "^1.0@beta",
        "drupal/google_analytics": "^2.1",
        "drupal/metatag": "^1.2",
        "drupal/addtoany": "^1.8",
        "drupal/migrate_tools": "^4.0@beta",
        "drupal/migrate_plus": "^4.0@beta",
        "drupal/migrate_upgrade": "^3.0",
        "drupal/migrate_manifest": "^1.5",
        "drupal/config_update": "^1.3",
        "drupal/paragraphs": "^1.1",
        "drupal/geshifilter": "^1.1",
        "drupal/xmlsitemap": "^1.0@alpha",
        "drupal/youtube": "^1.0@beta",
        "drupal/fontawesome": "^2.0",
        "drupal/config_filter": "^1.0",
        "drupal/config_split": "^1.2",
        "drupal/schema_metatag": "^1.0-rc4",
        "phpdocumentor/reflection-docblock": "^2.0",
        "drush/drush": "9.*",
        "drupal/libraries": "^3.0@alpha",
        "phing/phing": "^2.16",
        "drupal/tome": "^1.0@alpha",
        "drupal-tome/tome_drush": "dev-master",
        "drupal/search_api": "^1.10",
        "drupal/elasticsearch_connector": "^6.0-alpha1"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "platform": {
            "php": "7.0.32"
        },
        "preferred-install": "dist",
        "autoloader-suffix": "Drupal8"
    },
    "extra": {
        "_readme": [
            "By default Drupal loads the autoloader from ./vendor/autoload.php.",
            "To change the autoloader you can edit ./autoload.php.",
            "This file specifies the packages.drupal.org repository.",
            "You can read more about this composer repository at:",
            "https://www.drupal.org/node/2718229"
        ],
        "merge-plugin": {
            "include": [
                "core/composer.json"
            ],
            "recurse": false,
            "replace": false,
            "merge-extra": false
        },
        "installer-paths": {
            "core": ["type:drupal-core"],
            "modules/contrib/{$name}": ["type:drupal-module"],
            "profiles/contrib/{$name}": ["type:drupal-profile"],
            "themes/contrib/{$name}": ["type:drupal-theme"],
            "drush/contrib/{$name}": ["type:drupal-drush"],
            "modules/custom/{$name}": ["type:drupal-custom-module"],
            "themes/custom/{$name}": ["type:drupal-custom-theme"],
            "libraries/{$name}": ["type:drupal-library"]
        },
        "patches": {
            "drupal/core": {
                "rdf: Fatal error: Call to a member function url() on null": "https://www.drupal.org/files/issues/member-function-url-fix-2565247-4.patch"
            },
            "modules/contrib/module_filter": {
                "Module Filter issues notices after extend list changes": "https://www.drupal.org/files/issues/module_filter-undefined_index_recent_modules_submit-2857431-16.patch"
            }
        }  
    },
    "autoload": {
        "psr-4": {
            "Drupal\\Core\\Composer\\": "core/lib/Drupal/Core/Composer"
        }
    },
    "scripts": {
        "pre-autoload-dump": "Drupal\\Core\\Composer\\Composer::preAutoloadDump",
        "post-autoload-dump": [
          "Drupal\\Core\\Composer\\Composer::ensureHtaccess"
        ],
        "post-package-install": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",
        "post-package-update": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup"
    },
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ]
}

Ok so I'm a little behind on my Drupal updates with core currently on 8.6.2, and I'm using PHP 7.0 on both sandbox and prod which is ok for now but needs to be updated soon. I've also got a couple of patches which may be resolved already as I go through this exercise.

GoComposer

To run the GoComposer code I need to be in the top level directory of the codebase, i.e. the directory that contains my .git directory. Ensure this is the case then create a skeleton composer file for GoComposer

$ cd ..
$ composer require webkings-ca/gocomposer:dev-master
You are running composer with xdebug enabled. This has a major impact on runtime performance. See getcomposer.org/xdebug
./composer.json has been created
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 6 installs, 0 updates, 0 removals
  - Installing webkings-ca/gocomposer (dev-master 310fa76): Cloning 310fa76ed0 from cache
  - Installing symfony/polyfill-ctype (v1.13.1): Downloading (100%)         
  - Installing webmozart/assert (1.6.0): Downloading (100%)         
  - Installing webmozart/path-util (2.3.0): Loading from cache
  - Installing webflo/drupal-finder (1.2.0): Downloading (100%)         
  - Installing symfony/yaml (v3.4.36): Downloading (100%)         
symfony/yaml suggests installing symfony/console (For validating YAML files using the lint command)
Writing lock file
Generating autoload files
PHP 7.0.33-13+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Nov 28 2019 07:43:06) ( NTS )

Now I am ready to run gocomposer

$ composer gocomposer
You are running composer with xdebug enabled. This has a major impact on runtime performance. See getcomposer.org/xdebug
 
GoComposer is Initializing...
=============================
 
 
 Please Confirm: Composer.json file will be Created in /var/www/html/meedjum-composer/docroot (Recommended) (yes/no) [yes]:
 > 
 
 Please Confirm: Your Original Site will be Backed up to the following directory:  /var/www/html/meedjum-composer/docroot/backup (Recommended) (yes/no) [yes]:
 > 
 
 Please select which Settings.php file you want to use for your Environment (Recommended Version is: [0]) 
  [0] /var/www/html/meedjum-composer/docroot/sites/default/settings.php
 
 > 
 You have just selected: /var/www/html/meedjum-composer/docroot/sites/default/settings.php
 
 
 
  [ErrorException]                                               
  Use of undefined constant DRUPAL_ROOT - assumed 'DRUPAL_ROOT'  
 
gocomposer

Problem 1: DRUPAL_ROOT not defined

An exception was thrown soon after the process began - Use of undefined constant DRUPAL_ROOT - assumed 'DRUPAL_ROOT'which was thrown in my settings.local.php file on the following line:

$settings['container_yamls'][] = DRUPAL_ROOT . '/sites/default/local.services.yml';
?>

Now that's weird since it's saying that DRUPAL_ROOT isn't been set during the Drupal bootstrap process. So I changed it to $app_root which worked.

Problem 2: Removing drupal_coder

I then started the process again and get further, but was presented with

  - Removing drupal/coder (8.3.1)
    The package has modified files:
    D coder_sniffer/Drupal/Test/Arrays/ArrayUnitTest.inc
    D coder_sniffer/Drupal/Test/Arrays/ArrayUnitTest.inc.fixed
    D coder_sniffer/Drupal/Test/Arrays/ArrayUnitTest.php
    D coder_sniffer/Drupal/Test/Arrays/DisallowLongArraySyntaxUnitTest.php
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d7/DisallowLongArraySyntaxUnitTest.1.inc
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d7/disallow_long_array_d7.info
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d8/DisallowLongArraySyntaxUnitTest.2.inc
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d8/DisallowLongArraySyntaxUnitTest.2.inc.fixed
    D coder_sniffer/Drupal/Test/Arrays/disallow_long_array_d8/disallow_long_array_d8.info.yml
    D coder_sniffer/Drupal/Test/Classes/ClassCreateInstanceUnitTest.inc
    240 more files modified, choose "v" to view the full list
    Discard changes [y,n,v,d,?]? ?
    y - discard changes and apply the uninstall
    n - abort the uninstall and let you manually clean things up
    v - view modified files
    d - view local modifications (diff)
    ? - print help
    Discard changes [y,n,v,d,?]?

This is one of those occasions when none of the options are what I wanted, but I elected to uninstall drupal_coder

Problem 3: Config directory in the wrong place

The process continued and once it starts the database work, go for a cup of tea because it takes a lot of time. I got the following diagnostic part way

Current Step: Using Drush to update the Database... Press Enter to Continue...
------------------------------------------------------------------------------
 
 [error]  The directory <em class="placeholder">../config/syncem> does not exist. 
 [error]  The directory <em class="placeholder">../config/split/prodem> does not exist. 
 [error]  The directory <em class="placeholder">../config/split/sandboxem> does not exist.

The config directory is in the wrong place which surprised me so I elected to let the process complete and then I would investigate

GoComposer Completes

  views               make_place   post-update     Rebuild cache to allow       
                      holders_tr                   placeholder texts to be      
                      anslatable                   translatable.                
  views               remove_cor   post-update     Remove core key from views   
                      e_key                        configuration.               
 ------------------- ------------ --------------- ----------------------------- 
 
 Do you wish to run the specified pending updates? (yes/no) [yes]:
 > 
 
 
 [OK] Congrats!... You have Successfully updated your Site... The old site files and Database Sql dump are saved in the 
      backup folder                                                                                                     
 
 
 
 ! [NOTE] Your New Docroot is in the newly created /web directory... Dont forget to update you vhosts file by adding    
 !        /web to the site path...

The process finally completed, which included updated Drupal and all my modules to the latest version - you can just see in the top part of the listing above it is finishing the module updates. Now I investigated the problem with the sync directory and I was astounded to discover that the new web directory is actually a child of docroot! Say what? I assumed that docroot would be replaced by web, but instead it's created a complete subdirectory structure underneath it! That explains why the config directory is wrong in the settings.php file.

Fix Problem 3: Directory structure incorrect

GoComposer left me with a directory structure of

/var/www/html/meedjum-composer/docroot/web

So web needs moving to be a child of the the top level directory (meedjum-composer) whilst the files and directories of docroot need moving into meedjum-composer. First steps I need to clear down the top level directory.

$ pwd
/var/www/html/meedjum-composer
$ rm -r composer.*  #old stuff 
$ rm -rf vendor  #old stuff
$ rm -rf drush   #old stuff

Next copy the web directory in place and delete the obsolete docroot directory

$ cp -vaR docroot/. .
$ rm -rf docroot

Did it Work?

Before the solution can be tested, the web server docroot will need to be changed. This will vary dependent upon your web server (usually nginx or Apache2) and that server's configuration so it isn't covered here. Mine is simple because I am using the BadzillaVM so I changed my docroot setting in my Ansible playbook to point to /var/www/html/meedjum-composer/web and re-provisioned my virtual box.

I then successfully logged into my sandbox version of Badzilla and navigated to admin/reports/updates. I now have a Drupal site running core 8.8.0, and I have successfully moved away from the deprecated drupa/drupal composer template.

However the process didn't update the Admin Toolbar as you can see. So lets have a look at our new composer,json file and perform the update.

Update Admin Toolbar

The new composer.json file looks like:

{
    "name": "drupal-composer/drupal-project",
    "description": "Project template for Drupal 8 projects with composer",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "authors": [
        {
            "name": "",
            "role": ""
        }
    ],
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ],
    "require": {
        "php": ">=7.0.8",
        "composer/installers": "^1.2",
        "cweagans/composer-patches": "^1.6.5",
        "drupal/console": "^1.0.2",
        "drupal/core": "^8.6.2",
        "drupal/core-composer-scaffold": "^8.8.0",
        "drush/drush": "^9.7.1 | ^10.0.0",
        "vlucas/phpdotenv": "^4.0",
        "webflo/drupal-finder": "^1.0.0",
        "zaporylie/composer-drupal-optimizations": "^1.0",
        "drupal/token": "^1.5.0",
        "drupal/devel": "^1.2.0",
        "drupal/tome": "^1.0.0-alpha2",
        "drupal/schema_metatag": "^1.3.0",
        "drupal/addtoany": "^1.10.0",
        "drupal/ctools": "^3.0.0",
        "drupal/fontawesome": "^2.8.0",
        "drupal/config_update": "^1.5.0",
        "drupal/google_analytics": "^2.3.0",
        "drupal/search_api": "^1.10.0",
        "drupal/elasticsearch_connector": "^6.0.0-alpha1",
        "drupal/migrate_upgrade": "^3.0.0-rc5",
        "drupal/geshifilter": "^1.2.0",
        "drupal/libraries": "^3.0.0-alpha1",
        "drupal/migrate_plus": "^4.0.0",
        "drupal/config_split": "^1.4.0",
        "drupal/pathauto": "^1.3.0",
        "drupal/youtube": "^1.0.0-beta3",
        "drupal/migrate_tools": "^4.0.0",
        "drupal/admin_toolbar": "^1.24.0",
        "drupal/metatag": "^1.7.0",
        "drupal/config_inspector": "^1.0.0-beta2",
        "drupal/memcache": "^2.0.0-rc2",
        "drupal/migrate_manifest": "^1.7.0",
        "drupal/module_filter": "^3.1.0",
        "drupal/config_filter": "^1.3.0",
        "drupal/entity_reference_revisions": "^1.6.0",
        "drupal/paragraphs": "^1.3.0",
        "drupal/xmlsitemap": "^1.0.0-alpha3",
        "drupal/bootstrap": "^3.13.0"
    },
    "require-dev": {
        "drupal/core-dev": "^8.8.0"
    },
    "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "sort-packages": true
    },
    "autoload": {
        "classmap": [
            "scripts/composer/ScriptHandler.php"
        ],
        "files": [
            "load.environment.php"
        ]
    },
    "scripts": {
        "pre-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "pre-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "post-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ],
        "post-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ]
    },
    "extra": {
        "composer-exit-on-patch-failure": true,
        "patchLevel": {
            "drupal/core": "-p2"
        },
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            }
        },
        "installer-paths": {
            "web/core": [
                "type:drupal-core"
            ],
            "web/libraries/{$name}": [
                "type:drupal-library"
            ],
            "web/modules/contrib/{$name}": [
                "type:drupal-module"
            ],
            "web/profiles/contrib/{$name}": [
                "type:drupal-profile"
            ],
            "web/themes/contrib/{$name}": [
                "type:drupal-theme"
            ],
            "drush/Commands/contrib/{$name}": [
                "type:drupal-drush"
            ]
        }
    }
}

The "conflict" entry made me smile - I haven't seen that kind of entry before! But the focus is admin_toolbar and it can be seen that the reason it wasn't updated is because it has moved up a major version and the tilde on the specified version prohibited such a large jump. Let's fix this.

$ composer require drupal/admin_toolbar:2.0
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
./composer.json has been updated
> DrupalProject\composer\ScriptHandler::checkComposerVersion
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 0 installs, 1 update, 0 removals
  - Updating drupal/admin_toolbar (1.27.0 => 2.0.0): Downloading (100%)         
Package container-interop/container-interop is abandoned, you should avoid using it. Use psr/container instead.
Package phpunit/phpunit-mock-objects is abandoned, you should avoid using it. No replacement was suggested.
Writing lock file
Generating autoload files
Scaffolding files for drupal/core:
  - Copy [project-root]/.editorconfig from assets/scaffold/files/editorconfig
  - Copy [project-root]/.gitattributes from assets/scaffold/files/gitattributes
  - Copy [web-root]/.csslintrc from assets/scaffold/files/csslintrc
  - Copy [web-root]/.eslintignore from assets/scaffold/files/eslintignore
  - Copy [web-root]/.eslintrc.json from assets/scaffold/files/eslintrc.json
  - Copy [web-root]/.ht.router.php from assets/scaffold/files/ht.router.php
  - Copy [web-root]/.htaccess from assets/scaffold/files/htaccess
  - Copy [web-root]/example.gitignore from assets/scaffold/files/example.gitignore
  - Copy [web-root]/index.php from assets/scaffold/files/index.php
  - Copy [web-root]/INSTALL.txt from assets/scaffold/files/drupal.INSTALL.txt
  - Copy [web-root]/README.txt from assets/scaffold/files/drupal.README.txt
  - Copy [web-root]/robots.txt from assets/scaffold/files/robots.txt
  - Copy [web-root]/update.php from assets/scaffold/files/update.php
  - Copy [web-root]/web.config from assets/scaffold/files/web.config
  - Copy [web-root]/sites/README.txt from assets/scaffold/files/sites.README.txt
  - Copy [web-root]/sites/development.services.yml from assets/scaffold/files/development.services.yml
  - Copy [web-root]/sites/example.settings.local.php from assets/scaffold/files/example.settings.local.php
  - Copy [web-root]/sites/example.sites.php from assets/scaffold/files/example.sites.php
  - Copy [web-root]/sites/default/default.services.yml from assets/scaffold/files/default.services.yml
  - Copy [web-root]/sites/default/default.settings.php from assets/scaffold/files/default.settings.php
  - Copy [web-root]/modules/README.txt from assets/scaffold/files/modules.README.txt
  - Copy [web-root]/profiles/README.txt from assets/scaffold/files/profiles.README.txt
  - Copy [web-root]/themes/README.txt from assets/scaffold/files/themes.README.txt
> DrupalProject\composer\ScriptHandler::createRequiredFiles
PHP 7.0.33-13+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Nov 28 2019 07:43:06) ( NTS )
$ drush updb
 [success] No pending updates.
PHP 7.0.33-13+ubuntu16.04.1+deb.s

Ok so no schema changes required, and if I now point a browser at admin/reports/updates - everything looks ok as can be seen in the screenshot above.

Pin the Composer Dependencies!!!

I've always said that PHP deserves a better dependency management tool than Composer, but alas we are stuck with it. To limit the possibility (likelihood?) of disaster, you should always pin the version numbers for each Drupal project dependency. Pinning means setting a contrib module or theme or profile at a particular version so it cannot be accidentally updated on a composer update command.

So basically I will go through the composer.json file and remove all the tilde or carat symbols, and ensure the version is set to the current version on the system.

{
    "name": "drupal-composer/drupal-project",
    "description": "Project template for Drupal 8 projects with composer",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "authors": [
        {
            "name": "",
            "role": ""
        }
    ],
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ],
    "require": {
        "php": ">=7.0.8",
        "composer/installers": "^1.2",
        "cweagans/composer-patches": "^1.6.5",
        "drupal/addtoany": "1.12",
        "drupal/admin_toolbar": "2.0",
        "drupal/bootstrap": "3.21",
        "drupal/config_filter": "1.5",
        "drupal/config_inspector": "1.0",
        "drupal/config_split": "1.4",
        "drupal/config_update": "1.5",
        "drupal/console": "1.0.2",
        "drupal/core": "8.8.0",
        "drupal/core-composer-scaffold": "8.8.0",
        "drupal/ctools": "3.2",
        "drupal/entity_reference_revisions": "1.7",
        "drupal/fontawesome": "2.14",
        "drupal/geshifilter": "1.2",
        "drupal/google_analytics": "2.4",
        "drupal/libraries": "^3.0.0-alpha1",
        "drupal/memcache": "2.0",
        "drupal/metatag": "1.10",
        "drupal/module_filter": "3.1",
        "drupal/paragraphs": "1.10",
        "drupal/pathauto": "1.6",
        "drupal/schema_metatag": "1.4",
        "drupal/token": "1.5",
        "drupal/xmlsitemap": "1.0.0-alpha4",
        "drupal/youtube": "1.0",
        "drush/drush": "^9.7.1 | ^10.0.0",
        "vlucas/phpdotenv": "^4.0",
        "webflo/drupal-finder": "^1.0.0",
        "zaporylie/composer-drupal-optimizations": "^1.0"
    },
    "require-dev": {
        "drupal/core-dev": "^8.8.0"
    },
    "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "sort-packages": true
    },
    "autoload": {
        "classmap": [
            "scripts/composer/ScriptHandler.php"
        ],
        "files": [
            "load.environment.php"
        ]
    },
    "scripts": {
        "pre-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "pre-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "post-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ],
        "post-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ]
    },
    "extra": {
        "composer-exit-on-patch-failure": true,
        "patchLevel": {
            "drupal/core": "-p2"
        },
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            }
        },
        "installer-paths": {
            "web/core": [
                "type:drupal-core"
            ],
            "web/libraries/{$name}": [
                "type:drupal-library"
            ],
            "web/modules/contrib/{$name}": [
                "type:drupal-module"
            ],
            "web/profiles/contrib/{$name}": [
                "type:drupal-profile"
            ],
            "web/themes/contrib/{$name}": [
                "type:drupal-theme"
            ],
            "drush/Commands/contrib/{$name}": [
                "type:drupal-drush"
            ]
        }
    }
}

The composer.json file has now been modified and the drupal project files are pinned at particular versions for safety.

blog terms

Drupal 8 Development PHP Linux

Grow Mounted Partition of Headless VirtualBox

nigel — Sat, 29 Sep 2018 07:55:57 +0000

Grow Mounted Partition of Headless VirtualBox nigel Sat, 29/09/2018 - 08:55

This blog provides a tutorial for resizing (upwards) a partition on a headless VirtualBox Ubuntu Linux box. The task is accomplished in a few steps.

Clone the disk drive into VHD format
Increase the disk size of the clone
Attached the newly cloned drive into the VM
Increase the partition inside the VM

Ok so let's get started and see what we've got and where we need to get to.

$ VBoxManage list runningvms
"badzilla-d8" {86af2c61-3b68-47c5-a71e-8f45ac196cbe}
"Plex Media Server" {5efa7e7b-ae92-45a9-8244-bae451b53c0a}

It's my Plex server that has run out of disk space. I totally underestimated how much room Plex actually needs for its metadata. I'm going to extend the disk from 10GB to 30GB which should keep me going for a few years.

Clone the disk drive into VHD format

We have to clone for two reasons. Firstly when I created the disk in the first place, I created it as a fixed 10GB and it needs to be dynamic, and secondly only .VDI or .VHD format drives can be grown. Let's get our disk details.

$ VBoxManage showvminfo "Plex Media Server" | grep -i disk
Boot Device (3): HardDisk
SATA (0, 0): /home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vmdk (UUID: 0956c89c-b496-4b2e-985f-2f2d484f6e3a)

Now we know the name of our disk we can clone

$ sudo VBoxManage clonehd --format vhd "/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vmdk" "/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd"
[sudo] password for nigel: 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'vhd'. UUID: e4770417-731a-466c-9124-d1c97625cf7b

Now let's see what we have

$ ls -lash "/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vmdk" "/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd"
6.1G -rw------- 1 root  root  6.1G Sep 29 12:24 /home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd
6.0G -rw------- 1 nigel nigel 6.0G Sep 29 12:42 /home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vmdk

You can now see the two disk drives - the vmdk and the vhd format drives.

Increase the Disk Size of the VHD Clone

Now we can resize the disk to the required 30GB

$ sudo VBoxManage modifymedium '/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd' --resize 30000
[sudo] password for nigel: 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Attached the newly cloned drive into the VM

Better power off the original VM. In fact I should have done this earlier. Oops.

$ VBoxManage controlvm "Plex Media Server" poweroff

Now we can switchero the disks around with the following command.

$ VBoxManage storageattach 'Plex Media Server' \
> --storagectl "SATA" \
> --device 0 \
> --port 0 \
> --type hdd \
> --medium  '/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd'

Did it work?

$ VBoxManage showvminfo "Plex Media Server" | grep -i disk
Boot Device (3): HardDisk
SATA (0, 0): /home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd (UUID: e4770417-731a-466c-9124-d1c97625cf7b)

Looks encouraging so we'll start it up

$ nohup VBoxHeadless --startvm "Plex Media Server" &

and login in and check the running processes..

$ ssh nigel@192.168.0.203
nigel@192.168.0.203's password: 
$ ps -ef | grep -i plex
plex       976     1  0 15:18 ?        00:00:00 /bin/sh -c LD_LIBRARY_PATH=/usr/lib/plexmediaserver "/usr/lib/plexmediaserver/Plex Media Server"
plex       982   976  4 15:18 ?        00:00:03 /usr/lib/plexmediaserver/Plex Media Server
plex      1116   982 13 15:18 ?        00:00:08 Plex Plug-in [com.plexapp.system] /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.11.3.4803-c40bba82e /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/System.bundle
plex      1164   982  1 15:18 ?        00:00:00 /usr/lib/plexmediaserver/Plex DLNA Server
plex      1166   982  0 15:18 ?        00:00:00 /usr/lib/plexmediaserver/Plex Tuner Service /usr/lib/plexmediaserver/Resources/Tuner/Private /usr/lib/plexmediaserver/Resources/Tuner/Shared 1.11.3.4803-c40bba82e 32600 /waitmutex
plex      1213   982  8 15:18 ?        00:00:04 Plex Plug-in [com.plexapp.agents.imdb] /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.11.3.4803-c40bba82e /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/PlexMovie.bundle
nigel     1354  1325  0 15:19 pts/0    00:00:00 grep --color=auto -i plex

Increase the Partition Size Inside the VM

If we look at the file system storage space it is reporting our system drive is at 100% capacity. We'd expect that - we need to extend it to use our new space.

$ df -k | grep sda1
/dev/sda1                   6061632    5724660       6016 100% /

Now lets see what the situation is per partition

$ sudo fdisk /dev/sda
 
Welcome to fdisk (util-linux 2.27.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
 
 
Command (m for help): p
Disk /dev/sda: 29.3 GiB, 31457280000 bytes, 61440000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0660013b
 
Device     Boot    Start      End  Sectors Size Id Type
/dev/sda1  *        2048 12584959 12582912   6G 83 Linux
/dev/sda2       12587006 20969471  8382466   4G  5 Extended
/dev/sda5       12587008 20969471  8382464   4G 82 Linux swap / Solaris

Now let's see where our free space is

Command (m for help): F
Unpartitioned space /dev/sda: 19.3 GiB, 20720910336 bytes, 40470528 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
 
   Start      End  Sectors  Size
20969472 61439999 40470528 19.3G

That makes life a little difficult because there is a swap partition of 4GB immediately after our Linux 6GB. We are going to have to delete the swap, recreate it between 26GB -> 30GB and then grow our 6GB Linux partition to 26GB. Whilst this would normally be extremely dangerous, the good news is if we make a fatal mistake, we can start the process over since we still have the original vmdk drive. So let's go for it!

Firstly we delete the swap space and the extended partition

Command (m for help): d
Partition number (1,2,5, default 5): 5
 
Partition 5 has been deleted.
 
Command (m for help): d
Partition number (1,2, default 2): 2
 
Partition 2 has been deleted.

Now we are going to delete the Linux partition. What? Well deleting the partition won't delete the data providing we recreate our new partition from the very same sector on the disk.

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.
 
Command (m for help): F
Unpartitioned space /dev/sda: 29.3 GiB, 31456231424 bytes, 61437952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
 
Start      End  Sectors  Size
 2048 61439999 61437952 29.3G

Now recreate the partition only bigger!

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-61439999, default 2048): 
Last sector, +sectors or +size{K,M,G,T,P} (2048-61439999, default 61439999): 53055486
 
Created a new partition 1 of type 'Linux' and of size 25.3 GiB.

Next we recreate the extended partition with the remaining space and a new logical partition which will ultimately be the Linux Swap area

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): e
Partition number (2-4, default 2): 2
First sector (53055487-61439999, default 53055488): 
Last sector, +sectors or +size{K,M,G,T,P} (53055488-61439999, default 61439999): 
 
Created a new partition 2 of type 'Extended' and of size 4 GiB.
 
Command (m for help): n
All space for primary partitions is in use.
Adding logical partition 5
First sector (53057536-61439999, default 53057536): 
Last sector, +sectors or +size{K,M,G,T,P} (53057536-61439999, default 61439999): 
 
Created a new partition 5 of type 'Linux' and of size 4 GiB.

Now change the last partition type to Linux Swap

Command (m for help): t
Partition number (1,2,5, default 5): 
Partition type (type L to list all types): 82
 
Changed type of partition 'Linux' to 'Linux swap / Solaris'.

And finally write the partition table

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy
 
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

So we need to reboot for our changes to take effect

$ sudo su
[sudo] password for nigel: 
root@plex-media-server:/home/nigel# init 6

When we log in there is good and bad news: the filesystem isn't recognising the new size but fdisk does.

$ df -k | grep sda
/dev/sda1                   6061632    5667620      63056  99% /

and

$ sudo fdisk /dev/sda
 
Welcome to fdisk (util-linux 2.27.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
 
 
Command (m for help): p
Disk /dev/sda: 29.3 GiB, 31457280000 bytes, 61440000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0660013b
 
Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1           2048 53055486 53053439 25.3G 83 Linux
/dev/sda2       53055488 61439999  8384512    4G  5 Extended
/dev/sda5       53057536 61439999  8382464    4G 82 Linux swap / Solaris

Our final activity to make the additional space available for us is to use the resize2fs command:

$ sudo resize2fs /dev/sda1
resize2fs 1.42.13 (17-May-2015)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 2
The filesystem on /dev/sda1 is now 6631679 (4k) blocks long.

And now we have it:

$ df -k | grep sda
/dev/sda1                  25978144    5672208   19165592  23% /

Cleaning-up Activities

Since we deleted the swap partition, and moved it, we need to create the actual swap with the following command

$ sudo mkswap /dev/sda5

In addition, we could either delete the original vmdk disk or archive it somewhere to get 6GB space back on the parent machine.

blog terms

Hardware Linux VirtualBox

Move Plex Media Server to Linux VM

nigel — Sun, 18 Feb 2018 13:45:09 +0000

Move Plex Media Server to Linux VM nigel Sun, 18/02/2018 - 13:45

I've been playing with Plex Media Server for the last few weeks on my Linux laptop, and much to my surprise it seems to play back any video format I throw at it including DivX and XviD which is the format of a great many of my collection of movies. So buoyed by this I have decided to move the entire Plex configuration to a dedicated Ubuntu virtual machine which will reside on my Intel NUC I reviewed earlier.

My Intel NUC is headless, and whilst it is possible to create a VM image on the command line, it is much more convenient to create it on a GUI elsewhere and move it over to the NUC. To that end, I created the VM on my Mac knowing the network configuration would ultimately have to change when the VM is in situ on the NUC.

VM Installation

The steps are quite straightforward so there isn't much need for narrative. I had previously downloaded the Ubuntu 16.04 LTS server iso image (more of that in a minute).

I decided to go for 4GB RAM and 10GB fixed disk size which should be more than adequate.

Initial Network Connection

The initial network connection is set as above. We need a bridged network so I can use my pre-existing 192.168 network in my property. Obviously when the VM is moved to the Intel NIC the AirPort won't be a valid setting at that point - so we'll have to change it then.

Ubuntu 16.04 Installation

As mentioned previously, I had already downloaded the Ubuntu 16.04 LTS iso image which is selected for the installation above. Most of the installation screens haven't been captured for brevity because they are straightforward, but note I have selected openSSH server in the software packages to install - this is important since we will need to ssh into our headless Plex server.

We now need to edit the /etc/network/interfaces file to change the default DHCP IP assignment to static. Your configuration will obviously be different from mine here. I've elected to go for the address of 192.168.0.203 for my server.

nigel@plex-media-server:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
 
source /etc/network/interfaces.d/*
 
# The loopback network interface
auto lo
iface lo inet loopback
 
# The primary network interface
auto enp0s3
# iface enp0s3 inet dhcp
iface enp0s3 inet static
 address 192.168.0.203
 netmask 255.255.255.0
 network 192.168.0.1
 broadcast 192.168.1.255
 gateway 192.168.0.1
 dns-nameservers 90.207.238.97

Export the VM

I have already covered how to export a VM image in a previous blog here, so I don't need to go into too much detail in this blog. My VM has been exported and now needs to be copied to the NUC host machine using scp.

Nigels-MacBook-Pro:Documents nigel$ scp Plex\ Media\ Server.ova nigel@192.168.0.202:/home/nigel/VM/.
nigel@192.168.0.202's password: 
Plex Media Server.ova                                                                        100%  604MB   4.1MB/s   02:27    
Nigels-MacBook-Pro:Documents nigel$

Now log into the host machine and cd into the correct directory and import the VM.

nigel@devbox:~/VM$ VBoxManage import Plex\ Media\ Server.ova 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interpreting /home/nigel/VM/Plex Media Server.ova...
OK.
Disks:
  vmdisk1       10737418240     -1      http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized       Plex Media Server-disk001.vmdk  -1      -1
 
Virtual system 0:
 0: Suggested OS type: "Ubuntu_64"
    (change with "--vsys 0 --ostype "; use "list ostypes" to list all possible values)
 1: Suggested VM name "Plex Media Server"
    (change with "--vsys 0 --vmname ")
 2: Number of CPUs: 1
    (change with "--vsys 0 --cpus ")
 3: Guest memory: 4096 MB
    (change with "--vsys 0 --memory ")
 4: Sound card (appliance expects "", can change on import)
    (disable with "--vsys 0 --unit 4 --ignore")
 5: USB controller
    (disable with "--vsys 0 --unit 5 --ignore")
 6: Network adapter: orig Bridged, config 3, extra slot=0;type=Bridged
 7: CD-ROM
    (disable with "--vsys 0 --unit 7 --ignore")
 8: IDE controller, type PIIX4
    (disable with "--vsys 0 --unit 8 --ignore")
 9: IDE controller, type PIIX4
    (disable with "--vsys 0 --unit 9 --ignore")
10: SATA controller, type AHCI
    (disable with "--vsys 0 --unit 10 --ignore")
11: Hard disk image: source image=Plex Media Server-disk001.vmdk, target path=/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vmdk, controller=10;channel=0
    (change target path with "--vsys 0 --unit 11 --disk path";
    disable with "--vsys 0 --unit 11 --ignore")
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Successfully imported the appliance.

Now we need to modify the ethernet card settings and the audio as per my previous blog

$ VBoxManage modifyvm "Plex Media Server" --nic1 bridged --bridgeadapter1 enp3s0
$ VBoxManage modifyvm "Plex Media Server" --audio none

Ok now we need to be sure we are in the correct directory and start up our machine. Once this has been done, it will be possible to ssh into it.

$ pwd
/home/nigel/VirtualBox VMs
$ VBoxHeadless --startvm "Plex Media Server" &

Let's try to log into 192.168.0.203

nigel@nigel-Aspire-VN7-793G:/etc$ ssh nigel@192.168.0.203
The authenticity of host '192.168.0.203 (192.168.0.203)' can't be established.
ECDSA key fingerprint is SHA256:BLRRmyijEFgtS+KnkXQUudkdsMskpLrzR19zBugQ2ww.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.203' (ECDSA) to the list of known hosts.
nigel@192.168.0.203's password: 
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-87-generic x86_64)
 
 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
 
127 packages can be updated.
64 updates are security updates.
 
 
Last login: Sun Feb 18 19:58:43 2018 from 192.168.0.12
nigel@plex-media-server:~$

Success!

fstab entry to connect to CIFS partition

My FreeNAS server has a CIFS mount point which contains my media collection. It would therefore be useful for the drive to be automatically loaded on boot. A prerequisite for this is the cifs-utils package which will need to be installed.

$ sudo apt-get install cifs.utils

Next we need to create the mount point directory which will be /mnt/freenas

$ sudo mkdir /mnt/freenas

It would useful to check we can connect first on the command line. So lets type out the required mount command, but first we need to create a credentials file for FreeNAS. This will be owned by the root user and should be placed in the root user's directory and made read only for root.

# cat .cifs 
username=xxxxxx
password=xxxxxx
# ls -las .cifs
4 -r-------- 1 root root 39 Feb 18 21:37 .cifs

Now the actual mount command and a follow up directory listing to make sure it has been mounted correctly.

$ sudo mount.cifs //192.168.0.200/personal  /mnt/freenas    -o vers=3.0,credentials=/root/.cifs,uid=1000
$ ls -las /mnt/freenas | grep Documents
     0 drwxr-xr-x 2 nigel root         0 Aug 14  2017 Documents

Now we need to unmount this and add the entry to the /etc/fstab file.

$ sudo umount /mnt/freenas
$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#                
# / was on /dev/sda1 during installation
UUID=600eb961-4caa-4a24-b8cf-c35efdd71ca4 /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=523ed080-41af-4dab-839f-070f40c26ada none            swap    sw              0       0
 
# CIFS FreeNas mount
//192.168.0.200/personal        /mnt/freenas    cifs    credentials=/root/.cifs,uid=1000,vers=3.0       0      0
$ sudo mount -a
$ ls -las /mnt/freenas | grep Documents
     0 drwxr-xr-x 2 nigel root         0 Aug 14  2017 Documents

That's another job done! Finally we can think about moving the Plex configuration over to our new VM.

Commence the Plex Move from Source to Destination

Firstly navigate to Settings->Server->library and disable the setting Empty trash automatically after every scan as shown above. Save this setting change.

Stop the Plex Media Server running on the source machine. This is achieved on the command line with

$ sudo service plexmediaserver stop
[sudo] password for nigel: 
$ ps -ef | grep plex
nigel     6274  1647  0 09:51 pts/1    00:00:00 grep --color=auto plex
$

Note I double checked that there were no Plex services running just in case.

install Plex on Destination Machine

In my case my destination machine is a VM on my Intel NUC so let's see if the VM is currently running.

nigel@devbox:~$ VBoxManage list vms
"badzilla-d8" {86af2c61-3b68-47c5-a71e-8f45ac196cbe}
"Plex Media Server" {5efa7e7b-ae92-45a9-8244-bae451b53c0a}
nigel@devbox:~$ VBoxManage list runningvms
nigel@devbox:~$

So the good news is it is there. The bad news is it isn't currently running. Let's spark it up.

nigel@devbox:~$ VBoxHeadless --startvm "Plex Media Server" &

Ok let's ssh into the VM so we can install the new copy of Plex

nigel@devbox:~$ ssh nigel@192.168.0.203
The authenticity of host '192.168.0.203 (192.168.0.203)' can't be established.                                  
ECDSA key fingerprint is SHA256:BLRRmyijEFgtS+KnkXQUudkdsMskpLrzR19zBugQ2ww.                                    
Are you sure you want to continue connecting (yes/no)? yes                                                      
Warning: Permanently added '192.168.0.203' (ECDSA) to the list of known hosts.                                  
nigel@192.168.0.203's password:                                                                                 
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-87-generic x86_64)                                               
 
 * Documentation:  https://help.ubuntu.com                                                                      
 * Management:     https://landscape.canonical.com                                                              
 * Support:        https://ubuntu.com/advantage                                                                 
 
19 packages can be updated.
10 updates are security updates.
 
 
Last login: Sun Feb 18 22:11:59 2018 from 192.168.0.14
nigel@plex-media-server:~$

On a separate machine, download the latest Ubuntu 64 bit Plex Server, and scp it into the destination machine file system.

nigel@nigel-Aspire-VN7-793G:~/Downloads$ scp plexmediaserver_1.11.3.4803-c40bba82e_amd64.deb nigel@192.168.0.203:/tmp
nigel@192.168.0.203's password: 
plexmediaserver_1.11.3.4803-c40bba82e_amd64.deb                               100%  104MB 817.5KB/s   02:10    
nigel@nigel-Aspire-VN7-793G:~/Downloads$

Now switch back to the term window of the destination machine and install Plex.

nigel@plex-media-server:~$ sudo dpkg -i /tmp/plexmediaserver*.deb
[sudo] password for nigel: 
Selecting previously unselected package plexmediaserver.
(Reading database ... 93857 files and directories currently installed.)
Preparing to unpack .../plexmediaserver_1.11.3.4803-c40bba82e_amd64.deb ...
Unpacking plexmediaserver (1.11.3.4803-c40bba82e) ...
Setting up plexmediaserver (1.11.3.4803-c40bba82e) ...
Created symlink from /etc/systemd/system/multi-user.target.wants/plexmediaserver.service to /lib/systemd/system/plexmediaserver.service.
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
nigel@plex-media-server:~

The installation process will invoke Plex so stop the service

nigel@plex-media-server:~$ sudo service plexmediaserver stop

Copy to Destination Machine

We now need to copy Plex from the source to the destination machine. The best method is to scp the entire Plex file structure from one machine to the other. The file structure is at /var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/ and is owned by the user plex as you can see here:

nigel@nigel-Aspire-VN7-793G:/$ ls -las /var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/
total 44                                                                                                        
4 drwxr-xr-x 10 plex plex 4096 Feb 24 09:51 .                                                                   
4 drwxr-xr-x  3 plex plex 4096 Jan 28 20:11 ..                                                                  
4 drwxr-xr-x  5 plex plex 4096 Feb 24 09:45 Cache                                                               
4 drwxr-xr-x  3 plex plex 4096 Jan 28 20:11 Codecs                                                              
4 drwxr-xr-x  3 plex plex 4096 Jan 28 20:11 Crash Reports                                                       
4 drwxr-xr-x  3 plex plex 4096 Feb 24 09:14 Logs                                                                
4 drwxr-xr-x  3 plex plex 4096 Jan 28 20:18 Media                                                               
4 drwxr-xr-x  6 plex plex 4096 Jan 30 21:44 Metadata                                                            
4 drwxr-xr-x  3 plex plex 4096 Feb 22 20:25 Plug-ins                                                            
4 drwxr-xr-x  7 plex plex 4096 Jan 28 20:11 Plug-in Support                                                     
4 -rw-------  1 plex plex 1002 Feb 24 09:45 Preferences.xml
nigel@nigel-Aspire-VN7-793G:/$

So to get all the permissions and ownerships correct during the copy we should assign a password to the user plex so when we scp we can provide the prompted for password.

root@plex-media-server:/home/nigel# passwd plex 
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
root@plex-media-server:/home/nigel#

Ok - so if we switch back to the source machine we need to execute the scp command like this:

nigel@nigel-Aspire-VN7-793G:/$ sudo -H -u plex scp -rp  /var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/* plex@192.168.0.203:"/var/lib/plexmediaserver/Library/Application\\ Support/Plex\\ Media\\ Server/."

Note that I am running the scp as the plex user on my source machine through the superuser, and also note the double back-whacks and quotes on the destination directory structure.

Next we should start the Plex service on the destination machine and check it came up ok.

nigel@plex-media-server:~$ sudo service plexmediaserver start
nigel@plex-media-server:~$ ps -ef | grep plex
plex     17501     1  0 13:19 ?        00:00:00 /bin/sh -c LD_LIBRARY_PATH=/usr/lib/plexmediaserver "/usr/lib/plexmediaserver/Plex Media Server"
plex     17502 17501 22 13:19 ?        00:00:02 /usr/lib/plexmediaserver/Plex Media Server
plex     17517 17502 89 13:19 ?        00:00:07 Plex Plug-in [com.plexapp.system] /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.11.3.4803-c40bba82e /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/System.bundle
nigel    17541  1453  0 13:19 pts/0    00:00:00 grep --color=auto plex
nigel@plex-media-server:~$

Web Browser Destination Access

You should now be able to point your browser to Plex's new home. You need to change the URL to include your IP address, which in my case is 192.168.0.203 with a port of 32400. Yours should be similar but probably not identical. Check out the first image above for the entire URL although you won't need to include the index.html.

You may have to log in again but mine went straight to the 'Recently Added' view as you can see in the second image above. So my media and my libraries have been found ok. This shouldn't be a surprise - don't forget earlier in the tutorial I connected the VM to my FreeNAS where the media is held - and the canonical path is exactly the same to the media on the VM (destination) as it is on my source machine (my laptop).

Empty trash automatically

Now we know everything is working ok, we should re-enable the emptying of trash - see the screenshot above.

Tidying up

A little housekeeping is required. We now have two systems using the same BadzillaPlex name - so let's remove it from my laptop. To do that issue:

$ sudo dpkg --remove plexmediaserver
[sudo] password for nigel: 
(Reading database ... 334962 files and directories currently installed.)
Removing plexmediaserver (1.10.1.4602-f54242b6b) ...
Removed symlink /etc/systemd/system/multi-user.target.wants/plexmediaserver.service.
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for desktop-file-utils (0.22-1ubuntu5.1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...

blog terms

Hardware Linux VirtualBox Plex

Moving a VM Dev Environment to VirtualBox Headless

nigel — Mon, 15 Jan 2018 11:00:12 +0000

Moving a VM Dev Environment to VirtualBox Headless nigel Mon, 15/01/2018 - 11:00

I've been using a VM image on my Macbook for a while now, and using it as my Drupal development environment. It has been handcrafted with MariaDB, NGINX, Jenkins and other utilities that have built up over time such as Mailcatcher and npm. It's become increasingly inconvenient having the VM on the laptop I use for client work, so I decided to move it to my Intel NUC machine. This will provide a centralised location in my apartment to access the VM from any of my other laptops and devices.

However since my Intel NUC doesn't have an installed GUI since it is a server, a headless solution is required. To manage this I will need VirtualBox Headless - here is the tutorial on how to achieve this. My NUC machine was installed with Ubuntu 17.04 (Zesty Zapus) but other flavours of Debian based Linux will be similar.

Installing VirtualBox

To install VirtualBox, login to the host box over the local network (in my case my Intel NUC) and sudo su. Then add the repository and the public key with

root@devbox:/# wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | apt-key add -
OK
root@devbox:/#

Update and install. Note - there's every chance you will be presented with a list of numerous dependencies that will have to be installed at the same time. For brevity I haven't included the output below.

root@devbox:/# apt-get update
root@devbox:/# apt-get install virtualbox

In addition it's a good idea to install the dkms package to ensure that the VirtualBox host kernel modules (vboxdrv, vboxnetflt and vboxnetadp) are properly updated if the linux kernel version changes during the next apt-get upgrade. I did this but no updates were required.

root@devbox:/# apt-get install dkms
Reading package lists... Done
Building dependency tree       
Reading state information... Done
dkms is already the newest version (2.3-3ubuntu1.2).
dkms set to manually installed.
0 to upgrade, 0 to newly install, 0 to remove and 7 not to upgrade.
root@devbox:/#

Now we can check the installed version of VirtualBox to make sure it is installed ok.

root@devbox:/# vboxmanage --version
5.1.22_Ubuntur115126
root@devbox:/#

Export the Existing VM from VirtualBox

Now I exported the existing VM from my MacBook. This is achieved through the VirtualBox GUI. Go to File->Export Appliance... and select the VM from the list (first image above). Then change the name of the output file if you desire. I opted to do this by adding the date on to the end (image two). The third image shows the appliance's settings - if you are ok with this click Export. The final image shows the progress screen - budget on a few minutes.

Create a directory for the VM on the remote host machine.

nigel@devbox:~$ cd ~
nigel@devbox:~$ mkdir VM
nigel@devbox:~$

And scp it over to that machine from the MacBook. This could take half an hour or so.

$ scp badzilla-d8-2018-01-15.ova nigel@192.168.0.202:/home/nigel/VM/.

Import VM Image

Now let's import our VM image so VirtualBox knows about it. Note I am now logged in as myself, not as superuser.

nigel@devbox:~/VM$ VBoxManage import badzilla-d8-2018-01-15.ova 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interpreting /home/nigel/VM/badzilla-d8-2018-01-15.ova...
OK.
Disks:
  vmdisk2	16106127360	-1	http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized	badzilla-d8-2018-01-15-disk001.vmdk	-1	-1	
 
Virtual system 0:
 0: Suggested OS type: "Ubuntu_64"
    (change with "--vsys 0 --ostype "; use "list ostypes" to list all possible values)
 1: Suggested VM name "badzilla-d8"
    (change with "--vsys 0 --vmname ")
 2: Number of CPUs: 1
    (change with "--vsys 0 --cpus ")
 3: Guest memory: 1024 MB
    (change with "--vsys 0 --memory ")
 4: Sound card (appliance expects "", can change on import)
    (disable with "--vsys 0 --unit 4 --ignore")
 5: USB controller
    (disable with "--vsys 0 --unit 5 --ignore")
 6: Network adapter: orig Bridged, config 3, extra slot=0;type=Bridged
 7: CD-ROM
    (disable with "--vsys 0 --unit 7 --ignore")
 8: IDE controller, type PIIX4
    (disable with "--vsys 0 --unit 8 --ignore")
 9: IDE controller, type PIIX4
    (disable with "--vsys 0 --unit 9 --ignore")
10: SATA controller, type AHCI
    (disable with "--vsys 0 --unit 10 --ignore")
11: Hard disk image: source image=badzilla-d8-2018-01-15-disk001.vmdk, target path=/home/nigel/VirtualBox VMs/badzilla-d8/badzilla-d8-2018-01-15-disk001.vmdk, controller=10;channel=0
    (change target path with "--vsys 0 --unit 11 --disk path";
    disable with "--vsys 0 --unit 11 --ignore")
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Successfully imported the appliance.
nigel@devbox:~/VM$

Modify Existing Network Configuration

An attempt to start the VM will result in an error - see below diagnostic.

nigel@devbox:~/VM$ VBoxHeadless --startvm badzilla-d8 
Oracle VM VirtualBox Headless Interface 5.1.22_Ubuntu
(C) 2008-2017 Oracle Corporation
All rights reserved.
 
Error: failed to start machine. Error message: Failed to open/create the internal network 'HostInterfaceNetworking-en0: Wi-Fi (AirPort)' (you might need to modprobe vboxnetflt to make it accessible) (VERR_INTNET_FLT_IF_NOT_FOUND).
Failed to attach the network LUN (VERR_INTNET_FLT_IF_NOT_FOUND)
nigel@devbox:~/VM$

This is due to the network configuration which is currently set to use the MacBook's WiFi Airport device as a bridge. No good in an Intel NUC machine! So let's see how the configuration currently looks:

nigel@devbox:~/VM$ VBoxManage showvminfo badzilla-d8 | grep NIC
NIC 1:           MAC: 080027D2CF47, Attachment: Bridged Interface 'en0: Wi-Fi (AirPort)', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none
NIC 2:           disabled
NIC 3:           disabled
NIC 4:           disabled
NIC 5:           disabled
NIC 6:           disabled
NIC 7:           disabled
NIC 8:           disabled
nigel@devbox:~/VM$

Let's see what we've got in this machine...

nigel@devbox:~/VM$ VBoxManage list bridgedifs
Name:            enp3s0
GUID:            33706e65-3073-4000-8000-94c69114b597
DHCP:            Disabled
IPAddress:       192.168.0.202
NetworkMask:     255.255.255.0
IPV6Address:     fe80:0000:0000:0000:96c6:91ff:fe14:b597
IPV6NetworkMaskPrefixLength: 64
HardwareAddress: 94:c6:91:14:b5:97
MediumType:      Ethernet
Status:          Up
VBoxNetworkName: HostInterfaceNetworking-enp3s0
 
Name:            wlp2s0
GUID:            32706c77-3073-4000-8000-cc2f7140538a
DHCP:            Disabled
IPAddress:       0.0.0.0
NetworkMask:     0.0.0.0
IPV6Address:     
IPV6NetworkMaskPrefixLength: 0
HardwareAddress: cc:2f:71:40:53:8a
MediumType:      Ethernet
Status:          Down
VBoxNetworkName: HostInterfaceNetworking-wlp2s0
 
nigel@devbox:~/VM$

Ok - our eth0 port is actually called enp3s0. That's useful :) So let's set that to be our network adapter.

nigel@devbox:~/VM$ VBoxManage modifyvm badzilla-d8 --nic1 bridged --bridgeadapter1 enp3s0

Now lets see what happens when we try to start the VM

nigel@devbox:~/VM$ Oracle VM VirtualBox Headless Interface 5.1.22_Ubuntu
(C) 2008-2017 Oracle Corporation
All rights reserved.
 
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_ref

Ok - this looks more promising. The ALSA messages probably relate to a missing sound card. Let's check

Remove Soundcard

We are now clearly very close to having a configuration that works. By checking the connected devices on my router I can see that my machine badzilla-d8 has connected to the network successfully. Further more I can ping it and ssh into it! All we need to do is sort out the sound card diagnostics.

$ ping 192.168.0.17
PING 192.168.0.17 (192.168.0.17): 56 data bytes
64 bytes from 192.168.0.17: icmp_seq=0 ttl=64 time=4.286 ms
64 bytes from 192.168.0.17: icmp_seq=1 ttl=64 time=1.941 ms
64 bytes from 192.168.0.17: icmp_seq=2 ttl=64 time=2.101 ms
^C
--- 192.168.0.17 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.941/2.776/4.286/1.070 ms
$ ssh nigel@192.168.0.17
nigel@192.168.0.17's password: 
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64)
 
 * Documentation:  https://help.ubuntu.com/
 
  System information as of Mon Jan 15 16:19:31 GMT 2018
 
  System load:  0.08               Processes:           117
  Usage of /:   52.0% of 14.10GB   Users logged in:     0
  Memory usage: 5%                 IP address for eth0: 192.168.0.17
  Swap usage:   0%
 
  Graph this data and manage this system at:
    https://landscape.canonical.com/
 
Last login: Mon Jan 15 16:16:02 2018 from 192.168.0.12
$

Ok so let's see what the configuration says about our audio card:

nigel@devbox:~/VM$ VBoxManage showvminfo badzilla-d8 | grep ALSA
Audio:           enabled (Driver: ALSA, Controller: AC97, Codec: AD1980)
nigel@devbox:~/VM$

Power off the machine before we do anything to the configuration

nigel@devbox:~/VM$ VBoxManage controlvm badzilla-d8 poweroff
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
nigel@devbox:~/VM$

Now we can remove the audio, check it's definitely gone, and start the VM

nigel@devbox:~/VM$ VBoxManage modifyvm badzilla-d8 --audio none
nigel@devbox:~/VM$ VBoxManage showvminfo badzilla-d8 | grep audio
nigel@devbox:~/VM$ VBoxHeadless --startvm badzilla-d8 &
[1] 29795
nigel@devbox:~/VM$ Oracle VM VirtualBox Headless Interface 5.1.22_Ubuntu
(C) 2008-2017 Oracle Corporation
All rights reserved.
 
 
nigel@devbox:~/VM$

Ok we are up and running! Now so long as we can connect via NFS on my MAC and my Kubuntu laptop, then we should be in good shape.

NFS Connect to VM from Mac OS

Open Finder and press cmd-K to bring up the Connect to Server modal. The NFS url format must be like shown in the first image - and obviously you must have a corresponding /etc/hosts entry for the url. In my case this will 192.168.0.17.

Once connected, you will see whatever files you've got on your server. In my case I map to /var/www/html so the list of folders are effectively my projects for each website I am working on.

NFS Connect to VM from Kubuntu Linux

The NFS connection from Kubuntu is best achieved on the command line or by adding an entry to the /etc/fstab file. Once that's achieved, the directory structure can be listed. Success!

nigel@devbox:~$ sudo mount -t nfs -o vers=3 192.168.0.17:/var/www/html /mnt/projects                       
nigel@devbox:~$ ls -las /mnt/projects/
total 84                                                                                               
 4 drwxrwxr-x 13 www-data www-data  4096 Jan 15 18:13 .                                                     
 4 drwxr-xr-x  3 root     root      4096 Jan 15 19:15 ..
 4 drwxrwxr-x  3 nigel    www-data  4096 Aug 11 22:43 badzilla_d6
 4 drwxrwxr-x  7 www-data www-data  4096 Feb 19  2017 badzilla_d8
 4 -rw-rw-r--  1 www-data www-data  4096 Feb  9  2017 ._blog_theme_d8
 4 drwxrwxr-x 13 www-data www-data  4096 Feb 12  2017 blog_theme_d8
 4 -rw-rw-r--  1 www-data www-data  4096 Jan 15 18:13 ._index.html
12 -rw-rw-r--  1 www-data www-data 11510 Feb  6  2017 index.html
 4 drwxrwxr-x  7 nigel    www-data  4096 Oct 19 21:07 kitboot
12 drwxr-xr-x  4 nigel    users    12288 Dec 27 16:41 lakeland
 4 drwxrwxr-x  8 nigel    www-data  4096 Oct 22 10:49 meedjum
 4 drwxrwxr-x  4 nigel    www-data  4096 Apr 23  2017 multisite
 4 drwxrwxr-x 11 www-data www-data  4096 Feb 12  2017 personal-blog
 4 -rw-rw-r--  1 www-data www-data  1301 Aug  6  2015 README.txt
 4 drwxrwxr-x  9 nigel    www-data  4096 May  7  2017 saasidate
 4 drwxrwxr-x  9 nigel    www-data  4096 Dec 23 16:19 serverless-vote
nigel@devbox:~$

blog terms

Linux Development VirtualBox

Installation of Kubuntu for Dual Windows Boot on Acer Aspire V 17 Nitro Notebook VN7-793G Black Edition

nigel — Sat, 06 Jan 2018 10:53:57 +0000

Installation of Kubuntu for Dual Windows Boot on Acer Aspire V 17 Nitro Notebook VN7-793G Black Edition nigel Sat, 06/01/2018 - 10:53

Introduction

This is a tutorial on how to install Kubuntu (Ubuntu with the KDE desktop) on an Acer Aspire V 17 Nitro Notebook VN7-793G Black Edition laptop. The laptop is a top of the range gaming laptop, but dressed with decorum in a very stylish businesslike black case so could feasibly used at client sites without raising any eyebrows. The laptop comes pre-installed with Windows 10 - hardly my favourite operating system, but I do on occasion have a need to use it when using some legacy apps. So Kubuntu will need to co-exist with Windows 10.

Download Kubuntu

I have opted for the LTS (Long Term Support) version of Kubuntu which can be downloaded in iso format from here. I chose the 64bit 16.04 version of Ubuntu which weighed in at 1.5GB. The downloaded iso file needs to be copied over to a USB stick. I've tried the dd command method in the past in the Mac OS terminal and I've never had any joy with it. So my preferred method of creating a bootable USB stick is to use the Rufus Windows utility. We are ready for the installation but before we do that, we need to consider the composition of our dual operating system hard drives.

Disk Management Overview

The Aspire V17 Nirto comes with two hard disk drives: a 256GB SSD drive for the Operating System and a 1TB SATA drive for user data. This is configured as shown above. Since I want to retain Windows and have a dual boot laptop, I need a solution where both Windows and Kubuntu can coexist on the ultra fast SSD and the user data can be shared on the SATA drive.

Disk Management

Navigate on the Windows 10 settings screen to the Disk Management utility. You will see the disk drive information as shown above.

Disk Shrinking

The disk utility in Windows allows us the opportunity to shrink existing utilities. So I elected to half the partition space between both the OS and the Data drives. The suggested shrink size which the utility provided was spot on 50% for the C: drive (see first picture above). The data drive wasn't so I rounded the size of the shrink to 48000 bytes (see second picture). Once both partitions were shrunk the net result can be seen in picture three.

BIOS Boot Order

Put the bootable USB Stick with the Kubuntu image into the laptop and reboot. When the reboot sequence reloads the machine, tap F2 to enter the BIOS. Navigate to the Boot Sequence and you'll see the USB somewhere in the priority list. Select it and repeatedly press F6 to move it to the top, then tap F10 and save.

Kubuntu Installation

When the laptop boots, hit enter to load Kubuntu. You will be presented with the screen above after a few moments. Click on Install Kubuntu.

Installation Welcome

The installation wizard will launch and the Welcome screen will load. Click on Continue.

Installation WiFi

Set the language then click Continue and select your WiFi and add the password. Click Connect then click Continue.

Installation Prepare

The Prepare screen provides the opportunity to install 3rd party graphics and MP3 tools etc. I've ticked those since I want to use Kubuntu to play back media.

Installation Partitioning

The next screen in the installation wizard is where the fun starts - setting up the partitions. Any false move here could mean curtains for the Windows bundled installation and therefore my desired dual boot. So tread carefully here! The wizard offers a guided partition scheme or it can be set up manually. The guided mechanism is actually quite dumb - it only wants to trash entire disk drives (and therefore Windows) - so don't use it! click the Manual radio button and then Continue.

Installation Manually setting partitions

Once the manual screen is loaded, extend the modal so you can see all the partitions on the hard drives. We are going to need to set up four:

EFI Partition on the SSD to hold the Grub Loader. This needs to be around 600MB or so; I rounded it up to 1GB
Root Partition on the SSD for Ubuntu / KDE. I used the rest of the space remaining less 4GB for swap space (see below)
Swap Partition of 4GB on the SSD
User data partition on the SATA drive mounted at /home - half a terabyte.

Installation Partitions Confirmation

Once the partitions have been set up, you will see the list of partitions and that should match what you had visualised before you started this process. I have highlighted the new partitions. Note I have also set the boot location to the EFI partition I created - DON'T FORGET THIS! It isn't too late - YET - to change your mind or fix any problems. So inspect this list carefully and be sure you've got what you want. Click Install Now and the final confirmation screen appears listing the proposed changes. Double check this then click Continue if you are confident.

Installation Timezone and Keyboard

The next two screens in the wizard are self-explanatory - select the timezones and the keyboard settings.

Installation User

The final installation step is to create the user you will be logging in as. In addition set a suitable for the laptop.

Installation Boot Loader Failed to Write

The installation should run to completion after the user information has been input. Unfortunately my installation failed in the final step - the point it writes the grub loader to the SSD drive. I'm unsure why this happened - perhaps it was because my BIOS was in Secure Boot mode - but if I'd rebooted my machine at this point I would've had a Kubuntu installation that couldn't be booted. So time to find a fix...

Boot Repair Tool

I needed to install the boot repair tool to get past my boot loader issue. Bring up a terminal session - Kubuntu provides konsole

$ sudo apt-add-repository ppa:yannubuntu/boot-repair
$ sudo apt-get update
$ sudo apt-get install -y boot-repair
$ boot-repair

This will run the graphical Boot Repair tool

I elected (I like to think wisely) to tick the box to backup and rename the Windows EFI files, and then I clicked Apply. At the end of the repair process there is a link provided which gives a report. Don't lose this URL! Point your web browser at it which will be of the format https://paste.ubuntu.com/xxxxxxx. At the end of the report you will find the following

Boot successfully repaired.
 
You can now reboot your computer.
Please do not forget to make your BIOS boot on nvme0n1p5/EFI/ubuntu/shimx64.efi file!
 
You may want to retry after deactivating the [Backup and rename Windows EFI files] option.

Make a note of that file that your laptop should boot to!

Set BIOS Supervisor Password

We now need to tell our system where it can find the Grub loader shim to boot Kubuntu. That menu option is under Security but is greyed out - to enable it you will have to set a Supervisor password.

Set Grub Loader Location

Now the option we want - Select an UEFI file as trusted for executing (1) is available in the menu options. Press enter and it's time now to traverse the filesystem until you find the correct file. At the top level you'll see UEFI (2). Then ubuntu (3). Then shimx64.efi (4). When you press enter you will be prompted to provide a name which will appear in the boot menu. I chose Kubuntu.

Re-order the Boot Priorities

Now navigate to the Boot menu. At the bottom of the list you will see the entry you just added. Select it and hit F6 repeatedly to move it to the top of the list. Press F10 to save and exit. You may also want to remove the Supervisor password now your work is almost done. I didn't screenshot this - merely go back to the Security menu and change the password to nothing.

Grub Menu

The laptop boots and we have the Grub shim menu system. Topmost and default is Kubuntu (although Grub can't distinguish it from Ubuntu of course) but I can also select Windows should I wish.

Finally...

Once logged in, a screenshot of a very bare Kubuntu. Next of course I'll be adding the essential apps such as Chrome, a dock, PHPStorm, Sublime Text, Gimp etc etc. Just in case you need further proof, here's proof the /home directory mounted correctly on the SATA drive

nigel@nigel-Aspire-VN7-793G:~$ df -kh | grep -v run
Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
/dev/nvme0n1p6  103G  5.2G   93G   6% /
tmpfs           7.8G  140K  7.8G   1% /dev/shm
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/nvme0n1p5  9.4G  6.8M  9.3G   1% /boot/efi
/dev/sda2       462G  277M  438G   1% /home
nigel@nigel-Aspire-VN7-793G:~$

There's no denying it - it was quite a lengthy project but fun at the same time and now I have a fantastic laptop that can dual boot Kubuntu and Windows.

blog terms

Linux Hardware

Review of Intel NUC5CPYH with Ubuntu 17.04 Installation

nigel — Sat, 09 Dec 2017 12:41:18 +0000

Review of Intel NUC5CPYH with Ubuntu 17.04 Installation nigel Sat, 09/12/2017 - 12:41

Rating

The Intel NUC5CPYH is one of a series of Next Unit of Computing small form-factor computers designed by Intel. The barebone kits consist of the motherboard, in a plastic case with a fan, an external power supply and VESA mounting plate. I opted for the NUC5CPYH model from 2015 which is now end of life, but still for sale with considerable discount at Amazon (see Bill of Materials below). The machine comes with zero RAM and no hard disk drive so they have to be sourced separately - I went for the maximum 8GM RAM (only one slot available by the way) and a 120GB SSD hard disk.

The machine has 4 USB3 ports, support for two videos (VGA and HDMI), optical output, 7.1 sound, a Celeron N3050 1.6GB CPU, SD card reader, WiFi and a gigabyte LAN connector.

Both Windows and Linux can be supported, and I decided to install Ubuntu 17.04 Server. My use case is I wanted to move my development architecture to a containerised setup using Docker, and therefore it also made sense to move my proxy Squid server from my Raspberry Pi discussed here, over to Docker at the same time thus freeing up the Raspberry Pi for other jobs as and when they come along.

Bill of Materials

Intel NUC	Intel NUC5CPYH NUC Kit	Amazon.co.uk	£115.00
RAM	Crucial 8GB DDR3L 1600 MT/s (PC3L-12800) SODIMM 204-Pin -CT102464BF160B	Amazon.co.uk	£58.44
SSD	SanDisk SSD PLUS 120 GB up to 530 MB/s Sata III 2.5 inch Internal SSD	Amazon.co.uk	£48.99
	TOTAL		£222.43

Installation

To install the RAM and the SSD the base needs to be removed by backing off the screws in the device's feet. Note the screws won't come all the way out - doing so will compromise the cables linked between the base and the motherboard.

Once the screws have been backed off, the base can be carefully pulled away from the case. Be careful not to pull out the cables accidentally. The slot for the RAM is at the front of the pic.

The RAM can now be slotted in. This is achieved by carefully sliding the connectors into the slots with the memory stick resting against the supporting clips then gently pressing it down until a satisfying click is heard.

The SSD slot is actually a caddy which is built into the base and the SSD can be easily slid in although it is a fairly tight fit. Push it gently to ensure the connectors marry up. The base can now be screwed close and the peripherals connected (screen, keyboard, mouse) before powering up using the multi-country power connector.

Configuration

The machine comes with a lovely BIOS dashboard. When it starts to boot up, press the F2 key to access it.

If you click on advanced you can get to the main dashboard. Here you will see your configuration - living proof it can find its 8GB memory!

Installation of Ubuntu 17.04

I downloaded an ISO image of Ubuntu 17.04 AMD 64 bit from here and using the Rufus ISO writer, wrote this to a Cruzer USB memory stick. Note that the SD drive is not bootable so USB has to be used. I placed the USB stick in a spare USB slot and navigated to the boot menu on the BIOS screens.

The order required (under Legacy) is the SSD drive first, then the Sandisk USB stick second. That way once the OS has been installed on the SSD and the installation process has rebooted it won't start the installation again.

Ubuntu Software Installation

The software installation is straightforward. Since I am going to containerise all my development services I don't need to install too much. The standard utilities are always going to be useful, but also don't forget OpenSSH server - this will enable you to ssh into the box so you can consign the screen keyboard and mouse to the cupboard.

Ubuntu Static IP Address

By default as part of the installation process, the LAN will be configured to request an IP address from DHCP server. I would like mine to be static so to achieve that, edit /etc/network/interface

root@devbox:/etc/network# cat interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
 
source /etc/network/interfaces.d/*
 
# The loopback network interface
auto lo
iface lo inet loopback
 
# The primary network interface
auto enp3s0
#iface enp3s0 inet dhcp
iface enp3s0 inet static
 address 192.168.0.202
 netmask 255.255.255.0
 network 192.168.0.1
 broadcast 192.168.1.255
 gateway 192.168.0.1
 dns-nameservers 90.207.238.97
 
 
root@devbox:/etc/network#

Obviously you will need to change the settings to suit your own circumstances. I am using my ISP's nameserver and I have allocated this machine 192.168.0.202.

Conclusion

So far I have been very impressed with the Intel NUC. It is built with a great deal of quality - just packing up the packaging I was astonished by the sheer weight of this small 4" or so box. Yes I have bought was it is effectively the bottom of the range (i.e. cheapest) offering, and it is now two years since it was launched - a lifetime for hardware. But it performs extremely well for what I need - my own containerised development environment. It has the headroom (8GB RAM) to run all I will need it to and still have plenty of horsepower left should I need to expand.

blog terms

Linux Hardware

Ad Blocker using Squid Proxy on Raspberry Pi

nigel — Sun, 17 Sep 2017 15:33:40 +0000

Ad Blocker using Squid Proxy on Raspberry Pi nigel Sun, 17/09/2017 - 16:33

As mentioned in my previous blog Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox it is possible (in fact quite usual) to use Squid to prevent user access to certain websites. This can include ad servers which serve up annoying ad content when trying to read articles. This tutorial shows how we can extend the functionality of the Squid server we have already set up to block ads.

The banning of ad content is achieved by downloading a blacklist of prohibited urls from a service provider, although it has to be mentioned that these lists depend upon the community to contribute any urls they discover and thus there can never be 100% confidence that such a list is ever complete.

Once the list is downloaded from the service provider, Squid needs to be informed of its presence in its configuration and the ban will take the form of an acl. Ok now we know what we need to achieve let's start the process.

Lets log into the Raspberry Pi, switch to superuser, and download the blacklist.

Nigels-MacBook-Pro:Projects nigel$ ssh pi@192.168.0.201
pi@192.168.0.201s password: 
 
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
 
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Sep 10 14:33:42 2017 from 192.168.0.7
pi@raspberrypi:~ $ sudo su
root@raspberrypi:/home/pi# cd /etc/squid3/
root@raspberrypi:/etc/squid3# ls -las
total 832
  4 drwxr-xr-x   2 root root   4096 Sep  9 08:44 .
  4 drwxr-xr-x 112 root root   4096 Sep  9 11:17 ..
  4 -rw-r--r--   1 root root   1547 Dec 24  2016 errorpage.css
  4 -rw-r--r--   1 root root    421 Dec 24  2016 msntauth.conf
272 -rw-r--r--   1 root root 277585 Sep  9 08:44 squid.conf
272 -rw-r--r--   1 root root 277565 Sep  3 11:06 squid.conf.local-vm
272 -rw-r--r--   1 root root 277526 Sep  2 19:22 squid.conf.original
root@raspberrypi:/etc/squid3# curl -sS -L --compressed "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=0&mimetype=plaintext" > ad_block.txt 
root@raspberrypi:/etc/squid3# ls -las
total 872
  4 drwxr-xr-x   2 root root   4096 Sep 17 15:20 .
  4 drwxr-xr-x 112 root root   4096 Sep  9 11:17 ..
 40 -rw-r--r--   1 root root  40042 Sep 17 15:20 ad_block.txt
  4 -rw-r--r--   1 root root   1547 Dec 24  2016 errorpage.css
  4 -rw-r--r--   1 root root    421 Dec 24  2016 msntauth.conf
272 -rw-r--r--   1 root root 277585 Sep  9 08:44 squid.conf
272 -rw-r--r--   1 root root 277565 Sep  3 11:06 squid.conf.local-vm
272 -rw-r--r--   1 root root 277526 Sep  2 19:22 squid.conf.original
root@raspberrypi:/etc/squid3#

The blacklist is now in place - you can see it in the directory listing as file ad_block.txt. The next job is to add this to the Squid configuration as an acl. This needs to be placed above the http_access allow all line we put in the config in the previous blog.

## disable ads ( http://pgl.yoyo.org/adservers/ )
acl ads dstdom_regex "/etc/squid3/ad_block.txt"
http_access deny ads

You can use your favourite editor for this - I used vi - and then reboot the server.

root@raspberrypi:/etc/squid3# vi ad_block.txt 
root@raspberrypi:/etc/squid3# service squid3 restart

We are now ready to test our solution by landing on an ad heavy website such as NME. There are two excellent methods of doing this - one is checking the access.log in real time for TCP_DENIED records, and the other is to eyeball the site and see if the ads really are missing. Let's go for option one first.

root@raspberrypi:/etc/squid3# tail -f /var/log/squid3/access.log
1505665437.619     38 192.168.0.4 TCP_MISS/200 630 GET http://pagead2.googlesyndication.com/activeview? - HIER_DIRECT/216.58.208.130 image/gif
1505665440.152   3632 192.168.0.4 TCP_MISS/200 4009 CONNECT bs.serving-sys.com:443 - HIER_DIRECT/82.199.68.72 -
1505665449.828   8361 192.168.0.4 TCP_MISS/200 630 CONNECT ml314.com:443 - HIER_DIRECT/34.252.181.159 -
1505665452.178     78 192.168.0.4 TCP_MISS/204 740 GET http://beacon.krxd.net/event.gif? - HIER_DIRECT/54.228.222.160 image/gif
1505665491.360  69091 192.168.0.4 TCP_MISS/200 746 CONNECT www.facebook.com:443 - HIER_DIRECT/31.13.90.36 -
1505665493.069  65946 192.168.0.4 TCP_MISS/200 5038 CONNECT match.adsrvr.org:443 - HIER_DIRECT/54.247.91.116 -
1505665502.280  65538 192.168.0.4 TCP_MISS/200 5552 CONNECT odr.mookie1.com:443 - HIER_DIRECT/35.157.149.45 -
1505665519.263  91582 192.168.0.4 TCP_MISS/200 3318 CONNECT bam.nr-data.net:443 - HIER_DIRECT/162.247.242.21 -
1505665544.650 120399 192.168.0.4 TCP_MISS/200 1227 CONNECT session.timecommerce.net:443 - HIER_DIRECT/54.192.2.48 -
1505665554.600      7 192.168.0.4 TCP_MISS/503 4491 GET http://pagead2.googlesyndication.com/activeview? - HIER_NONE/- text/html
1505665577.132      8 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665577.146      9 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665577.357    200 192.168.0.4 TCP_MISS/200 27785 GET http://www.nme.com/ - HIER_DIRECT/54.230.11.205 text/html
1505665577.396    249 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665577.495      7 192.168.0.4 TCP_DENIED/403 4577 GET http://pagead2.googlesyndication.com/activeview? - HIER_NONE/- text/html
1505665577.567      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665577.615      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665577.748    337 192.168.0.4 TCP_MISS/200 364 GET http://www.summerhamster.com/bcn? - HIER_DIRECT/52.27.8.169 image/gif
1505665579.364     58 192.168.0.4 TCP_MISS/304 446 GET http://dnn506yrbagrg.cloudfront.net/pages/scripts/0025/8842.js - HIER_DIRECT/54.230.11.179 -
1505665579.380      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665579.669     69 192.168.0.4 TCP_MISS/200 576 GET http://mpp.vindicosuite.com/bg/? - HIER_DIRECT/130.211.103.172 application/javascript
1505665579.676      4 192.168.0.4 TCP_DENIED/403 4350 POST http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665579.761      3 192.168.0.4 TCP_DENIED/403 4176 GET http://b.scorecardresearch.com/b? - HIER_NONE/- text/html
1505665579.785     79 192.168.0.4 TCP_MISS/304 341 GET http://img.en25.com/i/elqCfg.min.js - HIER_DIRECT/23.207.182.156 application/x-javascript
1505665579.794    197 192.168.0.4 TCP_MISS/200 8677 GET http://uid1.vindicosuite.com/js/tm.js? - HIER_DIRECT/35.186.160.37 application/x-javascript
1505665579.802     64 192.168.0.4 TCP_MISS/304 577 GET http://cdn.optimizely.com/js/103816086.js - HIER_DIRECT/2.21.189.139 text/javascript
1505665579.852      3 192.168.0.4 TCP_DENIED/403 6207 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665579.928     83 192.168.0.4 TCP_MISS/200 619 GET http://uconnect.tealiumiq.com/ulog/_error? - HIER_DIRECT/54.72.62.235 image/gif
1505665579.939      7 192.168.0.4 TCP_DENIED/403 10853 GET http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665579.989      2 192.168.0.4 TCP_DENIED/403 3718 CONNECT googleads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665580.067      3 192.168.0.4 TCP_DENIED/403 6209 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665580.696      3 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665580.728      3 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665580.756      3 192.168.0.4 TCP_DENIED/403 4061 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665580.758      2 192.168.0.4 TCP_DENIED/403 4061 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665580.763      2 192.168.0.4 TCP_DENIED/403 4446 GET http://r.skimresources.com/api/? - HIER_NONE/- text/html
1505665580.829      2 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665581.101    222 192.168.0.4 TCP_MISS/200 413 GET http://s1642912926.t.eloqua.com/visitor/v200/svrGP? - HIER_DIRECT/142.0.160.13 image/gif
1505665581.101    225 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665581.462     53 192.168.0.4 TCP_MISS/200 726 GET http://0914.global.ssl.fastly.net/ad2/script/x.js? - HIER_DIRECT/151.101.60.249 text/javascript
1505665581.492     44 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665581.496     43 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665581.661      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665581.666      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665581.732      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665582.039    204 192.168.0.4 TCP_MISS/204 657 GET http://gwiq.globalwebindex.net/gwiq/? - HIER_DIRECT/69.16.175.10 text/javascript
1505665582.099     90 192.168.0.4 TCP_MISS/200 8775 GET http://ksassets.timeincuk.net/wp/uploads/sites/55/2016/10/cropped-nme-site-icon-300x300.jpg - HIER_DIRECT/52.85.74.98 image/jpeg
1505665582.208      2 192.168.0.4 TCP_DENIED/403 4197 GET http://b.scorecardresearch.com/b? - HIER_NONE/- text/html
1505665582.230    114 192.168.0.4 TCP_MISS/304 534 GET http://ksassets.timeincuk.net/wp/uploads/sites/55/2016/10/cropped-nme-site-icon-300x300.jpg - HIER_DIRECT/52.85.74.98 -
1505665582.314    262 192.168.0.4 TCP_MISS/200 605 GET http://po.st/v1/status? - HIER_DIRECT/74.217.253.90 application/javascript
1505665582.414     71 192.168.0.4 TCP_MISS/200 410 GET http://p.po.st/p? - HIER_DIRECT/208.146.36.215 image/gif
1505665586.135      2 192.168.0.4 TCP_DENIED/403 3925 POST http://t.skimresources.com/api/link - HIER_NONE/- text/html
1505665592.588  13319 192.168.0.4 TCP_MISS/200 4157 CONNECT c.disquscdn.com:443 - HIER_DIRECT/104.16.77.166 -
1505665592.589  15014 192.168.0.4 TCP_MISS/200 154 CONNECT cdn.shopify.com:443 - HIER_DIRECT/151.101.62.110 -
1505665592.591  15018 192.168.0.4 TCP_MISS/200 154 CONNECT cdn.shopify.com:443 - HIER_DIRECT/151.101.62.110 -
1505665592.592  15460 192.168.0.4 TCP_MISS/200 213 CONNECT csi.gstatic.com:443 - HIER_DIRECT/216.58.201.163 -
1505665592.592  13326 192.168.0.4 TCP_MISS/200 3666 CONNECT disqus.com:443 - HIER_DIRECT/151.101.64.134 -
1505665601.000      4 192.168.0.4 TCP_DENIED/403 6210 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665601.173      4 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665601.173      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665603.359      8 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665603.359      3 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665603.557    150 192.168.0.4 TCP_MISS/200 27785 GET http://www.nme.com/ - HIER_DIRECT/54.230.11.205 text/html
1505665603.560    200 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665603.720      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665603.889    173 192.168.0.4 TCP_MISS/200 364 GET http://www.summerhamster.com/bcn? - HIER_DIRECT/52.27.8.169 image/gif
1505665605.489     24 192.168.0.4 TCP_MISS/304 446 GET http://dnn506yrbagrg.cloudfront.net/pages/scripts/0025/8842.js - HIER_DIRECT/54.230.11.179 -
1505665605.595     24 192.168.0.4 TCP_MISS/304 341 GET http://img.en25.com/i/elqCfg.min.js - HIER_DIRECT/23.207.182.156 application/x-javascript
1505665605.601     34 192.168.0.4 TCP_MISS/200 576 GET http://mpp.vindicosuite.com/bg/? - HIER_DIRECT/130.211.103.172 application/javascript
1505665605.615      3 192.168.0.4 TCP_DENIED/403 4350 POST http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665605.620      2 192.168.0.4 TCP_DENIED/403 4176 GET http://b.scorecardresearch.com/b? - HIER_NONE/- text/html
1505665605.679      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665605.686      4 192.168.0.4 TCP_DENIED/403 6207 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665605.734      6 192.168.0.4 TCP_DENIED/403 10875 GET http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665605.745      2 192.168.0.4 TCP_DENIED/403 3718 CONNECT googleads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665605.748    207 192.168.0.4 TCP_MISS/200 8677 GET http://uid1.vindicosuite.com/js/tm.js? - HIER_DIRECT/35.186.160.37 application/x-javascript
1505665605.776     38 192.168.0.4 TCP_MISS/200 619 GET http://uconnect.tealiumiq.com/ulog/_error? - HIER_DIRECT/54.72.62.235 image/gif
1505665605.882      3 192.168.0.4 TCP_DENIED/403 4062 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665605.887      3 192.168.0.4 TCP_DENIED/403 4062 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665605.892      3 192.168.0.4 TCP_DENIED/403 4446 GET http://r.skimresources.com/api/? - HIER_NONE/- text/html
1505665605.998    111 192.168.0.4 TCP_MISS/200 413 GET http://s1642912926.t.eloqua.com/visitor/v200/svrGP? - HIER_DIRECT/142.0.160.13 image/gif
1505665606.169      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665606.175      2 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665606.669    208 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665606.788     22 192.168.0.4 TCP_MISS/200 728 GET http://0914.global.ssl.fastly.net/ad2/script/x.js? - HIER_DIRECT/151.101.60.249 text/javascript
1505665606.813     22 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665606.817     24 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665607.000      3 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665607.003      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
^C
root@raspberrypi:/etc/squid3#

Looks like there is a bunch of them! To check let's pipe the access file through cordcount looking for TCP_DENIED

root@raspberrypi:/etc/squid3# cat /var/log/squid3/access.log | grep TCP_DENIED | wc
     44     440    5192
root@raspberrypi:/etc/squid3#

There we are - there have been 44 instances that have been blocked.

Eyeball the Websites

NME was totally successful - empty spaces in the screen where the blocked ads would've been seen. In this instance two ads have been blocked.

The native Android app Whisper was partially successful. The main ad shows a diagnostic that the ad failed to launch - I'm happy with that - but it failed to block the small ad at the bottom of the screen.

Automating the Fetching of the Blacklist

The list should be refreshed occasionally - like every week - in case there are new entries. The best way of achieving this is to put the earlier curl command in a shell script and then add a cron entry.
/etc/squid3/newads.sh

#!/bin/bash
curl -sS -L --compressed "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=0&mimetype=plaintext" > /etc/squid3/ad_block.txt 
 
## restart squid
/usr/sbin/squid3 restart

And now ad an entry to the root crontab - I have elected to download the script once a week just into Sunday morning.
crontab -e

# Squid3 fetch ad blocker blacklist
5 4 * * 1 /etc/squid3/newads.sh  > /dev/null 2>&1

blog terms

Squid Raspberry Pi Linux Hardware

Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox

nigel — Sat, 09 Sep 2017 21:43:47 +0000

Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox nigel Sat, 09/09/2017 - 22:43

This blog shows you how to build a Squid proxy server on an inexpensive Raspberry Pi machine to help to test web application development in a local VM sandbox.

So here's the problem. Say you are using a VirtualBox hosting Linux in a MacBook to develop your web application. This is practically an industry standard these days; certainly in the open source world anyway. You will have defined the IP address allocated to your VM in your /etc/hosts along with the domain name you have chosen.

You have elected a 'browser first' strategy for the markup, but how do you test your website against hand held devices? Option 1 is using the simulator in Chrome and other browsers which do a very good job of emulating most common devices, but they aren't perfect and frequently the results will differ between the simulator and the physical device. But since it isn't possible to edit the /etc/hosts file in hand held devices, how do you point them to your arbitrarily named website domain? Well the second option is to use an external servicessuch as localtunnel that can expose an IP address to the Internet - but a local service must be running constantly on the VM and firewalls have to be configured to allow incoming http requests which is not always desirable.

There is a third way - use a proxy server to redirect the urls tapped into the hand held devices to the VM in your host machine (Mac / Windows or Linux). The easiest way to achieve this is to use Squid - a mature and free proxy server used throughout the IT industry. It has immense capabilities - such as blacklisting genres of websites (e.g. porn / sex sites) so young family members aren't exposed to what they shouldn't be. But we really only need simple redirection.

So my choice is Squid which runs on Linux. Should I buy even more new hardware for my apartment? Or repurpose equipment from my hardware junkyard in the spare bedroom? Nope. I've always wanted a Raspberry Pi to play with, and now I have an excuse to buy one!

Bill of Materials

Device	Raspberry Pi 3 Model B Quad Core CPU 1.2 GHz 1 GB RAM Motherboard	Amazon.co.uk	£32.50
SD Card	SanDisk Ultra 32 GB microSDHC Class 10 Memory Card up to 48 Mb/s - White/Grey	Amazon.co.uk	£13.50
Box	Raspberry Pi 3 Case - Black/Grey	Amazon.co.uk	£5.50
Keyboard	Mofii Full-size Colorful Wired USB LED Backlit Gaming Keyboard for PC & MAC - UK Layout	Amazon.co.uk	£12.99
Power Cable	2M / 6.5ft USB 2.0 A To MICRO B Cable Data and Charging Lead Cable - Black	eBay.co.uk	£1.65
Wall Plug	DC5V 2A UK 3pin USB Charger Wall Plug Power Adapter	eBay.co.uk	£3.45
Mouse	USB Mouse - spare	n/a	£-
Network Cable	RJ45 CAT5 cable - spare	n/a	£-
	TOTAL		£68.80

So the total price was significantly less than buying a new desktop or laptop computer and obviously the USB keyboard can be reused on other projects. I could've reduced my exposure by sharing phone charging cables around my apartment rather than buy another one to be dedicated for the Raspberry Pi but I wanted a 5V 2A power supply so it made sense to spend a small amount of extra money.

Building the System

There are plenty of tutorials on the Internet for setting up the base Raspberry Pi operation system - so I won't duplicate it here, but please note you will need to install the Raspberry Pi version of Debian Linux (Raspbian) on the SD card.

Here's a list of useful links:

 Raspberry Pi Software Guide
 How to set up and use a Raspberry Pi 3 with a Mac
 SETTING WIFI UP VIA THE COMMAND LINE - Only needed if you are not going to use a static IP address through the LAN connection
 Use Ssh to Talk With Your Raspberry Pi.   

Once your Raspberry Pi is set up and you are ready to go, you can either ssh into the machine (as I did) or connect to it using a keyboard and mouse. If you choose SSH, you will have to enable that service first - see the link above if you are unsure how to do this.

Update the distribution

Always a good idea to update the box in case there are pending patches which weren't in the initial downloaded distribution. When I ran the commands below I got a welter of updates (which I've removed from the blow listing due to brevity reasons)

pi@raspberrypi:~ $ sudo apt-get update
pi@raspberrypi:~ $ sudo apt-get upgrade
pi@raspberrypi:~ $ sudo apt-get dist-upgrade

Set a Static IP Address

The Raspberry Pi 3 does come with WifI so theoretically it is possible to set it up to have a DHCP dynamically assigned IP address. However that would be saving up problems should the IP address change due to rebooting or power interruptions. It would mean a whole load of settings would need to be changed in our system each time that happens. Most routers come with a built-in DHCP server and the range of IP addresses can be set - see below. I have already set my range to be 192.168.0.2 to 192.168.0.199 so anything 200 and above is reserved for devices that need a static address.

I now start up the Raspberry Pi, connected to my LAN with a RJ45 cable, and it gets allocated a dynamic IP address by the DHCP server on my router. This can be checked through the router UI (see below) or by issuing ifconfig when logged on to the Pi.

Become the super user and edit the /etc/dhcpcd.conf file with vi or nano.

pi@raspberrypi:~ $ sudo su
root@raspberrypi:/home/pi# cd /etc/
root@raspberrypi:/etc# vi dhcpcd.conf

Now add the following text to the end of it. Note that I am just using the tail command to list the edits I have already made.

root@raspberrypi:/etc# tail -5 dhcpcd.conf
 
interface eth0
static ip_address=192.168.0.201/24        
static routers=192.168.0.1
static domain_name_servers=90.207.238.97
root@raspberrypi:/etc#

You will need to change the values according to your circumstances. I am using 201 for my Pi server, 1 is my router and the domain name server setting is pointing to my ISP's DNS address. Now reboot the Pi.

root@raspberrypi:/etc# init 6
Connection to 192.168.0.12 closed by remote host.
Connection to 192.168.0.12 closed.

Once the device has completed its startup sequence, go back to your router and check you've got an entry for your Pi - see screenshot below. My two servers can be seen highlighted - FreeNAS using 200 and the Pi at 201. We are good to go!

Install Squid

Ok let's install Squid

pi@raspberrypi:~ $ sudo apt-get install squid3
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libecap2 libmnl0 libnetfilter-conntrack3 squid-langpack squid3-common
Suggested packages:
  squidclient squid-cgi squid-purge smbclient ufw winbindd
The following NEW packages will be installed:
  libecap2 libmnl0 libnetfilter-conntrack3 squid-langpack squid3 squid3-common
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,370 kB of archives.
After this operation, 8,372 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://mirrordirector.raspbian.org/raspbian/ jessie/main libmnl0 armhf 1.0.3-5 [10.9 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ jessie/main libnetfilter-conntrack3 armhf 1.0.4-1 [40.0 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid-langpack all 20140506-1 [150 kB]
Get:4 http://mirrordirector.raspbian.org/raspbian/ jessie/main libecap2 armhf 0.2.0-3 [13.8 kB]
Get:5 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid3-common all 3.4.8-6+deb8u4 [259 kB]
Get:6 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid3 armhf 3.4.8-6+deb8u4 [1,897 kB]
Fetched 2,370 kB in 3s (634 kB/s)   
Selecting previously unselected package libmnl0:armhf.
(Reading database ... 115447 files and directories currently installed.)
Preparing to unpack .../libmnl0_1.0.3-5_armhf.deb ...
Unpacking libmnl0:armhf (1.0.3-5) ...
Selecting previously unselected package libnetfilter-conntrack3:armhf.
Preparing to unpack .../libnetfilter-conntrack3_1.0.4-1_armhf.deb ...
Unpacking libnetfilter-conntrack3:armhf (1.0.4-1) ...
Selecting previously unselected package squid-langpack.
Preparing to unpack .../squid-langpack_20140506-1_all.deb ...
Unpacking squid-langpack (20140506-1) ...
Selecting previously unselected package libecap2.
Preparing to unpack .../libecap2_0.2.0-3_armhf.deb ...
Unpacking libecap2 (0.2.0-3) ...
Selecting previously unselected package squid3-common.
Preparing to unpack .../squid3-common_3.4.8-6+deb8u4_all.deb ...
Unpacking squid3-common (3.4.8-6+deb8u4) ...
Selecting previously unselected package squid3.
Preparing to unpack .../squid3_3.4.8-6+deb8u4_armhf.deb ...
Unpacking squid3 (3.4.8-6+deb8u4) ...
Processing triggers for man-db (2.7.5-1~bpo8+1) ...
Processing triggers for systemd (215-17+deb8u7) ...
Setting up libmnl0:armhf (1.0.3-5) ...
Setting up libnetfilter-conntrack3:armhf (1.0.4-1) ...
Setting up squid-langpack (20140506-1) ...
Setting up libecap2 (0.2.0-3) ...
Setting up squid3-common (3.4.8-6+deb8u4) ...
Setting up squid3 (3.4.8-6+deb8u4) ...
Creating Squid HTTP proxy 3.x spool directory structure
2017/07/30 18:55:00 kid1| Set Current Directory to /var/spool/squid3
2017/07/30 18:55:00 kid1| Creating missing swap directories
2017/07/30 18:55:00 kid1| No cache_dir stores are configured.
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Processing triggers for systemd (215-17+deb8u7) ...
pi@raspberrypi:~ $

Configuring Squid

First change directory to the squid configuration Squid directory and save the original configuration file just in case it needs to revered sometime.

root@raspberrypi:~# cd /etc/squid3/
root@raspberrypi:/etc/squid3# cp squid.conf squid.conf.original
root@raspberrypi:/etc/squid3#

By default Squid doesn't allow any access to the Internet so that will need fixing. Like most ACL based systems, once Squid finds a rule that applies to a particular URL then it stops looking for more rules. So early in the config file you need to tell Squid to allow all HTTP traffic.

http_access allow all

You will also need to tell Squid about your local network. I am using the 192.168.0.0 range of IP addresses - find that in the configuration and uncomment it

acl localnet src 192.168.0.0/16	# RFC1918 possible internal network

Finally we don't want any caching since we will be using our Squid mainly for redirection purposes and when recent website changes need to be seen for testing purposes, so add this at the caching section

cache deny all

Below is a list of the changes in a form of a diff file which some may find more useful than the separate settings I've mentioned above.

root@raspberrypi:/etc/squid3# diff -Naur  squid.conf.original squid.conf
--- squid.conf.original	2017-09-02 19:22:23.804970154 +0000
+++ squid.conf	2017-09-03 10:41:48.190318779 +0000
@@ -1037,10 +1037,11 @@
 # should be allowed
 #acl localnet src 10.0.0.0/8	# RFC1918 possible internal network
 #acl localnet src 172.16.0.0/12	# RFC1918 possible internal network
-#acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
+acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
 #acl localnet src fc00::/7       # RFC 4193 local private network range
 #acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines
 
+
 acl SSL_ports port 443
 acl Safe_ports port 80		# http
 acl Safe_ports port 21		# ftp
@@ -1180,6 +1181,7 @@
 #Default:
 # Deny, unless rules exist in squid.conf.
 #
+http_access allow all
 
 #
 # Recommended minimum Access Permission configuration:
@@ -2999,6 +3001,9 @@
 # DISK CACHE OPTIONS
 # -----------------------------------------------------------------------------
 
+cache deny all
+
+
 #  TAG: cache_replacement_policy
 #	The cache replacement policy parameter determines which
 #	objects are evicted (replaced) when disk space is needed.
root@raspberrypi:/etc/squid3#

We are now done with the configuration in the conf file so we can restart the Squid service so these changes take effect

root@raspberrypi:/etc/squid3# service squid3 restart
root@raspberrypi:/etc/squid3#

Won't be the worst idea to copy the configuration just for safety.

root@raspberrypi:/etc/squid3# cp squid.conf squid.conf.local-vm
root@raspberrypi:/etc/squid3#

You are now probably wondering how does Squid possibly know where my Virtual Hosts in my Virtual Machine are!! This is the simple bit :) - you need to add all those sites you are working on into the /etc/hosts file on your Pi. You will already have these listed in your Macbook hosts file so simply copy and paste. My file ended up looking like this below:

root@raspberrypi:/etc/squid3# cat /etc/hosts
127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters
 
127.0.1.1	raspberrypi
 
192.168.0.14 badzilla-d8.localhost
192.168.0.14 meedjum.localhost
root@raspberrypi:/etc/squid3#

The last two entries - badzilla-d8.localhost and meedjum.localhost are the two sites I am working on. The migration of Badzilla from D6 to D8. Yey!

Handheld Device Client Configuration

We are now ready to test our solution, and for this I'll use screenshots from Android and iOS.

Android

Go to Settings->WiFi, then long tap on your network and tap 'Modify network config', then tap 'Show advanced options' and add the configuration for the proxy, then tap save.

Now point a web browser to your web development domain

iOS

Go to settings, tap on your WiFi, tap on the information icon, tap on 'Manual' under HTTP Proxy, and enter the values.

Now point a web browser to your web development domain - this is very early in the process as you can see. Somehow two seach boxes are on this early build!

blog terms

Raspberry Pi Squid Linux Drupal Development VirtualBox

livetvmatches.com: nginx Drupal 8 configuration with Microcaching and 410 Wildcards

nigel — Thu, 07 Apr 2016 16:43:53 +0000

livetvmatches.com: nginx Drupal 8 configuration with Microcaching and 410 Wildcards nigel Thu, 07/04/2016 - 17:43

Ok for the impatient here's the nginx configuration for my new website that includes Microcaching and some wildcard configuration for generating HTTP 410 'Gone Away' at the webserver before the framework / CMS is hit.
/etc/nginx/nginx.conf

worker_processes  1;
error_log  /var/log/nginx/error.log;
events {
    worker_connections  1024;
    use epoll;
}
 
http {
    include       mime.types;
    default_type  application/octet-stream;
 
    fastcgi_cache_path /var/cache/nginx2 levels=1:2 keys_zone=microcache:1m max_size=1000m;
    log_format cache '$remote_addr - $remote_user [$time_local] "$request" '
                '$status $upstream_cache_status $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for"';
 
    sendfile        on;
 
    keepalive_timeout  65;
 
    gzip  on;
    gzip_min_length 1100;
    gzip_buffers 4 32k;
    gzip_types text/plain application/x-javascript text/xml text/css;
 
    include vhosts.d/*.conf;
}

/etc/nginx/vhosts.d/mywebsite.conf

server {
  listen                *:80;
  server_name           mywebsite.com www.mywebsite.com;
  client_max_body_size 1m;
  root /srv/www/htdocs/mywebsite/docroot;
    index   index.php;
 
  access_log            /var/log/nginx/mywebsite_access.log;
  error_log             /var/log/nginx/lmywebsite_error.log;
 
  error_page 410 /custom_410.html;
  location  = /custom_410.html {
    root /srv/www/htdocs/mywebsite/docroot/;
    internal;
  }
 
  location ~ \..*/.*\.php$ {
    return 403;
  }
 
  # Block access to hidden directories
  location ~ (^|/)\. {
    return 403;
  }
 
  location ~ ^/sites/.*/private/ {
    return 403;
  }
 
  # No php is touched for static content
  location / {
    try_files $uri @rewrite;
  }
 
  # pass the PHP scripts to FastCGI server
  location ~ \.php$ {
    fastcgi_index index.php;
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
 
    set $no_cache "";
    if ($request_method !~ ^(GET|HEAD)$) {
      set $no_cache "1";
    }       
 
    if ($no_cache = "1") {
      add_header Set-Cookie "_mcnc=1; Max-Age=2; Path=/";
      add_header X-Microcachable "0";
    }
 
    if ($http_cookie ~ SESS) {
      set $no_cache "1";
    }               
 
    fastcgi_no_cache $no_cache;
    fastcgi_cache_bypass $no_cache;
    fastcgi_cache microcache;
    fastcgi_cache_lock on;
    fastcgi_cache_key $server_name|$request_uri;
    fastcgi_cache_valid 404 30m;
    fastcgi_cache_valid 200 10s;
    fastcgi_max_temp_file_size 10M;
    fastcgi_cache_use_stale updating;
 
    # The address or socket on which FastCGI requests are accepted. Set yours in www.conf
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_pass_header Set-Cookie;
    fastcgi_pass_header Cookie;
    fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
  }
 
  # Clean URLs
  location @rewrite {
    rewrite ^ /index.php;
  }
 
  # Bogus links from previous owner of website
  location ~ ^/.*(wp-content|feed|category|streaming-|online-|live-|watch-|highlights-|stream-)(.*) {
    return 410;
  }
 
  # Image styles
  location ~ ^/sites/.*/files/styles/ {
    try_files $uri @rewrite;
  }
 
  location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
    expires max;
    log_not_found off;
  }
 
  location = /themes/yourtheme/favicon.ico {
    log_not_found off;
    access_log off;
  }
 
  location = /robots.txt {
    allow all;
    log_not_found off;
    access_log off;
  }
 
}

/srv/www/htdocs/mywebsite/docroot/custom410.html

<html>
<head>
<style>--
                body {font-family: arial,sans-serif}
                img { border:none; }
//-->style>
   <meta name="robots" content="noindex">
   <title>Page Gone - 410 Errortitle>
head>
<body>
<blockquote>
<hr>
<h1>Error 410 - Page deleted or goneh1>
This might be because:
<ul>
  <li>You have typed the web address incorrectly, or the page you were looking for may have been deleted.li>
ul>
blockquote>
body>
html>

Ok - so this isn't a tutorial on nginx or Microcache or Drupal since there are many others online, but this configuration worked for me. Microcaching is recommended for short term caching of dynamic content and goes well with nginx - The Benefits of Microcaching with NGINX. The configuration will need a caching directory to be configured - and I went for /var/cache/nginx as can be seen in the config files - so be sure to create the directory beforehand as root or it won't work.

Tuning the cache values will depend on your use case. Most blogs suggest 1 or 2 seconds but since my data isn't particularly dynamic I opted for 10 seconds. To test the performance of your cache you should use the Apache Benchmark (ab) command line utility. I would take its figures with a HUGE pinch of salt but notwithstanding that it is very good for comparison purposes and proves whether changes made are heading in the correct direction.

The configuration for '410 gone away' errors were a necessity in my case. I learned my domain http://livetvmatches.com had been preowned purely by the fact I was getting thousands of 404 Not Found errors by the Google bot, and they were all URLs showing illegal streaming of live football that my domain used to service or perhaps use as a redirection starting point. Whilst Google won't score a site down for 404s, they now recognise 410 Gone Away and will desist from trying the url again. By looking at my configuration - searching for various words within the URL - you should gain an appreciation how to achieve the same effect on your site.

blog terms

nginx Drupal Drupal 8 Linux

Configure XDebug with PuPHPet, Vagrant, Apache, Ubuntu, PHP-FPM and PhpStorm

nigel — Sat, 20 Jun 2015 09:23:28 +0000

Configure XDebug with PuPHPet, Vagrant, Apache, Ubuntu, PHP-FPM and PhpStorm nigel Sat, 20/06/2015 - 10:23

PuPHPet is a terrific GUI to set up virtual machines for Web development and will build your orchestration script with Vagrant and Puppet. It now comes with a facility to include XDebug in your Vagrantfile meaning there is no longer an excuse for print_r() and error_log() debug traces in your code.

However, actually getting XDebug configured correctly with the IDE PhpStorm and Chrome browser is fiendishly fiddly, and there are many contradictory articles on the net on how to achieve it. Here is my take on how you shoud set up XDebug, in my case for Drupal development, although of course it'll work on any PHP framework or CMS. My development host is OS X on an ancient 17" Macbook Pro using the free Virtual Box product.

Firstly you should get your virtual machine working. I went for Ubuntu Trusty 64bit, Apache, with PHP 5.4 and most importantly don't forget to check the box for installing XDebug!!!

There are other resources on the Internet for installing VirtualBox, PhpStorm and Vagrant and getting your PuPHPet script up and running, so I won't cover that territory here.

There are three areas you need to configure to get XDebug working: Your Ubuntu virtual machine, your host's copy of PhpStorm, and your Chrome (or in my case Chromium) browser.

VM Configuration

Let's have a look at your virtual machine first. You will need to vagrant ssh into it and then sudo su so you have supervisor rights You need to determine where your xdebug.ini file is. To do that I built the mlocate database with the updatedb command then looked for the xdebug.ini. If you look at the output below, the file we want is /etc/php5/fpm/conf.d/20-xdebug.ini

[03:10 PM]-[root@local]-[/etc/php5/fpm/conf.d] 
# updatedb
 
[03:10 PM]-[root@local]-[/etc/php5/fpm/conf.d] 
# locate xdebug.ini
/etc/php5/cli/conf.d/20-xdebug.ini
/etc/php5/fpm/conf.d/20-xdebug.ini
/etc/php5/mods-available/xdebug.ini

Edit the /etc/php5/fpm/conf.d/20-xdebug.ini file. Yours will look something like this but you will need to change some values so don't just jump in and blithely copy/paste!

zend_extension="/usr/lib/php5/20121212/xdebug.so"
xdebug.idekey="PHPSTORM"  
xdebug.remote_host=10.0.2.15
xdebug.remote_port=100000
xdebug.remote_enable=1  
xdebug.remote_autostart=0  
xdebug.remote_handler="dbgp"

The first line is the location of the xdebug library. Use locate again to get the precise path on your VM

[03:18 PM]-[root@local]-[/etc/php5/fpm/conf.d] 
# locate xdebug.so
/usr/lib/php5/20121212/xdebug.so

The second line is the cookie ID which should be PHPSTORM to coincide with the Chrome extension's built-in configuration we will deal with later. The third line is the IP Address of the local Ethernet adapter. Discover this with ifconfig

[03:21 PM]-[root@local]-[/etc/php5/fpm/conf.d] 
# ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:34:6f:30  
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe34:6f30/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:60997 errors:0 dropped:0 overruns:0 frame:0
          TX packets:49863 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:8756579 (8.7 MB)  TX bytes:4807502 (4.8 MB)
 
eth1      Link encap:Ethernet  HWaddr 08:00:27:40:9c:6a  
          inet addr:192.168.56.101  Bcast:192.168.56.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe40:9c6a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:127578 errors:0 dropped:0 overruns:0 frame:0
          TX packets:131555 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:21689518 (21.6 MB)  TX bytes:153361934 (153.3 MB)
 
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:100167 errors:0 dropped:0 overruns:0 frame:0
          TX packets:100167 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:171493822 (171.4 MB)  TX bytes:171493822 (171.4 MB)

The fourth configuration is the port for XDebug. You should use 10000 (or rather not 9000) to avoid a clash with PHP-FPM The rest of the configuration is default and can be left as is. Once your xdebug.ini file is saved you need to restart Apache and FPM

[03:27 PM]-[root@local]-[/var/www/whatsapp-viewer]-[git feature/bulk-operations] 
# service apache2 restart
 * Restarting web server apache2                                                                         [ OK ] 
 
[03:27 PM]-[root@local]-[/var/www/whatsapp-viewer]-[git feature/bulk-operations] 
# service php5-fpm restart
php5-fpm stop/waiting
php5-fpm start/running, process 24257

PhpStorm Configuration

Go to File->Default Settings->Language & Frameworks->PHP and makde sure your Interpreter is set correctly to your VM's PHP interpreter.

Open up the Debug tab and ensure your settings look like those displayed here - note the port of 10000 matches the value we already put into the xdebug.ini file

Go to Run->Edit Configurations->Defaults->PHP Remote Debug and ensure that the server is your development domain and the ide key is PHPSTORM as we mentioned before.

Chrome Browser Configuration

Next you need to be able to switch debugging on and off on your Chrome browser. There are different ways of achieving this but by far the best is by installing the XDebug extension to the browser.

Once the extension has been added made sure the configuration represents PhpStorm

Runtime

Ok it is now time to check out our configuration. You'll need to enable debugging in your browser, set the listener to on in PhpStorm, and set some breakpoints. Let's show you!

This small image shows the PhpStorm controls in the top right corner. Make sure the fourth icon from the left (represented by a small telephone symbol) is listening for PHP debug connections

Go to your code and click in the margin to set a breakpoint which will stop the execution so you can inspect variables

In Chrome make sure you can see the XDebug icon in the far right of the URL bar and it is green and enabled

Navigate around your application in the Chrome browser and once you hit the breakpoint the execution will pause and you will be able to inspect your variables! Success!

blog terms

Drupal Development Linux Apache PHP