VirtualBox

Restoring an accidentally deleted .vagrant directory

nigel — Sat, 12 Feb 2022 11:30:59 +0000

Restoring an accidentally deleted .vagrant directory nigel Sat, 12/02/2022 - 11:30

Introduction

I woke up one morning to discover to my horror all my virtual machines had disappeared off my Macbook! This was disconcerting to say the least! Opening VirtualBox UI revealed that it had 'lost' all my VMs - the list of available VMs in the left hand pane was totally empty.

My VMs are kept on a USB memory stick which may have been a contributory factor, but the drive had not been ejected as far as I could tell.

It was early morning, I hadn't had a coffee when I started to troubleshoot, and I stupidly decided to see if I could vagrant up my most precious project - my football statistics website. Disaster! Vagrant immediately tried to download a fresh copy of Ubuntu from the cloud so I hit ctrl-c. I'm unsure what happened next - perhaps I deleted the .vagrant directory to try and fix things; perhaps Vagrant did. It was probably the former.

Regardless of blame, I was forced to find a way of regenerating the .vagrant directory so that my football VM would load correctly. These are the steps I took.

Import VMs back into Virtual Box UI

The first thing to do is import the VMs back into the Virtual Box UI. This is quite easy - click on the Tools part of the UI which will expose the large plus sign, which will open a file system dialog box, then navigate to the directory where the VMs are stored. There will be a directory per VM - open up the directory you want according to its name, then select the .vbox file. That will import the VM. You can see I've already done this for five of my VMs. I will import more of the 30 or so left if I find I need them again!

Re-associate VM with Vagrant

So importing the VM into VirtualBox won't automatically associate the VM with Vagrant. Any attempt to vagrant up will result in Vagrant fetching a new instance from the Cloud. The trick is to manually create the starting point of a .vagrant directory with an id file which will contain the VM's unique identity. The identifier can be obtained by using the VBoxManage list vms command. This can be explained better below:

$ VBoxManage list vms | grep Football
"Football VM" {b2f47455-eee0-457c-93a4-4d0f840598e0}
$ mkdir -p .vagrant/machines/default/virtualbox
$ touch .vagrant/machines/default/virtualbox/id
$ echo "b2f47455-eee0-457c-93a4-4d0f840598e0" > .vagrant/machines/default/virtualbox/id

Before I typed those commands I had already moved to the project's directory. I then got the unique identifier of the VM, recreated the directory structure and the id file, the copied the id into the file.

SSH Authentication

Doing a vagrant up will get you so far - the box will indeed start up (like if you invoked it in the Virtual Box UI) but the authentication will fail. However it's worth doing this and hitting ctrl-c when the authentication fails because Vagrant will recreate the remainder of the .vagrant directory for you and the VM will be running - essential for the next instruction.

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: [vagrant-hostsupdater] Checking for host entries
==> default: [vagrant-hostsupdater] Checking for host entries
==> default: Checking if box 'geerlingguy/ubuntu1804' version '1.1.9' is up to date...
==> default: [vagrant-hostsupdater] Checking for host entries
==> default: [vagrant-hostsupdater] Checking for host entries
==> default: [vagrant-hostsupdater] Checking for host entries
==> default: Clearing any previously set forwarded ports...
==> default: Fixed port collision for 22 => 2222. Now on port 2200.
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: hostonly
==> default: Forwarding ports...
    default: 22 (guest) => 2200 (host) (adapter 1)
==> default: [vagrant-hostsupdater] Checking for host entries
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2200
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
^C==> default: Waiting for cleanup before exiting...

The SSH configuration can be determined by:

$ vagrant ssh-config
Host default
  HostName 127.0.0.1
  User vagrant
  Port 2200
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/nigel/.vagrant.d/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL

The SSH key we need to use must be known to the local ssh-agent. Find out which keys are available with the command below. You will almost certainly have at least one entry - the one you use for git.

$ ssh-add -L

Usually the default id_rsa will be available. Pick one. Whichever is chosen, copy the private part of the key into the IdentityFile mentioned above in the output of the vagrant ssh-config command, overwriting the content already in the file. Now do:

$ vagrant halt
$ vagrant up

Wrap Up

Your system shoud now be back! My Vagrantfile runs an Ansible playbook containing my VM config - as can be seen in the screenshot above. Once that completed I could ssh into the VM as normal. All good!!

blog terms

Development VirtualBox Ansible

Launch of the BadzillaVM!

nigel — Wed, 10 Oct 2018 20:17:50 +0000

Launch of the BadzillaVM! nigel Wed, 10/10/2018 - 21:17

Welcome to the BadzillaVM

Welcome to the BadzillaVM!

The Repository can be cloned from here: https://github.com/sanddevil/badzillavm

BadzillaVM is a lightweight VM for Drupal, based heavily on Geerlingguy's DrupalVM, and just like his, built with Ansible.

BadzillaVM is significantly pared down from DrupalVM, and as such it isn't as feature-rich as DrupalVM. It does however offer something DrupalVM does not.

It offers the ability to run four different version of PHP (web) simultaneously. In addition, there is a simple mechanism to switch between PHP (cli) versions on the command line.

Currently, PHP 5.6, 7.0, 7.1 and 7.2 are supported.

This means that BadzillaVM is useful for agencies that perhaps have multiple clients running Drupal builds on differing versions of PHP. It is a breeze with BadzillaVM to configure multiple virtual hosts, each of which can be assigned any of the available versions of PHP.

BadzillaVM needs Vagrant and VirtualBox to run. You should install those two products first at their latest versions.

BadzillaVM is a simple VM with lightweight functionality. It isn't intended to support complex Drupal ecosystems. Developers seeking an advanced product set should have a look at DrupalVM instead.

BadzillaVM installs to a Ubuntu 16.04LTS virtual machine. The build contains:

nginx
PHP 5.6, 7.0, 7.1, 7.2
MariaDB Server and Client
Memcached
Drush Loader
Firewall
Java
Jenkins
Mailhog
Adminer
Pimpmylog
Composer
NodeJS
XHPRof
XDebug

This is a useful toolset for building professional standard Drupal websites

Installation Guide

1 - Install Vagrant and VirtualBox

Download and install Vagrant and VirtualBox

You can also use an alternative provider like Parallels or VMware. (Parallels Desktop 11+ requires the "Pro" or "Business" edition and the Parallels Provider, and VMware requires the paid Vagrant VMware integration plugin).

Notes:

For faster provisioning (macOS/Linux only): Install Ansible on your host machine, so Drupal VM can run the provisioning steps locally instead of inside the VM.
For stability: Because every version of VirtualBox introduces changes to networking, for the best stability, you should install Vagrant's `vbguest` plugin: `vagrant plugin install vagrant-vbguest`
NFS on Linux: If NFS is not already installed on your host, you will need to install it to use the default NFS synced folder configuration. See guides for Debian/Ubuntu, Arch, and RHEL/CentOS.
Version: Make sure you're running the latest releases of Vagrant, VirtualBox, and Ansible—as of early 2018, BadzillaVM recommends: Vagrant 2.0.x, VirtualBox 5.2.x, and Ansible 2.4.x

2 - Build the Virtual Machine

Download this project and put it wherever you want.
Create a file called `config.yml` in the directory in which you downloaded or cloned this repository.
Populate this file with your configuration (see below) This is mandatory - do not go past this step until you have added your own configuration into this file.
Type in `vagrant up`, and let Vagrant do its magic.

3 - Configuring the VM

You must configure your VM before you attempt `vagrant up` or the build will error.

A typical `config.yml` configuration will look similar to this:

---
vagrant_hostname: nigel-dev-box
 
user_vhosts_conf:
  web1:
    file_name: web1.conf 
    template_name: template-drupal8.conf.j2
    template_params:
      server_name: web1.test      
      root: /var/www/html/web1    
      php_version: 5.6
  meedjum:
    file_name: meedjum.conf 
    template_name: template-drupal8.conf.j2
    template_params:
      server_name: meedjum.test      
      root: /var/www/html/meedjum/docroot   
      php_version: 7.0
 
vagrant_synced_folders:
  - local_path: /Users/nigel/Projects/VHosts
    destination: /var/www/html
    type: nfs
    create: true

You must start the file with a line containing three hyphens.

You should then add a hostname. I've called mine nigel-dev-box.

I then define two virtual hosts, named web1 and meedjum respectively. file_name refers to the file that hold the nginx configuration. The template_name can be any defined in `roles/TroodoNmike.nginx-conf/templates`. The template_params is a list of the token substitutions which are the server_name, root, and php_version.

Finally the synchronise folders must be defined. Ensure you stick to the convention above. The values are self explanatory and they are an array so multiple values can be defined.

It is not a bad idea to ensure this configuration is saved in a vcs. To that end, I create my `config.yml` file in a sibling directory to BadzillaVM, and simlink it into the BadzillaVM directory. That way it is separate from the BadzillaVM repo, yet Vagrant will find it.

4 - Access the VM

The dashboard can be accessed by opening clicking through to the dashboard. To ssh into the VM, simply type `vagrant ssh`.

5 - Switching Versions of PHP

All four web versions of PHP run concurrently without any need for interaction. However it is necessary to manually switch between CLI versions of PHP. To do this, whilst inside the VM, simply issue the `switch-php` command:

$ switch-php
There are 4 choices for the alternative php (providing /usr/bin/php).
 
  Selection    Path             Priority   Status
------------------------------------------------------------
  0            /usr/bin/php7.2   72        auto mode
  1            /usr/bin/php5.6   56        manual mode
* 2            /usr/bin/php7.0   70        manual mode
  3            /usr/bin/php7.1   71        manual mode
  4            /usr/bin/php7.2   72        manual mode
 
Press <enter> to keep the current choice[*], or type selection number:

To help you remember which version of PHP you are currently using on the CLI, simply refer to the prompt. It will show something like:

PHP 7.0.32-1+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Oct  1 2018 11:45:35) ( NTS )
vagrant@nigel-dev-box:~$

License

This project is licensed under the MIT open source license.

Credits

Jeff Geerling's Drupal VM was both the inspiration and the starting point for this project. Huge thanks to the incredible work Jeff has done. His amazing book led me to try my own VM build. Please check his book out: Ansible for DevOps

About the Author

Badzilla (Nigel Milligan) is a London UK based Freelance Drupal Architect and Consultant and can be contacted via LinkedIn

blog terms

Drupal PHP VirtualBox BadzillaVM

Grow Mounted Partition of Headless VirtualBox

nigel — Sat, 29 Sep 2018 07:55:57 +0000

Grow Mounted Partition of Headless VirtualBox nigel Sat, 29/09/2018 - 08:55

This blog provides a tutorial for resizing (upwards) a partition on a headless VirtualBox Ubuntu Linux box. The task is accomplished in a few steps.

Clone the disk drive into VHD format
Increase the disk size of the clone
Attached the newly cloned drive into the VM
Increase the partition inside the VM

Ok so let's get started and see what we've got and where we need to get to.

$ VBoxManage list runningvms
"badzilla-d8" {86af2c61-3b68-47c5-a71e-8f45ac196cbe}
"Plex Media Server" {5efa7e7b-ae92-45a9-8244-bae451b53c0a}

It's my Plex server that has run out of disk space. I totally underestimated how much room Plex actually needs for its metadata. I'm going to extend the disk from 10GB to 30GB which should keep me going for a few years.

Clone the disk drive into VHD format

We have to clone for two reasons. Firstly when I created the disk in the first place, I created it as a fixed 10GB and it needs to be dynamic, and secondly only .VDI or .VHD format drives can be grown. Let's get our disk details.

$ VBoxManage showvminfo "Plex Media Server" | grep -i disk
Boot Device (3): HardDisk
SATA (0, 0): /home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vmdk (UUID: 0956c89c-b496-4b2e-985f-2f2d484f6e3a)

Now we know the name of our disk we can clone

$ sudo VBoxManage clonehd --format vhd "/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vmdk" "/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd"
[sudo] password for nigel: 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'vhd'. UUID: e4770417-731a-466c-9124-d1c97625cf7b

Now let's see what we have

$ ls -lash "/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vmdk" "/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd"
6.1G -rw------- 1 root  root  6.1G Sep 29 12:24 /home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd
6.0G -rw------- 1 nigel nigel 6.0G Sep 29 12:42 /home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vmdk

You can now see the two disk drives - the vmdk and the vhd format drives.

Increase the Disk Size of the VHD Clone

Now we can resize the disk to the required 30GB

$ sudo VBoxManage modifymedium '/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd' --resize 30000
[sudo] password for nigel: 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Attached the newly cloned drive into the VM

Better power off the original VM. In fact I should have done this earlier. Oops.

$ VBoxManage controlvm "Plex Media Server" poweroff

Now we can switchero the disks around with the following command.

$ VBoxManage storageattach 'Plex Media Server' \
> --storagectl "SATA" \
> --device 0 \
> --port 0 \
> --type hdd \
> --medium  '/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd'

Did it work?

$ VBoxManage showvminfo "Plex Media Server" | grep -i disk
Boot Device (3): HardDisk
SATA (0, 0): /home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vhd (UUID: e4770417-731a-466c-9124-d1c97625cf7b)

Looks encouraging so we'll start it up

$ nohup VBoxHeadless --startvm "Plex Media Server" &

and login in and check the running processes..

$ ssh nigel@192.168.0.203
nigel@192.168.0.203's password: 
$ ps -ef | grep -i plex
plex       976     1  0 15:18 ?        00:00:00 /bin/sh -c LD_LIBRARY_PATH=/usr/lib/plexmediaserver "/usr/lib/plexmediaserver/Plex Media Server"
plex       982   976  4 15:18 ?        00:00:03 /usr/lib/plexmediaserver/Plex Media Server
plex      1116   982 13 15:18 ?        00:00:08 Plex Plug-in [com.plexapp.system] /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.11.3.4803-c40bba82e /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/System.bundle
plex      1164   982  1 15:18 ?        00:00:00 /usr/lib/plexmediaserver/Plex DLNA Server
plex      1166   982  0 15:18 ?        00:00:00 /usr/lib/plexmediaserver/Plex Tuner Service /usr/lib/plexmediaserver/Resources/Tuner/Private /usr/lib/plexmediaserver/Resources/Tuner/Shared 1.11.3.4803-c40bba82e 32600 /waitmutex
plex      1213   982  8 15:18 ?        00:00:04 Plex Plug-in [com.plexapp.agents.imdb] /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.11.3.4803-c40bba82e /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/PlexMovie.bundle
nigel     1354  1325  0 15:19 pts/0    00:00:00 grep --color=auto -i plex

Increase the Partition Size Inside the VM

If we look at the file system storage space it is reporting our system drive is at 100% capacity. We'd expect that - we need to extend it to use our new space.

$ df -k | grep sda1
/dev/sda1                   6061632    5724660       6016 100% /

Now lets see what the situation is per partition

$ sudo fdisk /dev/sda
 
Welcome to fdisk (util-linux 2.27.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
 
 
Command (m for help): p
Disk /dev/sda: 29.3 GiB, 31457280000 bytes, 61440000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0660013b
 
Device     Boot    Start      End  Sectors Size Id Type
/dev/sda1  *        2048 12584959 12582912   6G 83 Linux
/dev/sda2       12587006 20969471  8382466   4G  5 Extended
/dev/sda5       12587008 20969471  8382464   4G 82 Linux swap / Solaris

Now let's see where our free space is

Command (m for help): F
Unpartitioned space /dev/sda: 19.3 GiB, 20720910336 bytes, 40470528 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
 
   Start      End  Sectors  Size
20969472 61439999 40470528 19.3G

That makes life a little difficult because there is a swap partition of 4GB immediately after our Linux 6GB. We are going to have to delete the swap, recreate it between 26GB -> 30GB and then grow our 6GB Linux partition to 26GB. Whilst this would normally be extremely dangerous, the good news is if we make a fatal mistake, we can start the process over since we still have the original vmdk drive. So let's go for it!

Firstly we delete the swap space and the extended partition

Command (m for help): d
Partition number (1,2,5, default 5): 5
 
Partition 5 has been deleted.
 
Command (m for help): d
Partition number (1,2, default 2): 2
 
Partition 2 has been deleted.

Now we are going to delete the Linux partition. What? Well deleting the partition won't delete the data providing we recreate our new partition from the very same sector on the disk.

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.
 
Command (m for help): F
Unpartitioned space /dev/sda: 29.3 GiB, 31456231424 bytes, 61437952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
 
Start      End  Sectors  Size
 2048 61439999 61437952 29.3G

Now recreate the partition only bigger!

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-61439999, default 2048): 
Last sector, +sectors or +size{K,M,G,T,P} (2048-61439999, default 61439999): 53055486
 
Created a new partition 1 of type 'Linux' and of size 25.3 GiB.

Next we recreate the extended partition with the remaining space and a new logical partition which will ultimately be the Linux Swap area

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): e
Partition number (2-4, default 2): 2
First sector (53055487-61439999, default 53055488): 
Last sector, +sectors or +size{K,M,G,T,P} (53055488-61439999, default 61439999): 
 
Created a new partition 2 of type 'Extended' and of size 4 GiB.
 
Command (m for help): n
All space for primary partitions is in use.
Adding logical partition 5
First sector (53057536-61439999, default 53057536): 
Last sector, +sectors or +size{K,M,G,T,P} (53057536-61439999, default 61439999): 
 
Created a new partition 5 of type 'Linux' and of size 4 GiB.

Now change the last partition type to Linux Swap

Command (m for help): t
Partition number (1,2,5, default 5): 
Partition type (type L to list all types): 82
 
Changed type of partition 'Linux' to 'Linux swap / Solaris'.

And finally write the partition table

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy
 
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

So we need to reboot for our changes to take effect

$ sudo su
[sudo] password for nigel: 
root@plex-media-server:/home/nigel# init 6

When we log in there is good and bad news: the filesystem isn't recognising the new size but fdisk does.

$ df -k | grep sda
/dev/sda1                   6061632    5667620      63056  99% /

and

$ sudo fdisk /dev/sda
 
Welcome to fdisk (util-linux 2.27.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
 
 
Command (m for help): p
Disk /dev/sda: 29.3 GiB, 31457280000 bytes, 61440000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0660013b
 
Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1           2048 53055486 53053439 25.3G 83 Linux
/dev/sda2       53055488 61439999  8384512    4G  5 Extended
/dev/sda5       53057536 61439999  8382464    4G 82 Linux swap / Solaris

Our final activity to make the additional space available for us is to use the resize2fs command:

$ sudo resize2fs /dev/sda1
resize2fs 1.42.13 (17-May-2015)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 2
The filesystem on /dev/sda1 is now 6631679 (4k) blocks long.

And now we have it:

$ df -k | grep sda
/dev/sda1                  25978144    5672208   19165592  23% /

Cleaning-up Activities

Since we deleted the swap partition, and moved it, we need to create the actual swap with the following command

$ sudo mkswap /dev/sda5

In addition, we could either delete the original vmdk disk or archive it somewhere to get 6GB space back on the parent machine.

blog terms

Hardware Linux VirtualBox

Drupal 8 and DRUD Tech's ddev Local Development Environment

nigel — Sat, 10 Mar 2018 21:12:50 +0000

Drupal 8 and DRUD Tech's ddev Local Development Environment nigel Sat, 10/03/2018 - 21:12

My favourite presentation at DrupalCamp London 2018 was Turbocharge Your Agency given by DRUD Tech co-founder Kevin Bridges and Jeffrey A. "jam" McGuire. They talked at length about the container solution ddev which enables developers to manage their local environments and thus optimise their time and in addition save their configuration under source control. Furthermore the demonstration showed that it was possible to flip between different configurations with different Drupal versions easily and quickly. The solution found here uses Docker and is wrapped by the lightning fast Golang.

I've been thinking for a while about moving from my existing hand crafted Virtual Machine towards Docker. My VM's configuration isn't under source control management, and has built up over the years into quite a complex beast. I have Jenkins, phing, composer, Drupal console, Mail Catcher, Compass, node, npm, Python, nginx, Apache, MariaDB and MongoDB that I can remember off the top of my head. It's becoming a difficult proposition to keep it all working. When I need to run a different version of PHP I manually switch simlinks around. Oh for something easier.

So I elected to investigate whether Drud Tech's offering could possibly help out. The demonstration given by their guys at DrupalCamp London 2018 showed the ease of moving between D7 and D8 repos. However they were fresh builds and for the tool to be of value to me, it would have to work from pre-existing codebases.

Installing a VirtualBox VM

This was purely a Proof of Concept so as a consequence I needed to keep anything I did in a ring-fenced environment. Therefore a VirtualBox Ubuntu 16.04LTS image was used as a starting point. I won't provide screenshots of the entire build process since I have previous blogs to cover this.

Once the build has completed and I have logged in I issue a ifconfig command to determine the IP address which is 10.0.2.15 (second screenshot). This will need to be used with the network configuration.

When I was building the VM I kept the default network configuration which is NAT. I have moved away from using Bridged Adapter. A bridged adapter on the face of it is very convenient - it automatically gets an IP address via DHCP from my ISP provided router at home, in the 192.168.0.0 range. However this falls foul at client sites with security policies prohibiting unknown MAC Addresses, and Internet cafes that don't use the same IP range - it simply won't work.

The solution is to use NAT and use the port forwarding facility. Navigate to Settings -> Network -> Advanced -> Port Forwarding and add values for ssh as per the third screenshot above.

I can now ssh into the ddev VM.

$ ssh -p 2222 nigel@127.0.0.1
The authenticity of host '[127.0.0.1]:2222 ([127.0.0.1]:2222)' can't be established.
ECDSA key fingerprint is SHA256:VRNxWpagSl2DGz7dmgRjBqdDdadXITdPPkItrMR9k38.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
nigel@127.0.0.1's password: 
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-87-generic x86_64)
 
 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
 
137 packages can be updated.
69 updates are security updates.
 
 
Last login: Mon Mar  5 21:50:35 2018
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

Install Docker

Don't be tempted to install the Docker version that is bundled in the Ubuntu default repos. Follow the instructions on the Docker site for installing the CE version of Docker from their private repo.

nigel@ddev:~$ sudo apt-get update
[sudo] password for nigel: 
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease
Reading package lists... Done

Install packages to allow apt to use a repository over HTTPS

nigel@ddev:~$ sudo apt-get install \
>     apt-transport-https \
>     ca-certificates \
>     curl \
>     software-properties-common
Reading package lists... Done
Building dependency tree       
Reading state information... Done
apt-transport-https is already the newest version (1.2.25).
ca-certificates is already the newest version (20170717~16.04.1).
curl is already the newest version (7.47.0-1ubuntu2.6).
software-properties-common is already the newest version (0.96.20.7).
0 to upgrade, 0 to newly install, 0 to remove and 2 not to upgrade.

That's good. Nothing to do there. Now let's add Docker’s official GPG key:

nigel@ddev:~$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK

Now I need to verify that we have the key with the fingerprint 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88, by searching for the last 8 characters of the fingerprint.

nigel@ddev:~$ sudo apt-key fingerprint 0EBFCD88
pub   4096R/0EBFCD88 2017-02-22
      Key fingerprint = 9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid                  Docker Release (CE deb) <docker@docker.com>
sub   4096R/F273FCD8 2017-02-22

All good. Next the following command to set up the stable repository.

nigel@ddev:~$ sudo add-apt-repository \
>    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
>    $(lsb_release -cs) \
>    stable"

Update apt again

nigel@ddev:~$ sudo apt-get update
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 https://download.docker.com/linux/ubuntu xenial InRelease [65.8 kB]
Get:6 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages [3,329 B]
Fetched 69.1 kB in 0s (140 kB/s)
Reading package lists... Done

Again we are looking good - we can see it's picked up the Docker repo ok. Let's install the latest version of Docker.

nigel@ddev:~$ sudo apt-get update

Let's check it's working..

nigel@ddev:~$ sudo docker run hello-world
[sudo] password for nigel: 
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
ca4f61b1923c: Pull complete 
Digest: sha256:083de497cff944f969d8499ab94f07134c50bcf5e6b9559b27182d3fa80ce3f7
Status: Downloaded newer image for hello-world:latest
 
Hello from Docker!
This message shows that your installation appears to be working correctly.

Success! Now we need docker-compose installing. Again, don't be tempted to install from the standard Ubuntu repos, because if you do with Ubuntu 16.04LTS you'll end up with docker-compose version 1.8.0 which does not meet the ddev requirements. So do it the proper way!

nigel@ddev:~$ sudo curl -L https://github.com/docker/compose/releases/download/1.19.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   617    0   617    0     0    795      0 --:--:-- --:--:-- --:--:--   795
100 8288k  100 8288k    0     0   355k      0  0:00:23  0:00:23 --:--:--  196k

Now make it executable

nigel@ddev:~$ sudo chmod +x /usr/local/bin/docker-compose

Let's see the version

nigel@ddev:~$ docker-compose --version
docker-compose version 1.19.0, build 9e633ef

Neat! We are almost there. Now we need to make sure Docker starts every time we boot the machine.

nigel@ddev:~/projects/meedjum$ sudo systemctl enable docker
Synchronizing state of docker.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable docker

Before I am done here, I need to add myself to the Docker group so I can execute Docker commands (or rather ddev can on my behalf)

nigel@ddev:~/projects/meedjum$ sudo usermod -aG docker $(whoami)

Rebooting will now serve two purposes - check that the above command works which will start Docker, and make sure my new group status has been picked up.

nigel@ddev:~/projects/meedjum$ sudo su
root@ddev:/home/nigel/projects/meedjum# init 6

Installing ddev

There are two ways of installing ddev - by downloading from the repository's releases - or use the install script in the ddev documentation. I opted for the latter.

nigel@ddev:~$ curl https://raw.githubusercontent.com/drud/ddev/master/install_ddev.sh | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2545  100  2545    0     0   5821      0 --:--:-- --:--:-- --:--:--  5823
ddev_linux.v0.15.0.tar.gz: OK
Download verified. Ready to place ddev in your /usr/local/bin.
Running "sudo mv /tmp/ddev /usr/local/bin/" Please enter your password if prompted.
[sudo] password for nigel: 
Bash completion for ddev was not installed. You may manually install /tmp/ddev_bash_completion.sh in your bash_completions.d directory.
ddev is now installed. Run "ddev" to verify your installation and see usage.

Installed - now check our installation

nigel@ddev:~$ ddev
This Command Line Interface (CLI) gives you the ability to interact with the ddev to create a development environment.
Usage:
ddev [command]
Available Commands:
auth-pantheon Provide a machine token for the global pantheon auth.
config Create or modify a ddev project configuration in the current directory
describe Get a detailed description of a running ddev project.
exec Execute a shell command in the container for a service. Uses the web service by default.
help Help about any command
hostname Manage your hostfile entries.
import-db Import the database of an existing project to the dev environment.
import-files Import the uploaded files directory of an existing project to the default public upload directory of your project.
list List projects
logs Get the logs from your running services.
pull Import files and database using a configured provider plugin.
remove Remove the development environment for a project.
restart Restart the development environment for a project.
ssh Starts a shell session in the container for a service. Uses web service by default.
start Start a ddev project.
stop Stop the development environment for a project.
version print ddev version and component versions
Flags:
-h, --help help for ddev
-j, --json-output If true, user-oriented output will be in JSON format.
Use "ddev [command] --help" for more information about a command.

We have it!

Getting Started with an Existing Drupal 8 Project

This blog has been built using Drupal 8 - so it made sense to use this for my first project. Let's create a directory for my projects and git clone. At this point I had already created a new .ssh key for my ddev VM and added it to my GitHub profile.

nigel@ddev:~$ mkdir projects
nigel@ddev:~$ cd projects/
nigel@ddev:~/projects$ git clone git@github.com:sanddevil/meedjum.git
Cloning into 'meedjum'...

Now let's cd into the project and config ddev for our Drupal 8 through its interactive shell.

nigel@ddev:~/projects/meedjum$ ddev config
Creating a new ddev project config in the current directory (/home/nigel/projects/meedjum) 
Once completed, your configuration will be written to /home/nigel/projects/meedjum/.ddev/config.yaml
 
Project name (meedjum): 
 
The docroot is the directory from which your site is served. This is a relative path from your project root (/home/nigel/projects/meedjum) 
You may leave this value blank if your site files are in the project root 
Docroot Location (docroot): 
Found a php codebase at /home/nigel/projects/meedjum/docroot. 
Project Type [php, drupal6, drupal7, drupal8, wordpress, typo3, backdrop] (php): drupal8
Configuration complete. You may now run 'ddev start'.

ddev is suggesting I should run start, so why not?

nigel@ddev:~/projects/meedjum$ ddev start
Network ddev_default created 
Starting environment for meedjum... 
ddev needs to add an entry to your hostfile.
It will require root privileges via the sudo command, so you may be required
to enter your password for sudo. ddev is about to issue the command: 
    sudo /usr/local/bin/ddev hostname meedjum.ddev.local 127.0.0.1 
Please enter your password if prompted. 
Running Command  Command=sudo /usr/local/bin/ddev hostname meedjum.ddev.local 127.0.0.1
[sudo] password for nigel: 
Pulling db (drud/mariadb-local:v0.8.0)... 
Creating ddev-meedjum-db ... done
Pulling dba (drud/phpmyadmin:v0.2.0)... 
Creating ddev-meedjum-web ... done
Creating ddev-meedjum-dba ...  
Creating ddev-meedjum-web ...  
 
Network ddev_default is external, skipping 
Pulling ddev-router (drud/ddev-router:v0.5.0)... 
Creating ddev-router ... done
 
Successfully started meedjum 
Your project can be reached at http://meedjum.ddev.local and https://meedjum.ddev.local

Installing Composer

Drupal 8 requires composer and that needs installing on our clean system. There are plenty of tutorials online on how to achieve this, so I'll whizz through this quickly.

nigel@ddev:~$ sudo apt-get update
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:5 https://download.docker.com/linux/ubuntu xenial InRelease
Reading package lists... Done

Install the dependencies

nigel@ddev:~$ sudo apt-get install curl php-cli php-mbstring git unzip
Reading package lists... Done

Download composer

nigel@ddev:~$ curl -sS https://getcomposer.org/installer -o composer-setup.php

Install it globally

nigel@ddev:~$ sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
All settings correct for using Composer
Downloading...
 
Composer (version 1.6.3) successfully installed to: /usr/local/bin/composer
Use it: php /usr/local/bin/composer

Now check it's been installed correctly

nigel@ddev:~$ composer
   ______
  / ____/___  ____ ___  ____  ____  ________  _____
 / /   / __ \/ __ `__ \/ __ \/ __ \/ ___/ _ \/ ___/
/ /___/ /_/ / / / / / / /_/ / /_/ (__  )  __/ /
\____/\____/_/ /_/ /_/ .___/\____/____/\___/_/
                    /_/
Composer version 1.6.3 2018-01-31 16:28:17

Building Out the Drupal 8 Project

The ddev start command creates a new subdirectory called .ddev in the project's root directory. Within the directory we have config.yaml and docker-compose.yaml. The config file contains the configuration of our ddev project such as the version of PHP, webpage url, type of project (drupal8) amongst other items. The docker yaml file contains the container configuration.

Interaction with the ddev project can be accomplished in two ways, either with

ddev exec command

or by spawning a shell with

ddev ssh

and using exit to terminate the interactive session.

Our first requirement is to get dependencies into our D8 project since it is vendor-ised and uses composer as its dependency management.

Ok my Drupal core was on 8.4.4 and 8.4.5 and then 8.5 had been released so it needed updating, and in my case would update to 8.4.5 initially

nigel@ddev:~/projects/meedjum/docroot$ composer update --with-dependencies
Gathering patches for root package.
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 51 installs, 0 updates, 0 removals
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.

Now since there were schema changes, those need to be applied.

nigel@ddev:~/projects/meedjum/docroot$ ddev exec drush updatedb
Performing node_update_8401                                          [ok] 
Cache rebuild complete.                                              [ok] 
Finished performing updates.                                         [ok] 
The following updates are pending:
 
node module : 
  8401 -   Run a node access rebuild, if required. 
 
Do you wish to run all pending updates? (y/n): y

Settings.php

This is where things started to go awry. I already have a fully constructed settings.php and settings.local.php. The settings.php file contains my Memcached settings. However the memcache(d) PHP libraries aren't shipped in the ddev product. Obviously by that fact ddev doesn't support Memcached.

When I attempted a drush status command I was inundated with errors.

nigel@ddev:~/projects/meedjum/docroot/sites/default$ vi settings.local.php
nigel@ddev:~/projects/meedjum/docroot/sites/default$ ddev exec drush status
Drupal\memcache\MemcacheException: No Memcache extension found in /var/www/html/docroot/modules/contrib/memcache/src/DrupalMemcacheFactory.php on line 162 #0 /var/www/html/docroot/modules/contrib/memcache/src/DrupalMemcacheFactory.php(62): Drupal\memcache\DrupalMemcacheFactory->initialize()

As a consequence I commented out the Memcache settings

// Memcache
//Memcache configuration
//$settings['memcache']['servers'] = ['127.0.0.1:11211' => 'default'];
//$settings['memcache']['bins'] = ['default' => 'default'];
//$settings['memcache']['key_prefix'] = 'badzillad8';
//$settings['cache']['default'] = 'cache.backend.memcache';
?>

Importing the Database

The next problem is ddev looks for the db credentials in settings.php whereas I have my credentials in my settings.local.php file. I changed the credentials in the settings.local.php file and I copied my database file to the /tmp directory.

nigel@ddev:~/projects/meedjum/docroot/sites/default$ ddev import-db  --extract-path /tmp
Provide the path to the database you wish to import. 
Import path: /tmp/db-2018-03-10.sql.gz
Failed to import database for meedjum: failed to write settings file for meedjum: Failed to get Drupal settings file path: settings files already exist and are being managed by the user

More errors! The import failed. I thought it could be because it required the credentials in settings.php but that didn't work either. So I decided to use Drush instead:

nigel@ddev:~/projects/meedjum/docroot/sites/default$ ddev exec drush sql-cli < /tmp/db-2018-03-10.sql

HTTP Port Forwarding

Just like our earlier forwarding of the ssh port so we can log into the VM, we also have to forward http port 80 traffic. To do this, we go back to the virtual machine network configuration, and set up a redirect from port 9090 (somewhat arbitrary) on the host to port 80 on the guest. See the first screenshot. This will take us to the the default website on the VM. We need to explicitly tell a browser which site it is - to do that we add an entry on the machine where we will be using our browser - which in my case will be my MacBook.

So I have added the entry to /etc/hosts of the working ddev domain name and the localhost IP address

127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost
 
127.0.0.1 meedjum.ddev.local

Now we need to reboot our VM and then login back in and issue a ddev start on our project. We can also check whether everything is good with a docker ps command.

root@ddev:/home/nigel# init 6
Connection to 127.0.0.1 closed by remote host.
Connection to 127.0.0.1 closed.
Nigels-MacBook-Pro:tmp nigel$ ssh -p 2222 nigel@127.0.0.1
nigel@ddev:~$ cd projects/meedjum/
nigel@ddev:~/projects/meedjum$ ddev start
Starting environment for meedjum... 
nigel@ddev:~/projects/meedjum$ docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS                             PORTS                                                                                      NAMES
5fd9a60cb90c        drud/ddev-router:v0.5.0           "/app/docker-entrypo…"   10 seconds ago      Up 10 seconds (healthy)            0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8025->8025/tcp, 0.0.0.0:8036->8036/tcp   ddev-router
cea77b7cbc6b        drud/nginx-php-fpm-local:v1.1.0   "/start.sh"              7 hours ago         Up 11 seconds (healthy)            443/tcp, 0.0.0.0:32771->80/tcp, 0.0.0.0:32770->8025/tcp                                    ddev-meedjum-web
2dc08e116820        drud/phpmyadmin:v0.2.0            "/run.sh phpmyadmin"     7 hours ago         Up 11 seconds (health: starting)   0.0.0.0:32769->80/tcp                                                                      ddev-meedjum-dba
f239a52ae806        drud/mariadb-local:v0.8.0         "/docker-entrypoint.…"   7 hours ago         Up 12 seconds (healthy)            0.0.0.0:32768->3306/tcp                                                                    ddev-meedjum-db

We can now point a web browser at the url we typed into our /etc/hosts file - see third image above. Ok - so we are close but not quite there yet! Looks like this could be the old trusted host pattern issue since it doesn't know about my domain meedjum.ddev.test.

Add ddev Domain to Trusted Hosts in Settings.php

...or rather in my case my settings.local.php file. I added the ddev domain name to the trusted hosts pattern array

$settings['trusted_host_patterns'] = [
  'meedjum.test',
  'meedjum.ddev.local',
];
?>

Copying the Assets

Adding the trusted host entry now means the web site loads successfully but of course the assets are missing and need copying down from my prod site. ddev provides a command for this:

nigel@ddev:~/projects/meedjum/docroot$ ddev import-files --src=/tmp/files.tar.gz
Successfully imported files for meedjum

100% my fault, but my file structure ended up as sites/default/files/files because I'd tarballed my prod files directory within the tarball, instead of just its contents. So I moved everything back one directory.

nigel@ddev:~/projects/meedjum/docroot$ mv sites/default/files/files/* sites/default/files/.
nigel@ddev:~/projects/meedjum/docroot$ mv sites/default/files/files/.htaccess sites/default/files/.
nigel@ddev:~/projects/meedjum/docroot$ rmdir sites/default/files/files

Completed Project

Ok so in the interim I fixed the missing Font Awesome library - not something I'd noticed before, but obviously it's been gitignored and it isn't in the composer.json file. Anyhoo the site is now fully functional and you can see from the URL bar I need to append the port :9090 as per my previous port forwarding fix.

Verdict

Playing with ddev was an interesting experience, not without challenges, but ultimately rewarding. Here are some of the points I made as I went along.

The ddev exec command is run as root. This is a problem because it basically means using composer with it - such as ddev exec composer update - would trash the file permissions of my codebase. Ownerships would become root, and that would lead to problems down the line. Composer should never be run as root. So the net effect is composer needs to be installed on the host system. Ideally I wanted a totally un-poluted filesystem without installing anything on the host beyond ddev. It also means since composer has a dependency on PHP, that needs to be installed, and now we have a system with different PHP CLI and FPM versions.
I couldn't get the ddev import-db command to work. A minor inconvenience and of course it is easy to use alternatives with the command line or drush.
Now I've got a working system - what next? I have a welter of phing targets to deploy my code, usually invoked by Jenkins. I could add phing at project level but how would I get Jenkins docker-ised up and running across all projects? I sense that could be tricky.
I can see ddev's value. I would place it in a small agency with both legacy D6 and D7 support projects alongside new D8 and WP builds. I can't see it as an enterprise solution - there is too much missing such as Memcache and the previously mentioned phing and Jenkins.

But all in all I really enjoyed my Saturday watching the football whilst playing with ddev and writing this blog - despite my teams losing!

blog terms

Drupal Drupal 8 Drupal 7 Drupal 6 devops Docker VirtualBox

Move Plex Media Server to Linux VM

nigel — Sun, 18 Feb 2018 13:45:09 +0000

Move Plex Media Server to Linux VM nigel Sun, 18/02/2018 - 13:45

I've been playing with Plex Media Server for the last few weeks on my Linux laptop, and much to my surprise it seems to play back any video format I throw at it including DivX and XviD which is the format of a great many of my collection of movies. So buoyed by this I have decided to move the entire Plex configuration to a dedicated Ubuntu virtual machine which will reside on my Intel NUC I reviewed earlier.

My Intel NUC is headless, and whilst it is possible to create a VM image on the command line, it is much more convenient to create it on a GUI elsewhere and move it over to the NUC. To that end, I created the VM on my Mac knowing the network configuration would ultimately have to change when the VM is in situ on the NUC.

VM Installation

The steps are quite straightforward so there isn't much need for narrative. I had previously downloaded the Ubuntu 16.04 LTS server iso image (more of that in a minute).

I decided to go for 4GB RAM and 10GB fixed disk size which should be more than adequate.

Initial Network Connection

The initial network connection is set as above. We need a bridged network so I can use my pre-existing 192.168 network in my property. Obviously when the VM is moved to the Intel NIC the AirPort won't be a valid setting at that point - so we'll have to change it then.

Ubuntu 16.04 Installation

As mentioned previously, I had already downloaded the Ubuntu 16.04 LTS iso image which is selected for the installation above. Most of the installation screens haven't been captured for brevity because they are straightforward, but note I have selected openSSH server in the software packages to install - this is important since we will need to ssh into our headless Plex server.

We now need to edit the /etc/network/interfaces file to change the default DHCP IP assignment to static. Your configuration will obviously be different from mine here. I've elected to go for the address of 192.168.0.203 for my server.

nigel@plex-media-server:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
 
source /etc/network/interfaces.d/*
 
# The loopback network interface
auto lo
iface lo inet loopback
 
# The primary network interface
auto enp0s3
# iface enp0s3 inet dhcp
iface enp0s3 inet static
 address 192.168.0.203
 netmask 255.255.255.0
 network 192.168.0.1
 broadcast 192.168.1.255
 gateway 192.168.0.1
 dns-nameservers 90.207.238.97

Export the VM

I have already covered how to export a VM image in a previous blog here, so I don't need to go into too much detail in this blog. My VM has been exported and now needs to be copied to the NUC host machine using scp.

Nigels-MacBook-Pro:Documents nigel$ scp Plex\ Media\ Server.ova nigel@192.168.0.202:/home/nigel/VM/.
nigel@192.168.0.202's password: 
Plex Media Server.ova                                                                        100%  604MB   4.1MB/s   02:27    
Nigels-MacBook-Pro:Documents nigel$

Now log into the host machine and cd into the correct directory and import the VM.

nigel@devbox:~/VM$ VBoxManage import Plex\ Media\ Server.ova 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interpreting /home/nigel/VM/Plex Media Server.ova...
OK.
Disks:
  vmdisk1       10737418240     -1      http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized       Plex Media Server-disk001.vmdk  -1      -1
 
Virtual system 0:
 0: Suggested OS type: "Ubuntu_64"
    (change with "--vsys 0 --ostype "; use "list ostypes" to list all possible values)
 1: Suggested VM name "Plex Media Server"
    (change with "--vsys 0 --vmname ")
 2: Number of CPUs: 1
    (change with "--vsys 0 --cpus ")
 3: Guest memory: 4096 MB
    (change with "--vsys 0 --memory ")
 4: Sound card (appliance expects "", can change on import)
    (disable with "--vsys 0 --unit 4 --ignore")
 5: USB controller
    (disable with "--vsys 0 --unit 5 --ignore")
 6: Network adapter: orig Bridged, config 3, extra slot=0;type=Bridged
 7: CD-ROM
    (disable with "--vsys 0 --unit 7 --ignore")
 8: IDE controller, type PIIX4
    (disable with "--vsys 0 --unit 8 --ignore")
 9: IDE controller, type PIIX4
    (disable with "--vsys 0 --unit 9 --ignore")
10: SATA controller, type AHCI
    (disable with "--vsys 0 --unit 10 --ignore")
11: Hard disk image: source image=Plex Media Server-disk001.vmdk, target path=/home/nigel/VirtualBox VMs/Plex Media Server/Plex Media Server-disk001.vmdk, controller=10;channel=0
    (change target path with "--vsys 0 --unit 11 --disk path";
    disable with "--vsys 0 --unit 11 --ignore")
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Successfully imported the appliance.

Now we need to modify the ethernet card settings and the audio as per my previous blog

$ VBoxManage modifyvm "Plex Media Server" --nic1 bridged --bridgeadapter1 enp3s0
$ VBoxManage modifyvm "Plex Media Server" --audio none

Ok now we need to be sure we are in the correct directory and start up our machine. Once this has been done, it will be possible to ssh into it.

$ pwd
/home/nigel/VirtualBox VMs
$ VBoxHeadless --startvm "Plex Media Server" &

Let's try to log into 192.168.0.203

nigel@nigel-Aspire-VN7-793G:/etc$ ssh nigel@192.168.0.203
The authenticity of host '192.168.0.203 (192.168.0.203)' can't be established.
ECDSA key fingerprint is SHA256:BLRRmyijEFgtS+KnkXQUudkdsMskpLrzR19zBugQ2ww.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.203' (ECDSA) to the list of known hosts.
nigel@192.168.0.203's password: 
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-87-generic x86_64)
 
 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
 
127 packages can be updated.
64 updates are security updates.
 
 
Last login: Sun Feb 18 19:58:43 2018 from 192.168.0.12
nigel@plex-media-server:~$

Success!

fstab entry to connect to CIFS partition

My FreeNAS server has a CIFS mount point which contains my media collection. It would therefore be useful for the drive to be automatically loaded on boot. A prerequisite for this is the cifs-utils package which will need to be installed.

$ sudo apt-get install cifs.utils

Next we need to create the mount point directory which will be /mnt/freenas

$ sudo mkdir /mnt/freenas

It would useful to check we can connect first on the command line. So lets type out the required mount command, but first we need to create a credentials file for FreeNAS. This will be owned by the root user and should be placed in the root user's directory and made read only for root.

# cat .cifs 
username=xxxxxx
password=xxxxxx
# ls -las .cifs
4 -r-------- 1 root root 39 Feb 18 21:37 .cifs

Now the actual mount command and a follow up directory listing to make sure it has been mounted correctly.

$ sudo mount.cifs //192.168.0.200/personal  /mnt/freenas    -o vers=3.0,credentials=/root/.cifs,uid=1000
$ ls -las /mnt/freenas | grep Documents
     0 drwxr-xr-x 2 nigel root         0 Aug 14  2017 Documents

Now we need to unmount this and add the entry to the /etc/fstab file.

$ sudo umount /mnt/freenas
$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#                
# / was on /dev/sda1 during installation
UUID=600eb961-4caa-4a24-b8cf-c35efdd71ca4 /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=523ed080-41af-4dab-839f-070f40c26ada none            swap    sw              0       0
 
# CIFS FreeNas mount
//192.168.0.200/personal        /mnt/freenas    cifs    credentials=/root/.cifs,uid=1000,vers=3.0       0      0
$ sudo mount -a
$ ls -las /mnt/freenas | grep Documents
     0 drwxr-xr-x 2 nigel root         0 Aug 14  2017 Documents

That's another job done! Finally we can think about moving the Plex configuration over to our new VM.

Commence the Plex Move from Source to Destination

Firstly navigate to Settings->Server->library and disable the setting Empty trash automatically after every scan as shown above. Save this setting change.

Stop the Plex Media Server running on the source machine. This is achieved on the command line with

$ sudo service plexmediaserver stop
[sudo] password for nigel: 
$ ps -ef | grep plex
nigel     6274  1647  0 09:51 pts/1    00:00:00 grep --color=auto plex
$

Note I double checked that there were no Plex services running just in case.

install Plex on Destination Machine

In my case my destination machine is a VM on my Intel NUC so let's see if the VM is currently running.

nigel@devbox:~$ VBoxManage list vms
"badzilla-d8" {86af2c61-3b68-47c5-a71e-8f45ac196cbe}
"Plex Media Server" {5efa7e7b-ae92-45a9-8244-bae451b53c0a}
nigel@devbox:~$ VBoxManage list runningvms
nigel@devbox:~$

So the good news is it is there. The bad news is it isn't currently running. Let's spark it up.

nigel@devbox:~$ VBoxHeadless --startvm "Plex Media Server" &

Ok let's ssh into the VM so we can install the new copy of Plex

nigel@devbox:~$ ssh nigel@192.168.0.203
The authenticity of host '192.168.0.203 (192.168.0.203)' can't be established.                                  
ECDSA key fingerprint is SHA256:BLRRmyijEFgtS+KnkXQUudkdsMskpLrzR19zBugQ2ww.                                    
Are you sure you want to continue connecting (yes/no)? yes                                                      
Warning: Permanently added '192.168.0.203' (ECDSA) to the list of known hosts.                                  
nigel@192.168.0.203's password:                                                                                 
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-87-generic x86_64)                                               
 
 * Documentation:  https://help.ubuntu.com                                                                      
 * Management:     https://landscape.canonical.com                                                              
 * Support:        https://ubuntu.com/advantage                                                                 
 
19 packages can be updated.
10 updates are security updates.
 
 
Last login: Sun Feb 18 22:11:59 2018 from 192.168.0.14
nigel@plex-media-server:~$

On a separate machine, download the latest Ubuntu 64 bit Plex Server, and scp it into the destination machine file system.

nigel@nigel-Aspire-VN7-793G:~/Downloads$ scp plexmediaserver_1.11.3.4803-c40bba82e_amd64.deb nigel@192.168.0.203:/tmp
nigel@192.168.0.203's password: 
plexmediaserver_1.11.3.4803-c40bba82e_amd64.deb                               100%  104MB 817.5KB/s   02:10    
nigel@nigel-Aspire-VN7-793G:~/Downloads$

Now switch back to the term window of the destination machine and install Plex.

nigel@plex-media-server:~$ sudo dpkg -i /tmp/plexmediaserver*.deb
[sudo] password for nigel: 
Selecting previously unselected package plexmediaserver.
(Reading database ... 93857 files and directories currently installed.)
Preparing to unpack .../plexmediaserver_1.11.3.4803-c40bba82e_amd64.deb ...
Unpacking plexmediaserver (1.11.3.4803-c40bba82e) ...
Setting up plexmediaserver (1.11.3.4803-c40bba82e) ...
Created symlink from /etc/systemd/system/multi-user.target.wants/plexmediaserver.service to /lib/systemd/system/plexmediaserver.service.
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
nigel@plex-media-server:~

The installation process will invoke Plex so stop the service

nigel@plex-media-server:~$ sudo service plexmediaserver stop

Copy to Destination Machine

We now need to copy Plex from the source to the destination machine. The best method is to scp the entire Plex file structure from one machine to the other. The file structure is at /var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/ and is owned by the user plex as you can see here:

nigel@nigel-Aspire-VN7-793G:/$ ls -las /var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/
total 44                                                                                                        
4 drwxr-xr-x 10 plex plex 4096 Feb 24 09:51 .                                                                   
4 drwxr-xr-x  3 plex plex 4096 Jan 28 20:11 ..                                                                  
4 drwxr-xr-x  5 plex plex 4096 Feb 24 09:45 Cache                                                               
4 drwxr-xr-x  3 plex plex 4096 Jan 28 20:11 Codecs                                                              
4 drwxr-xr-x  3 plex plex 4096 Jan 28 20:11 Crash Reports                                                       
4 drwxr-xr-x  3 plex plex 4096 Feb 24 09:14 Logs                                                                
4 drwxr-xr-x  3 plex plex 4096 Jan 28 20:18 Media                                                               
4 drwxr-xr-x  6 plex plex 4096 Jan 30 21:44 Metadata                                                            
4 drwxr-xr-x  3 plex plex 4096 Feb 22 20:25 Plug-ins                                                            
4 drwxr-xr-x  7 plex plex 4096 Jan 28 20:11 Plug-in Support                                                     
4 -rw-------  1 plex plex 1002 Feb 24 09:45 Preferences.xml
nigel@nigel-Aspire-VN7-793G:/$

So to get all the permissions and ownerships correct during the copy we should assign a password to the user plex so when we scp we can provide the prompted for password.

root@plex-media-server:/home/nigel# passwd plex 
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
root@plex-media-server:/home/nigel#

Ok - so if we switch back to the source machine we need to execute the scp command like this:

nigel@nigel-Aspire-VN7-793G:/$ sudo -H -u plex scp -rp  /var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/* plex@192.168.0.203:"/var/lib/plexmediaserver/Library/Application\\ Support/Plex\\ Media\\ Server/."

Note that I am running the scp as the plex user on my source machine through the superuser, and also note the double back-whacks and quotes on the destination directory structure.

Next we should start the Plex service on the destination machine and check it came up ok.

nigel@plex-media-server:~$ sudo service plexmediaserver start
nigel@plex-media-server:~$ ps -ef | grep plex
plex     17501     1  0 13:19 ?        00:00:00 /bin/sh -c LD_LIBRARY_PATH=/usr/lib/plexmediaserver "/usr/lib/plexmediaserver/Plex Media Server"
plex     17502 17501 22 13:19 ?        00:00:02 /usr/lib/plexmediaserver/Plex Media Server
plex     17517 17502 89 13:19 ?        00:00:07 Plex Plug-in [com.plexapp.system] /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.11.3.4803-c40bba82e /usr/lib/plexmediaserver/Resources/Plug-ins-c40bba82e/System.bundle
nigel    17541  1453  0 13:19 pts/0    00:00:00 grep --color=auto plex
nigel@plex-media-server:~$

Web Browser Destination Access

You should now be able to point your browser to Plex's new home. You need to change the URL to include your IP address, which in my case is 192.168.0.203 with a port of 32400. Yours should be similar but probably not identical. Check out the first image above for the entire URL although you won't need to include the index.html.

You may have to log in again but mine went straight to the 'Recently Added' view as you can see in the second image above. So my media and my libraries have been found ok. This shouldn't be a surprise - don't forget earlier in the tutorial I connected the VM to my FreeNAS where the media is held - and the canonical path is exactly the same to the media on the VM (destination) as it is on my source machine (my laptop).

Empty trash automatically

Now we know everything is working ok, we should re-enable the emptying of trash - see the screenshot above.

Tidying up

A little housekeeping is required. We now have two systems using the same BadzillaPlex name - so let's remove it from my laptop. To do that issue:

$ sudo dpkg --remove plexmediaserver
[sudo] password for nigel: 
(Reading database ... 334962 files and directories currently installed.)
Removing plexmediaserver (1.10.1.4602-f54242b6b) ...
Removed symlink /etc/systemd/system/multi-user.target.wants/plexmediaserver.service.
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for desktop-file-utils (0.22-1ubuntu5.1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...

blog terms

Hardware Linux VirtualBox Plex

Moving a VM Dev Environment to VirtualBox Headless

nigel — Mon, 15 Jan 2018 11:00:12 +0000

Moving a VM Dev Environment to VirtualBox Headless nigel Mon, 15/01/2018 - 11:00

I've been using a VM image on my Macbook for a while now, and using it as my Drupal development environment. It has been handcrafted with MariaDB, NGINX, Jenkins and other utilities that have built up over time such as Mailcatcher and npm. It's become increasingly inconvenient having the VM on the laptop I use for client work, so I decided to move it to my Intel NUC machine. This will provide a centralised location in my apartment to access the VM from any of my other laptops and devices.

However since my Intel NUC doesn't have an installed GUI since it is a server, a headless solution is required. To manage this I will need VirtualBox Headless - here is the tutorial on how to achieve this. My NUC machine was installed with Ubuntu 17.04 (Zesty Zapus) but other flavours of Debian based Linux will be similar.

Installing VirtualBox

To install VirtualBox, login to the host box over the local network (in my case my Intel NUC) and sudo su. Then add the repository and the public key with

root@devbox:/# wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | apt-key add -
OK
root@devbox:/#

Update and install. Note - there's every chance you will be presented with a list of numerous dependencies that will have to be installed at the same time. For brevity I haven't included the output below.

root@devbox:/# apt-get update
root@devbox:/# apt-get install virtualbox

In addition it's a good idea to install the dkms package to ensure that the VirtualBox host kernel modules (vboxdrv, vboxnetflt and vboxnetadp) are properly updated if the linux kernel version changes during the next apt-get upgrade. I did this but no updates were required.

root@devbox:/# apt-get install dkms
Reading package lists... Done
Building dependency tree       
Reading state information... Done
dkms is already the newest version (2.3-3ubuntu1.2).
dkms set to manually installed.
0 to upgrade, 0 to newly install, 0 to remove and 7 not to upgrade.
root@devbox:/#

Now we can check the installed version of VirtualBox to make sure it is installed ok.

root@devbox:/# vboxmanage --version
5.1.22_Ubuntur115126
root@devbox:/#

Export the Existing VM from VirtualBox

Now I exported the existing VM from my MacBook. This is achieved through the VirtualBox GUI. Go to File->Export Appliance... and select the VM from the list (first image above). Then change the name of the output file if you desire. I opted to do this by adding the date on to the end (image two). The third image shows the appliance's settings - if you are ok with this click Export. The final image shows the progress screen - budget on a few minutes.

Create a directory for the VM on the remote host machine.

nigel@devbox:~$ cd ~
nigel@devbox:~$ mkdir VM
nigel@devbox:~$

And scp it over to that machine from the MacBook. This could take half an hour or so.

$ scp badzilla-d8-2018-01-15.ova nigel@192.168.0.202:/home/nigel/VM/.

Import VM Image

Now let's import our VM image so VirtualBox knows about it. Note I am now logged in as myself, not as superuser.

nigel@devbox:~/VM$ VBoxManage import badzilla-d8-2018-01-15.ova 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interpreting /home/nigel/VM/badzilla-d8-2018-01-15.ova...
OK.
Disks:
  vmdisk2	16106127360	-1	http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized	badzilla-d8-2018-01-15-disk001.vmdk	-1	-1	
 
Virtual system 0:
 0: Suggested OS type: "Ubuntu_64"
    (change with "--vsys 0 --ostype "; use "list ostypes" to list all possible values)
 1: Suggested VM name "badzilla-d8"
    (change with "--vsys 0 --vmname ")
 2: Number of CPUs: 1
    (change with "--vsys 0 --cpus ")
 3: Guest memory: 1024 MB
    (change with "--vsys 0 --memory ")
 4: Sound card (appliance expects "", can change on import)
    (disable with "--vsys 0 --unit 4 --ignore")
 5: USB controller
    (disable with "--vsys 0 --unit 5 --ignore")
 6: Network adapter: orig Bridged, config 3, extra slot=0;type=Bridged
 7: CD-ROM
    (disable with "--vsys 0 --unit 7 --ignore")
 8: IDE controller, type PIIX4
    (disable with "--vsys 0 --unit 8 --ignore")
 9: IDE controller, type PIIX4
    (disable with "--vsys 0 --unit 9 --ignore")
10: SATA controller, type AHCI
    (disable with "--vsys 0 --unit 10 --ignore")
11: Hard disk image: source image=badzilla-d8-2018-01-15-disk001.vmdk, target path=/home/nigel/VirtualBox VMs/badzilla-d8/badzilla-d8-2018-01-15-disk001.vmdk, controller=10;channel=0
    (change target path with "--vsys 0 --unit 11 --disk path";
    disable with "--vsys 0 --unit 11 --ignore")
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Successfully imported the appliance.
nigel@devbox:~/VM$

Modify Existing Network Configuration

An attempt to start the VM will result in an error - see below diagnostic.

nigel@devbox:~/VM$ VBoxHeadless --startvm badzilla-d8 
Oracle VM VirtualBox Headless Interface 5.1.22_Ubuntu
(C) 2008-2017 Oracle Corporation
All rights reserved.
 
Error: failed to start machine. Error message: Failed to open/create the internal network 'HostInterfaceNetworking-en0: Wi-Fi (AirPort)' (you might need to modprobe vboxnetflt to make it accessible) (VERR_INTNET_FLT_IF_NOT_FOUND).
Failed to attach the network LUN (VERR_INTNET_FLT_IF_NOT_FOUND)
nigel@devbox:~/VM$

This is due to the network configuration which is currently set to use the MacBook's WiFi Airport device as a bridge. No good in an Intel NUC machine! So let's see how the configuration currently looks:

nigel@devbox:~/VM$ VBoxManage showvminfo badzilla-d8 | grep NIC
NIC 1:           MAC: 080027D2CF47, Attachment: Bridged Interface 'en0: Wi-Fi (AirPort)', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none
NIC 2:           disabled
NIC 3:           disabled
NIC 4:           disabled
NIC 5:           disabled
NIC 6:           disabled
NIC 7:           disabled
NIC 8:           disabled
nigel@devbox:~/VM$

Let's see what we've got in this machine...

nigel@devbox:~/VM$ VBoxManage list bridgedifs
Name:            enp3s0
GUID:            33706e65-3073-4000-8000-94c69114b597
DHCP:            Disabled
IPAddress:       192.168.0.202
NetworkMask:     255.255.255.0
IPV6Address:     fe80:0000:0000:0000:96c6:91ff:fe14:b597
IPV6NetworkMaskPrefixLength: 64
HardwareAddress: 94:c6:91:14:b5:97
MediumType:      Ethernet
Status:          Up
VBoxNetworkName: HostInterfaceNetworking-enp3s0
 
Name:            wlp2s0
GUID:            32706c77-3073-4000-8000-cc2f7140538a
DHCP:            Disabled
IPAddress:       0.0.0.0
NetworkMask:     0.0.0.0
IPV6Address:     
IPV6NetworkMaskPrefixLength: 0
HardwareAddress: cc:2f:71:40:53:8a
MediumType:      Ethernet
Status:          Down
VBoxNetworkName: HostInterfaceNetworking-wlp2s0
 
nigel@devbox:~/VM$

Ok - our eth0 port is actually called enp3s0. That's useful :) So let's set that to be our network adapter.

nigel@devbox:~/VM$ VBoxManage modifyvm badzilla-d8 --nic1 bridged --bridgeadapter1 enp3s0

Now lets see what happens when we try to start the VM

nigel@devbox:~/VM$ Oracle VM VirtualBox Headless Interface 5.1.22_Ubuntu
(C) 2008-2017 Oracle Corporation
All rights reserved.
 
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_ref

Ok - this looks more promising. The ALSA messages probably relate to a missing sound card. Let's check

Remove Soundcard

We are now clearly very close to having a configuration that works. By checking the connected devices on my router I can see that my machine badzilla-d8 has connected to the network successfully. Further more I can ping it and ssh into it! All we need to do is sort out the sound card diagnostics.

$ ping 192.168.0.17
PING 192.168.0.17 (192.168.0.17): 56 data bytes
64 bytes from 192.168.0.17: icmp_seq=0 ttl=64 time=4.286 ms
64 bytes from 192.168.0.17: icmp_seq=1 ttl=64 time=1.941 ms
64 bytes from 192.168.0.17: icmp_seq=2 ttl=64 time=2.101 ms
^C
--- 192.168.0.17 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.941/2.776/4.286/1.070 ms
$ ssh nigel@192.168.0.17
nigel@192.168.0.17's password: 
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64)
 
 * Documentation:  https://help.ubuntu.com/
 
  System information as of Mon Jan 15 16:19:31 GMT 2018
 
  System load:  0.08               Processes:           117
  Usage of /:   52.0% of 14.10GB   Users logged in:     0
  Memory usage: 5%                 IP address for eth0: 192.168.0.17
  Swap usage:   0%
 
  Graph this data and manage this system at:
    https://landscape.canonical.com/
 
Last login: Mon Jan 15 16:16:02 2018 from 192.168.0.12
$

Ok so let's see what the configuration says about our audio card:

nigel@devbox:~/VM$ VBoxManage showvminfo badzilla-d8 | grep ALSA
Audio:           enabled (Driver: ALSA, Controller: AC97, Codec: AD1980)
nigel@devbox:~/VM$

Power off the machine before we do anything to the configuration

nigel@devbox:~/VM$ VBoxManage controlvm badzilla-d8 poweroff
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
nigel@devbox:~/VM$

Now we can remove the audio, check it's definitely gone, and start the VM

nigel@devbox:~/VM$ VBoxManage modifyvm badzilla-d8 --audio none
nigel@devbox:~/VM$ VBoxManage showvminfo badzilla-d8 | grep audio
nigel@devbox:~/VM$ VBoxHeadless --startvm badzilla-d8 &
[1] 29795
nigel@devbox:~/VM$ Oracle VM VirtualBox Headless Interface 5.1.22_Ubuntu
(C) 2008-2017 Oracle Corporation
All rights reserved.
 
 
nigel@devbox:~/VM$

Ok we are up and running! Now so long as we can connect via NFS on my MAC and my Kubuntu laptop, then we should be in good shape.

NFS Connect to VM from Mac OS

Open Finder and press cmd-K to bring up the Connect to Server modal. The NFS url format must be like shown in the first image - and obviously you must have a corresponding /etc/hosts entry for the url. In my case this will 192.168.0.17.

Once connected, you will see whatever files you've got on your server. In my case I map to /var/www/html so the list of folders are effectively my projects for each website I am working on.

NFS Connect to VM from Kubuntu Linux

The NFS connection from Kubuntu is best achieved on the command line or by adding an entry to the /etc/fstab file. Once that's achieved, the directory structure can be listed. Success!

nigel@devbox:~$ sudo mount -t nfs -o vers=3 192.168.0.17:/var/www/html /mnt/projects                       
nigel@devbox:~$ ls -las /mnt/projects/
total 84                                                                                               
 4 drwxrwxr-x 13 www-data www-data  4096 Jan 15 18:13 .                                                     
 4 drwxr-xr-x  3 root     root      4096 Jan 15 19:15 ..
 4 drwxrwxr-x  3 nigel    www-data  4096 Aug 11 22:43 badzilla_d6
 4 drwxrwxr-x  7 www-data www-data  4096 Feb 19  2017 badzilla_d8
 4 -rw-rw-r--  1 www-data www-data  4096 Feb  9  2017 ._blog_theme_d8
 4 drwxrwxr-x 13 www-data www-data  4096 Feb 12  2017 blog_theme_d8
 4 -rw-rw-r--  1 www-data www-data  4096 Jan 15 18:13 ._index.html
12 -rw-rw-r--  1 www-data www-data 11510 Feb  6  2017 index.html
 4 drwxrwxr-x  7 nigel    www-data  4096 Oct 19 21:07 kitboot
12 drwxr-xr-x  4 nigel    users    12288 Dec 27 16:41 lakeland
 4 drwxrwxr-x  8 nigel    www-data  4096 Oct 22 10:49 meedjum
 4 drwxrwxr-x  4 nigel    www-data  4096 Apr 23  2017 multisite
 4 drwxrwxr-x 11 www-data www-data  4096 Feb 12  2017 personal-blog
 4 -rw-rw-r--  1 www-data www-data  1301 Aug  6  2015 README.txt
 4 drwxrwxr-x  9 nigel    www-data  4096 May  7  2017 saasidate
 4 drwxrwxr-x  9 nigel    www-data  4096 Dec 23 16:19 serverless-vote
nigel@devbox:~$

blog terms

Linux Development VirtualBox

Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox

nigel — Sat, 09 Sep 2017 21:43:47 +0000

Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox nigel Sat, 09/09/2017 - 22:43

This blog shows you how to build a Squid proxy server on an inexpensive Raspberry Pi machine to help to test web application development in a local VM sandbox.

So here's the problem. Say you are using a VirtualBox hosting Linux in a MacBook to develop your web application. This is practically an industry standard these days; certainly in the open source world anyway. You will have defined the IP address allocated to your VM in your /etc/hosts along with the domain name you have chosen.

You have elected a 'browser first' strategy for the markup, but how do you test your website against hand held devices? Option 1 is using the simulator in Chrome and other browsers which do a very good job of emulating most common devices, but they aren't perfect and frequently the results will differ between the simulator and the physical device. But since it isn't possible to edit the /etc/hosts file in hand held devices, how do you point them to your arbitrarily named website domain? Well the second option is to use an external servicessuch as localtunnel that can expose an IP address to the Internet - but a local service must be running constantly on the VM and firewalls have to be configured to allow incoming http requests which is not always desirable.

There is a third way - use a proxy server to redirect the urls tapped into the hand held devices to the VM in your host machine (Mac / Windows or Linux). The easiest way to achieve this is to use Squid - a mature and free proxy server used throughout the IT industry. It has immense capabilities - such as blacklisting genres of websites (e.g. porn / sex sites) so young family members aren't exposed to what they shouldn't be. But we really only need simple redirection.

So my choice is Squid which runs on Linux. Should I buy even more new hardware for my apartment? Or repurpose equipment from my hardware junkyard in the spare bedroom? Nope. I've always wanted a Raspberry Pi to play with, and now I have an excuse to buy one!

Bill of Materials

Device	Raspberry Pi 3 Model B Quad Core CPU 1.2 GHz 1 GB RAM Motherboard	Amazon.co.uk	£32.50
SD Card	SanDisk Ultra 32 GB microSDHC Class 10 Memory Card up to 48 Mb/s - White/Grey	Amazon.co.uk	£13.50
Box	Raspberry Pi 3 Case - Black/Grey	Amazon.co.uk	£5.50
Keyboard	Mofii Full-size Colorful Wired USB LED Backlit Gaming Keyboard for PC & MAC - UK Layout	Amazon.co.uk	£12.99
Power Cable	2M / 6.5ft USB 2.0 A To MICRO B Cable Data and Charging Lead Cable - Black	eBay.co.uk	£1.65
Wall Plug	DC5V 2A UK 3pin USB Charger Wall Plug Power Adapter	eBay.co.uk	£3.45
Mouse	USB Mouse - spare	n/a	£-
Network Cable	RJ45 CAT5 cable - spare	n/a	£-
	TOTAL		£68.80

So the total price was significantly less than buying a new desktop or laptop computer and obviously the USB keyboard can be reused on other projects. I could've reduced my exposure by sharing phone charging cables around my apartment rather than buy another one to be dedicated for the Raspberry Pi but I wanted a 5V 2A power supply so it made sense to spend a small amount of extra money.

Building the System

There are plenty of tutorials on the Internet for setting up the base Raspberry Pi operation system - so I won't duplicate it here, but please note you will need to install the Raspberry Pi version of Debian Linux (Raspbian) on the SD card.

Here's a list of useful links:

 Raspberry Pi Software Guide
 How to set up and use a Raspberry Pi 3 with a Mac
 SETTING WIFI UP VIA THE COMMAND LINE - Only needed if you are not going to use a static IP address through the LAN connection
 Use Ssh to Talk With Your Raspberry Pi.   

Once your Raspberry Pi is set up and you are ready to go, you can either ssh into the machine (as I did) or connect to it using a keyboard and mouse. If you choose SSH, you will have to enable that service first - see the link above if you are unsure how to do this.

Update the distribution

Always a good idea to update the box in case there are pending patches which weren't in the initial downloaded distribution. When I ran the commands below I got a welter of updates (which I've removed from the blow listing due to brevity reasons)

pi@raspberrypi:~ $ sudo apt-get update
pi@raspberrypi:~ $ sudo apt-get upgrade
pi@raspberrypi:~ $ sudo apt-get dist-upgrade

Set a Static IP Address

The Raspberry Pi 3 does come with WifI so theoretically it is possible to set it up to have a DHCP dynamically assigned IP address. However that would be saving up problems should the IP address change due to rebooting or power interruptions. It would mean a whole load of settings would need to be changed in our system each time that happens. Most routers come with a built-in DHCP server and the range of IP addresses can be set - see below. I have already set my range to be 192.168.0.2 to 192.168.0.199 so anything 200 and above is reserved for devices that need a static address.

I now start up the Raspberry Pi, connected to my LAN with a RJ45 cable, and it gets allocated a dynamic IP address by the DHCP server on my router. This can be checked through the router UI (see below) or by issuing ifconfig when logged on to the Pi.

Become the super user and edit the /etc/dhcpcd.conf file with vi or nano.

pi@raspberrypi:~ $ sudo su
root@raspberrypi:/home/pi# cd /etc/
root@raspberrypi:/etc# vi dhcpcd.conf

Now add the following text to the end of it. Note that I am just using the tail command to list the edits I have already made.

root@raspberrypi:/etc# tail -5 dhcpcd.conf
 
interface eth0
static ip_address=192.168.0.201/24        
static routers=192.168.0.1
static domain_name_servers=90.207.238.97
root@raspberrypi:/etc#

You will need to change the values according to your circumstances. I am using 201 for my Pi server, 1 is my router and the domain name server setting is pointing to my ISP's DNS address. Now reboot the Pi.

root@raspberrypi:/etc# init 6
Connection to 192.168.0.12 closed by remote host.
Connection to 192.168.0.12 closed.

Once the device has completed its startup sequence, go back to your router and check you've got an entry for your Pi - see screenshot below. My two servers can be seen highlighted - FreeNAS using 200 and the Pi at 201. We are good to go!

Install Squid

Ok let's install Squid

pi@raspberrypi:~ $ sudo apt-get install squid3
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libecap2 libmnl0 libnetfilter-conntrack3 squid-langpack squid3-common
Suggested packages:
  squidclient squid-cgi squid-purge smbclient ufw winbindd
The following NEW packages will be installed:
  libecap2 libmnl0 libnetfilter-conntrack3 squid-langpack squid3 squid3-common
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,370 kB of archives.
After this operation, 8,372 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://mirrordirector.raspbian.org/raspbian/ jessie/main libmnl0 armhf 1.0.3-5 [10.9 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ jessie/main libnetfilter-conntrack3 armhf 1.0.4-1 [40.0 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid-langpack all 20140506-1 [150 kB]
Get:4 http://mirrordirector.raspbian.org/raspbian/ jessie/main libecap2 armhf 0.2.0-3 [13.8 kB]
Get:5 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid3-common all 3.4.8-6+deb8u4 [259 kB]
Get:6 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid3 armhf 3.4.8-6+deb8u4 [1,897 kB]
Fetched 2,370 kB in 3s (634 kB/s)   
Selecting previously unselected package libmnl0:armhf.
(Reading database ... 115447 files and directories currently installed.)
Preparing to unpack .../libmnl0_1.0.3-5_armhf.deb ...
Unpacking libmnl0:armhf (1.0.3-5) ...
Selecting previously unselected package libnetfilter-conntrack3:armhf.
Preparing to unpack .../libnetfilter-conntrack3_1.0.4-1_armhf.deb ...
Unpacking libnetfilter-conntrack3:armhf (1.0.4-1) ...
Selecting previously unselected package squid-langpack.
Preparing to unpack .../squid-langpack_20140506-1_all.deb ...
Unpacking squid-langpack (20140506-1) ...
Selecting previously unselected package libecap2.
Preparing to unpack .../libecap2_0.2.0-3_armhf.deb ...
Unpacking libecap2 (0.2.0-3) ...
Selecting previously unselected package squid3-common.
Preparing to unpack .../squid3-common_3.4.8-6+deb8u4_all.deb ...
Unpacking squid3-common (3.4.8-6+deb8u4) ...
Selecting previously unselected package squid3.
Preparing to unpack .../squid3_3.4.8-6+deb8u4_armhf.deb ...
Unpacking squid3 (3.4.8-6+deb8u4) ...
Processing triggers for man-db (2.7.5-1~bpo8+1) ...
Processing triggers for systemd (215-17+deb8u7) ...
Setting up libmnl0:armhf (1.0.3-5) ...
Setting up libnetfilter-conntrack3:armhf (1.0.4-1) ...
Setting up squid-langpack (20140506-1) ...
Setting up libecap2 (0.2.0-3) ...
Setting up squid3-common (3.4.8-6+deb8u4) ...
Setting up squid3 (3.4.8-6+deb8u4) ...
Creating Squid HTTP proxy 3.x spool directory structure
2017/07/30 18:55:00 kid1| Set Current Directory to /var/spool/squid3
2017/07/30 18:55:00 kid1| Creating missing swap directories
2017/07/30 18:55:00 kid1| No cache_dir stores are configured.
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Processing triggers for systemd (215-17+deb8u7) ...
pi@raspberrypi:~ $

Configuring Squid

First change directory to the squid configuration Squid directory and save the original configuration file just in case it needs to revered sometime.

root@raspberrypi:~# cd /etc/squid3/
root@raspberrypi:/etc/squid3# cp squid.conf squid.conf.original
root@raspberrypi:/etc/squid3#

By default Squid doesn't allow any access to the Internet so that will need fixing. Like most ACL based systems, once Squid finds a rule that applies to a particular URL then it stops looking for more rules. So early in the config file you need to tell Squid to allow all HTTP traffic.

http_access allow all

You will also need to tell Squid about your local network. I am using the 192.168.0.0 range of IP addresses - find that in the configuration and uncomment it

acl localnet src 192.168.0.0/16	# RFC1918 possible internal network

Finally we don't want any caching since we will be using our Squid mainly for redirection purposes and when recent website changes need to be seen for testing purposes, so add this at the caching section

cache deny all

Below is a list of the changes in a form of a diff file which some may find more useful than the separate settings I've mentioned above.

root@raspberrypi:/etc/squid3# diff -Naur  squid.conf.original squid.conf
--- squid.conf.original	2017-09-02 19:22:23.804970154 +0000
+++ squid.conf	2017-09-03 10:41:48.190318779 +0000
@@ -1037,10 +1037,11 @@
 # should be allowed
 #acl localnet src 10.0.0.0/8	# RFC1918 possible internal network
 #acl localnet src 172.16.0.0/12	# RFC1918 possible internal network
-#acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
+acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
 #acl localnet src fc00::/7       # RFC 4193 local private network range
 #acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines
 
+
 acl SSL_ports port 443
 acl Safe_ports port 80		# http
 acl Safe_ports port 21		# ftp
@@ -1180,6 +1181,7 @@
 #Default:
 # Deny, unless rules exist in squid.conf.
 #
+http_access allow all
 
 #
 # Recommended minimum Access Permission configuration:
@@ -2999,6 +3001,9 @@
 # DISK CACHE OPTIONS
 # -----------------------------------------------------------------------------
 
+cache deny all
+
+
 #  TAG: cache_replacement_policy
 #	The cache replacement policy parameter determines which
 #	objects are evicted (replaced) when disk space is needed.
root@raspberrypi:/etc/squid3#

We are now done with the configuration in the conf file so we can restart the Squid service so these changes take effect

root@raspberrypi:/etc/squid3# service squid3 restart
root@raspberrypi:/etc/squid3#

Won't be the worst idea to copy the configuration just for safety.

root@raspberrypi:/etc/squid3# cp squid.conf squid.conf.local-vm
root@raspberrypi:/etc/squid3#

You are now probably wondering how does Squid possibly know where my Virtual Hosts in my Virtual Machine are!! This is the simple bit :) - you need to add all those sites you are working on into the /etc/hosts file on your Pi. You will already have these listed in your Macbook hosts file so simply copy and paste. My file ended up looking like this below:

root@raspberrypi:/etc/squid3# cat /etc/hosts
127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters
 
127.0.1.1	raspberrypi
 
192.168.0.14 badzilla-d8.localhost
192.168.0.14 meedjum.localhost
root@raspberrypi:/etc/squid3#

The last two entries - badzilla-d8.localhost and meedjum.localhost are the two sites I am working on. The migration of Badzilla from D6 to D8. Yey!

Handheld Device Client Configuration

We are now ready to test our solution, and for this I'll use screenshots from Android and iOS.

Android

Go to Settings->WiFi, then long tap on your network and tap 'Modify network config', then tap 'Show advanced options' and add the configuration for the proxy, then tap save.

Now point a web browser to your web development domain

iOS

Go to settings, tap on your WiFi, tap on the information icon, tap on 'Manual' under HTTP Proxy, and enter the values.

Now point a web browser to your web development domain - this is very early in the process as you can see. Somehow two seach boxes are on this early build!

blog terms

Raspberry Pi Squid Linux Drupal Development VirtualBox

Evaluate Windows 8 in VirtualBox running on openSUSE Linux

nigel — Sun, 24 Feb 2013 11:24:24 +0000

Evaluate Windows 8 in VirtualBox running on openSUSE Linux nigel Sun, 24/02/2013 - 11:24

Being a die-hard Linux user I have tried to eliminate Microsoft products from my life. Unfortunately I've not been totally successful, and I still have recourse to using a few software packages that are only available on the Windows platform. Therefore I have three options - (1) Use a second machine to run the Windows apps, (2) Reboot my Linux machine into a Windows partition, (3) Run a VirtualBox in Linux, and in that VirtualBox have a copy of Windows running. Clearly option (3) is the best solution, and this tutorial provides the steps required to install Windows 8 in a VirtualBox under openSUSE, my own particular favourite flavour of Linux.

The beauty of VirtualBox is it quickly allows an evaluation of an operating system to check whether it matches requirements, and since Windows 8 costs real money it is not a decision I'll take lightly. So my requirements are Windows 8 in VirtualBox must be able to:

Connect to my local FreeNAS server which is my central repository for all my personal data. My own cloud computing in my spare room
Accept input from my USB barcode scanner for when I scan books into my BookCAT collection
Apropos the above, be able to run the Windows apps BookCAT (for my book collection) and Photoshop

I have promised myself that should all those requirements be met, I'll invest in my own legal copy of MS Windows 8. So, on with the tutorial!

Firstly, you will need a copy of the GPL licensed VirtualBox on your machine. I am undertaking this tutorial using openSUSE, and one of the reasons for this is I like the very user friendly Yast utility for system administration. It is a breeze to install software and all its dependencies. The VirtualBox images are held in the standard openSUSE repositories. Couldn't be simpler!

An evaluation copy of MS Windows 8 is available for 90 days before it expires. There are all sorts of restrictions to this, primarily the evaluation copy cannot be upgraded later and thus any post-installation configuration will be lost. No matter for me. Also worth noting is you will need a Microsoft account somewhere before you are allowed to download Windows. It can be downloaded from http://msdn.microsoft.com/en-gb/evalcenter/jj554510.aspx. Once you have downloaded it, the file (which is an iso image) will need to be burnt to DVD. I used the KDE application k2b for this, although there are many utilities to do this.

Next start up VirtualBox since we are ready to start the build, and you will see the screen above.

Click on New and add a name to your virtual machine. I tried Windows 8 and rather cleverly the selection boxes underneath selected the correct options for me

I adjusted the memory slider to share the available memory between Linux and Windows. I'm tempted to buy additional memory for my laptop now - maybe an upgrade to 8GB could be in order?

Here I am saying I would like to configure later an amount of disk space reserved for this installation.

The type of disk required may vary depending upon your circumstances, but I went for a Virtual Hard Disk

I elected to set a precise amount of disk space.

Here you will need to specify the location of the disk drive and how much you want to allocate. Unless you are planning on installing loads of apps for your evaluation, there isn't much point in overstating the size. I went for 25GB which will be more than adequate.

Here's the summary screen so be sure to check everything matches your needs before clicking create

The VirtualBox instance will now be created and you'll see it listed down the lefthand side

Click on Start and the Windows installation is under way! The first time wizard dialog will be presented

Select the drive which has your DVD iso image. In my case, there's a choice of one!

A pre-flight summary screen before you light the blue touch paper

The Windows 8 installation starts and you will be presented with the country select options. As usual, there is no English (United Kingdom) language option. Shameful, Microsoft!

You can of course read all the Terms and Conditions before you accept

Next you will see a compatibility report. Treat this as an aide memoire - you will need to remove the installation disk once the install process is completed or else it will attempt a second install over your previous one

Remember that 25GB we allocated for our hard disk? This is where we select it.

The installation process commences and all the files are copied over to your hard disk. This will take some time so go and put the kettle on

Give your Windows 8 installation a name. It took me a couple of attempts to invent one that was valid and fitted in the character limits and restrictions

I couldn't see a compelling reason for doing anything other than Use Express Settings here. Your mileage may vary, if so you may wish to experiment with . If you do and stuff up, you can always delete your virtual machine and go again

I guess this is the way operating systems are going, but I can't say I'm particularly comfortable about it. I needed to sign in using my Microsoft account. I took a couple of goes to get it right as you can see

A rather pleasing colour changing screen now appears as the apps are installed on the machine.

And here we are - Windows 8 starts up. The odd thing is the system knows I am UK based and sets the contextual news correctly, but rather goes adrift with the weather!

Click on the Desktop and you will duly arrive on the desktop!

For those following this tutorial with the intention of getting Windows up and running, you have succeeded! I however have other criteria - I need to connect to my FreeNAS server on my local network. To do this, click on the folder icon and once the files / folders come up and right click on the Networkand then click on Map network drive

Empirically I noticed that no matter which configuration I went for, Windows 8 simply would not discover my FreeNAS server in the workgroup WORKGROUP and show a clickable icon. So I had to put the IP address and the share name in manually. If you don't have permissions to log onto this share, you will be presented (as I was) with a further dialog box requesting a user name and password.

Yippee! I have now connected to my FreeNAS server and I can see my directory listing. So, one of my criteria has been met!

I now installed BookCAT - the Windows book cataloguing software and the primary driver for this entire exercise. The next test was to plug my USB barcode scanner and scan the barcode of a book into BookCAT. All being well it will recognise the code as a valid ISBN and download the book details from Amazon. This went very well with only a minor problem - the first scan from the device issued an addition '9' at the front of the ISBN. Subsequent scans were ok. I have no idea why this is the case, but it is such a trivial inconvenience I am more than happy to forget about it. No point in wasting time investigating.

And here is the final image - Windows 8 happily sitting in its own window in a VirtualBox running under openSUSE Linux. It's Alchemy!! But it's also an excellent tool to enable for instance cross browser compatibility for web developers such as myself.

blog terms

Linux VirtualBox