Raspberry Pi

RasPlex Installation using a Raspberry Pi 3B+ with Rii Mini i8+ Remote

nigel — Sat, 23 May 2020 09:49:25 +0000

RasPlex Installation using a Raspberry Pi 3B+ with Rii Mini i8+ Remote nigel Sat, 23/05/2020 - 10:49

Tutorial on how to install RasPlex on a Raspberry Pi 3B+ with a Rii Mini i8+ remote control.

Download the RasPlex Image

The Raspberry Pi 3B+ is a little problematic for RasPlex which is borderline abandonware. There is no official GitHub image release for a 3B+, and attempting to install the latest 1.8.0 will result in the Raspberry Pi rainbow screen upon power up.

There is a long thread here that explains the problems with the 3B+ - however be warned the conversation gets seriously derailed with a nonsense focus on power supplies half way through. There is huge intimation that the fact that RasPlex doesn't boot on a 3B+ is because of the power supply issues. Disregard this - it doesn't boot on an official Raspberry Pi power supply either so power is not the issue!

Thankfully, albeit partially obscured by background noise, there is a link to a commit that does work. This is RasPlex-1.8.0.0-573b6d73-RPi2.arm.img.gz and can be downloaded from here.

Flashing the Image to MicroSD Card

I used balenaEtcher to flash the image to my MicroSD card although other tools are available. A little care is required here - I could've easily incorrectly selected the wrong media since by default balenaEtcher selected my SanDisk Ultra Fit USB 3.1 Flash Drive. I reselected the MicroSD which was in the card reader.

Boot the Raspberry Pi

Inserting the MicroSD card into the Raspberry Pi slot and powering up will now bring up the configuration screens for RasPlex. I didn't have to change any of the defaults - and it found my Plex server automatically as well! I elected not to enable pre-caching of the assets since my network is quick enough for the images to load practically instantaneously.

Once the RasPlex 'configuration' is complete, the system loads openPHT which is the user interface for the media centre - i.e. the Plex client.

Configuring openPHT

As I mentioned earlier, there is a Raspberry Pi 3B+ working image, which is what Ive used to build my system. It is imperative to ensure the system is pinned at that version and not automatically updated upon a reboot to a newer patch level. To do that change the settings by navigating to:

Left button -> Preferences -> Left button -> Down button (twice) -> Right button -> Down button (four times) to Software Update then untick the first option.

I also changed the keyboard to be uk, but apart from that there was nothing more to do.

Rii Mini i8+ Remote

Simply put, this 'just works'. It was already paired to its USB receiver before I used it in a previous Android TV project I've now abandoned for this project. I turned on the remote, inserted the receiver in a spare USB slot, and instantly it was sending keystrokes to the Raspberry Pi. Couldn't have been simpler.

Bill of Materials

Raspberry Pi	3BPLUS-R 1.4 GHz 1 GB RAM 64-Bit	£37.67
SD Card	LXSINO 32GB Micro SD Class 10	£6.99
Power	Aukru Charger 5V 3000mA Power Supply 3A	£9.99
Remote	Rii i8+ 2.4Ghz LED Backlit Mini Wireless Keyboard	n/a
	TOTAL	£54.65

The remote isn't available anymore, but models practically the same are for around £8.

blog terms

Hardware Raspberry Pi Plex

Raspberry Pi Music Box

nigel — Wed, 06 Feb 2019 10:45:32 +0000

Raspberry Pi Music Box nigel Wed, 06/02/2019 - 10:45

Introduction

This bog provides a tutorial for building a Raspberry Pi music box which can be controlled using a hand held device through a web interface. Ideal for a bedroom where there is no desire to have a monitor / keyboard /mouse combination. The cornerstone of the build is the Rune Audio distro which supports FLAC, Vorbis and mp3 amongst other formats. I will be using a Raspberry Pi 3 Model B+ along with an IQAudIO Pi-DAC+ daughter board.

There is fierce debate between the proponents of Rune Audio, MoOde Audio, and Volumio. I elected for Rune purely because there is a native Android app available, and my smartphone is a Samsung.

Bill of Materials

Raspberry Pi	3BPLUS-R 1.4 GHz 1 GB RAM 64-Bit	£34.56
DAC	IQAudIO PI-DAC+	£35.00
SD Card	LXSINO 32GB Micro SD Class 10	£7.99
Power	Official 5V 2.5A Power Adapter for the Raspberry Pi 3	£7.99
Case	IQAudIO PI-CASE+	£19.40
Speakers	Creative Gigaworks T40 Series II	£79.99
	TOTAL	£184.93

Of course I can't guarantee you'll get the same deals I got, but the pricing should at least be indicative. I already had the speakers I bought for a different project a few years ago that never saw the light of day, so that reduced my outlay to around £105 for this project.

Fit and test the DAC.

Fitting the DAC to the Raspberry Pi is quite easy although there are instructions on the IQaudIO site. The DAC comes with some spacers and screws which align with the Pi's own. Since this is a quick sound test and I will be building the components into a proper case at some point, I only tightened the screws finger tight.

Once this has been completed, it's time to test it works fine. IQaudIO provide downloadable bootable images that can be written to an SD disk and will automatically play mp3 files. The best file on their downloads page is IQ_0_quicktestDigi*.zip.

Unfortunately the zip file was corrupted when I tried this. I reported this to their webmaster.

Build the Pi-CASE+

I couldn't find any build instructions for the case, but it is fairly straightforward. The first point to note is there are some standoffs which should be screwed into the four Raspberry Pi screw holes on the corners of the PCB. The case comprises six perspex sides, and they are slotted to ensure there is only one way it can all fit together - so the second point is it is crucial to dry run this first before it is screwed together to ensure there is no attempt to situate any of the panels the wrong way around.

The panels fit together very snuggly without any slack and comes with some small plastic feet to prevent it rubbing and sliding on the chosen surface.

Flash the Rune Audio distro

Time to download our chosen repo. This is a small inconvenience because Rune Audio do not have an official Raspberry Pi 3B+ image. However there is a workaround - a community member has created a rar file download that can be fetched from the Rune Audio forums.

The file is in rar format so on a Mac you will need to install unrar

$ brew install unrar
==> Pouring unrar-5.7.1.mojave.bottle.tar.gz
/usr/local/Cellar/unrar/5.7.1: 7 files, 520.7KB

Then we can run the unrar command

$ unrar e RuneAudio_0.4_20180629_SC_Pi3B_Plus_v3_tml3nr.rar 
 
UNRAR 5.70 beta 1 freeware      Copyright (c) 1993-2019 Alexander Roshal
 
 
Extracting from RuneAudio_0.4_20180629_SC_Pi3B_Plus_v3_tml3nr.rar
 
Extracting  RuneAudio_0.4_20180629_SC_Pi3B+_v3_tml3nr.img             OK 
All OK

To write the image to the SD card I use Etcher. This is a simple utility app although the command line can be used too with dd - not recommended for those with low experience. Once Etcher has completed it will provide a useful desktop notification. At that point the SD card is ready to be inserted into the Raspberry Pi.

Booting Rune Audio

The Pi can be booted once a network cable has been attached. If all goes well, it should be possible to point a web browser at http://runeaudio.local (see screenshot above). We now have the genesis of a music box but it still needs configuring.

MPD Configuration

There are a number of config screens to work through to get the system functioning correctly. Some of the changes will be specific to my circumstances so your mileage may vary. The MPD (Music Player Daemon) configuration needed one change - the need to select the IQaudIO audio output interface.

Settings configuration

This one in particular may not suit all my readers, but I want to use the Europe/London timezone.

Network settings

It makes sense to use a static IP address on your home network - so we can use a proxy in order for all devices to find their way to Rune. More of this later, but for now navigate to Network and you should see a sight similar to the first screenshot above. Click on eth0 and you will see the second screenshot. I have populated this form to use 192.168.0.204. My router has already been configured to assign DHCP addresses between 192.168.0.2->199. Anything 200 and above is reserved for my static IP address home servers.

Sources configuration

The final configuration activity is to make Rune Audio aware of where the audio files are stored, and in my case they are on an NFS drive on my NAS server. Go to the Sources page as per the first screenshot above and click on Add New Mount. The form to complete for NFS mounts just wouldn't play ball at all, so in desperation I tried SMB/CIFS(OS X Share) and that worked - see the second screenshot. I called my source flac since this share contains all my high fidelity rips. I then repeated the process by creating a mount for mp3 for my low definition audio. The third screenshot shows the two mounts listed, and I have instigated a rebuild of the library, and this is shown by the updating spinner bottom left.

Playback

As soon as the library has been underway for a short period of time and has added a few artists / albums, playback can be tested. I picked an album at random for my end to end test, and it played seamlessly.

Native Android App - Runeaudio Remote Control

The native app scans the local network and automatically shows all the connected RuneAudio players, allowing you to control them from the same place. It also displays the RuneUI in a true full screen, with no browsers bars and weird scroll behaviours.The app uses the avahi protocol to scan the local network. It detects the connected RuneAudio players which expose the avahi service and lists them, showing also their IP and MAC addresses.

The resulting UI is actually the same as a web browser but I found it immensely frustrating. My habit is to tap the built-in Samsung back key to navigate and that immediately drops me out of the app. Restarting the app requires the initial network discovery to be run again, and mine steadfastly refused to connect initially, but then suddenly appeared to figure out my network and was rock steady afterwards.

Resolving runeaudio.local - Desktop

Rune Audio uses the Avahi protocol which theoretically means when added to a local network, pointing a browser to http://runeaudio.local should 'just work'. Whilst setting up Rune Audio I noticed this capability was very hit and miss. There are ways of getting a rock solid configuration - for a desktop the simple solution would be to add runeadio.local to the /etc/hosts file.

/etc/hosts

192.168.0.204 runeaudio.local

Obviously the IP address should be swapped with whichever you have used.

Resolving runeaudio.local - Handheld Device

Since /etc/hosts can't be edited on a handheld device such as a tablet or a smartphone, we need another solution. The workaround for this is to use a proxy server as per my blog where I utilised another Raspberry Pi as a proxy. Firstly log into the proxy server and edit its /etc/hosts like the desktop so you will have an entry

192.168.0.204 runeaudio.local

The Squid proxy will then need a reboot.

$ sudo service squid3 restart

You will then have to ensure your handheld device is using the proxy for internet access. Screenshots for achieving this on an iPad and an Android phone are in my other blog just mentioned and are not duplicated here for brevity. Just click though if you need guidance.

blog terms

Hardware Raspberry Pi

Ad Blocker using Squid Proxy on Raspberry Pi

nigel — Sun, 17 Sep 2017 15:33:40 +0000

Ad Blocker using Squid Proxy on Raspberry Pi nigel Sun, 17/09/2017 - 16:33

As mentioned in my previous blog Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox it is possible (in fact quite usual) to use Squid to prevent user access to certain websites. This can include ad servers which serve up annoying ad content when trying to read articles. This tutorial shows how we can extend the functionality of the Squid server we have already set up to block ads.

The banning of ad content is achieved by downloading a blacklist of prohibited urls from a service provider, although it has to be mentioned that these lists depend upon the community to contribute any urls they discover and thus there can never be 100% confidence that such a list is ever complete.

Once the list is downloaded from the service provider, Squid needs to be informed of its presence in its configuration and the ban will take the form of an acl. Ok now we know what we need to achieve let's start the process.

Lets log into the Raspberry Pi, switch to superuser, and download the blacklist.

Nigels-MacBook-Pro:Projects nigel$ ssh pi@192.168.0.201
pi@192.168.0.201s password: 
 
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
 
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Sep 10 14:33:42 2017 from 192.168.0.7
pi@raspberrypi:~ $ sudo su
root@raspberrypi:/home/pi# cd /etc/squid3/
root@raspberrypi:/etc/squid3# ls -las
total 832
  4 drwxr-xr-x   2 root root   4096 Sep  9 08:44 .
  4 drwxr-xr-x 112 root root   4096 Sep  9 11:17 ..
  4 -rw-r--r--   1 root root   1547 Dec 24  2016 errorpage.css
  4 -rw-r--r--   1 root root    421 Dec 24  2016 msntauth.conf
272 -rw-r--r--   1 root root 277585 Sep  9 08:44 squid.conf
272 -rw-r--r--   1 root root 277565 Sep  3 11:06 squid.conf.local-vm
272 -rw-r--r--   1 root root 277526 Sep  2 19:22 squid.conf.original
root@raspberrypi:/etc/squid3# curl -sS -L --compressed "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=0&mimetype=plaintext" > ad_block.txt 
root@raspberrypi:/etc/squid3# ls -las
total 872
  4 drwxr-xr-x   2 root root   4096 Sep 17 15:20 .
  4 drwxr-xr-x 112 root root   4096 Sep  9 11:17 ..
 40 -rw-r--r--   1 root root  40042 Sep 17 15:20 ad_block.txt
  4 -rw-r--r--   1 root root   1547 Dec 24  2016 errorpage.css
  4 -rw-r--r--   1 root root    421 Dec 24  2016 msntauth.conf
272 -rw-r--r--   1 root root 277585 Sep  9 08:44 squid.conf
272 -rw-r--r--   1 root root 277565 Sep  3 11:06 squid.conf.local-vm
272 -rw-r--r--   1 root root 277526 Sep  2 19:22 squid.conf.original
root@raspberrypi:/etc/squid3#

The blacklist is now in place - you can see it in the directory listing as file ad_block.txt. The next job is to add this to the Squid configuration as an acl. This needs to be placed above the http_access allow all line we put in the config in the previous blog.

## disable ads ( http://pgl.yoyo.org/adservers/ )
acl ads dstdom_regex "/etc/squid3/ad_block.txt"
http_access deny ads

You can use your favourite editor for this - I used vi - and then reboot the server.

root@raspberrypi:/etc/squid3# vi ad_block.txt 
root@raspberrypi:/etc/squid3# service squid3 restart

We are now ready to test our solution by landing on an ad heavy website such as NME. There are two excellent methods of doing this - one is checking the access.log in real time for TCP_DENIED records, and the other is to eyeball the site and see if the ads really are missing. Let's go for option one first.

root@raspberrypi:/etc/squid3# tail -f /var/log/squid3/access.log
1505665437.619     38 192.168.0.4 TCP_MISS/200 630 GET http://pagead2.googlesyndication.com/activeview? - HIER_DIRECT/216.58.208.130 image/gif
1505665440.152   3632 192.168.0.4 TCP_MISS/200 4009 CONNECT bs.serving-sys.com:443 - HIER_DIRECT/82.199.68.72 -
1505665449.828   8361 192.168.0.4 TCP_MISS/200 630 CONNECT ml314.com:443 - HIER_DIRECT/34.252.181.159 -
1505665452.178     78 192.168.0.4 TCP_MISS/204 740 GET http://beacon.krxd.net/event.gif? - HIER_DIRECT/54.228.222.160 image/gif
1505665491.360  69091 192.168.0.4 TCP_MISS/200 746 CONNECT www.facebook.com:443 - HIER_DIRECT/31.13.90.36 -
1505665493.069  65946 192.168.0.4 TCP_MISS/200 5038 CONNECT match.adsrvr.org:443 - HIER_DIRECT/54.247.91.116 -
1505665502.280  65538 192.168.0.4 TCP_MISS/200 5552 CONNECT odr.mookie1.com:443 - HIER_DIRECT/35.157.149.45 -
1505665519.263  91582 192.168.0.4 TCP_MISS/200 3318 CONNECT bam.nr-data.net:443 - HIER_DIRECT/162.247.242.21 -
1505665544.650 120399 192.168.0.4 TCP_MISS/200 1227 CONNECT session.timecommerce.net:443 - HIER_DIRECT/54.192.2.48 -
1505665554.600      7 192.168.0.4 TCP_MISS/503 4491 GET http://pagead2.googlesyndication.com/activeview? - HIER_NONE/- text/html
1505665577.132      8 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665577.146      9 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665577.357    200 192.168.0.4 TCP_MISS/200 27785 GET http://www.nme.com/ - HIER_DIRECT/54.230.11.205 text/html
1505665577.396    249 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665577.495      7 192.168.0.4 TCP_DENIED/403 4577 GET http://pagead2.googlesyndication.com/activeview? - HIER_NONE/- text/html
1505665577.567      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665577.615      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665577.748    337 192.168.0.4 TCP_MISS/200 364 GET http://www.summerhamster.com/bcn? - HIER_DIRECT/52.27.8.169 image/gif
1505665579.364     58 192.168.0.4 TCP_MISS/304 446 GET http://dnn506yrbagrg.cloudfront.net/pages/scripts/0025/8842.js - HIER_DIRECT/54.230.11.179 -
1505665579.380      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665579.669     69 192.168.0.4 TCP_MISS/200 576 GET http://mpp.vindicosuite.com/bg/? - HIER_DIRECT/130.211.103.172 application/javascript
1505665579.676      4 192.168.0.4 TCP_DENIED/403 4350 POST http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665579.761      3 192.168.0.4 TCP_DENIED/403 4176 GET http://b.scorecardresearch.com/b? - HIER_NONE/- text/html
1505665579.785     79 192.168.0.4 TCP_MISS/304 341 GET http://img.en25.com/i/elqCfg.min.js - HIER_DIRECT/23.207.182.156 application/x-javascript
1505665579.794    197 192.168.0.4 TCP_MISS/200 8677 GET http://uid1.vindicosuite.com/js/tm.js? - HIER_DIRECT/35.186.160.37 application/x-javascript
1505665579.802     64 192.168.0.4 TCP_MISS/304 577 GET http://cdn.optimizely.com/js/103816086.js - HIER_DIRECT/2.21.189.139 text/javascript
1505665579.852      3 192.168.0.4 TCP_DENIED/403 6207 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665579.928     83 192.168.0.4 TCP_MISS/200 619 GET http://uconnect.tealiumiq.com/ulog/_error? - HIER_DIRECT/54.72.62.235 image/gif
1505665579.939      7 192.168.0.4 TCP_DENIED/403 10853 GET http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665579.989      2 192.168.0.4 TCP_DENIED/403 3718 CONNECT googleads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665580.067      3 192.168.0.4 TCP_DENIED/403 6209 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665580.696      3 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665580.728      3 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665580.756      3 192.168.0.4 TCP_DENIED/403 4061 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665580.758      2 192.168.0.4 TCP_DENIED/403 4061 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665580.763      2 192.168.0.4 TCP_DENIED/403 4446 GET http://r.skimresources.com/api/? - HIER_NONE/- text/html
1505665580.829      2 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665581.101    222 192.168.0.4 TCP_MISS/200 413 GET http://s1642912926.t.eloqua.com/visitor/v200/svrGP? - HIER_DIRECT/142.0.160.13 image/gif
1505665581.101    225 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665581.462     53 192.168.0.4 TCP_MISS/200 726 GET http://0914.global.ssl.fastly.net/ad2/script/x.js? - HIER_DIRECT/151.101.60.249 text/javascript
1505665581.492     44 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665581.496     43 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665581.661      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665581.666      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665581.732      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665582.039    204 192.168.0.4 TCP_MISS/204 657 GET http://gwiq.globalwebindex.net/gwiq/? - HIER_DIRECT/69.16.175.10 text/javascript
1505665582.099     90 192.168.0.4 TCP_MISS/200 8775 GET http://ksassets.timeincuk.net/wp/uploads/sites/55/2016/10/cropped-nme-site-icon-300x300.jpg - HIER_DIRECT/52.85.74.98 image/jpeg
1505665582.208      2 192.168.0.4 TCP_DENIED/403 4197 GET http://b.scorecardresearch.com/b? - HIER_NONE/- text/html
1505665582.230    114 192.168.0.4 TCP_MISS/304 534 GET http://ksassets.timeincuk.net/wp/uploads/sites/55/2016/10/cropped-nme-site-icon-300x300.jpg - HIER_DIRECT/52.85.74.98 -
1505665582.314    262 192.168.0.4 TCP_MISS/200 605 GET http://po.st/v1/status? - HIER_DIRECT/74.217.253.90 application/javascript
1505665582.414     71 192.168.0.4 TCP_MISS/200 410 GET http://p.po.st/p? - HIER_DIRECT/208.146.36.215 image/gif
1505665586.135      2 192.168.0.4 TCP_DENIED/403 3925 POST http://t.skimresources.com/api/link - HIER_NONE/- text/html
1505665592.588  13319 192.168.0.4 TCP_MISS/200 4157 CONNECT c.disquscdn.com:443 - HIER_DIRECT/104.16.77.166 -
1505665592.589  15014 192.168.0.4 TCP_MISS/200 154 CONNECT cdn.shopify.com:443 - HIER_DIRECT/151.101.62.110 -
1505665592.591  15018 192.168.0.4 TCP_MISS/200 154 CONNECT cdn.shopify.com:443 - HIER_DIRECT/151.101.62.110 -
1505665592.592  15460 192.168.0.4 TCP_MISS/200 213 CONNECT csi.gstatic.com:443 - HIER_DIRECT/216.58.201.163 -
1505665592.592  13326 192.168.0.4 TCP_MISS/200 3666 CONNECT disqus.com:443 - HIER_DIRECT/151.101.64.134 -
1505665601.000      4 192.168.0.4 TCP_DENIED/403 6210 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665601.173      4 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665601.173      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665603.359      8 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665603.359      3 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665603.557    150 192.168.0.4 TCP_MISS/200 27785 GET http://www.nme.com/ - HIER_DIRECT/54.230.11.205 text/html
1505665603.560    200 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665603.720      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665603.889    173 192.168.0.4 TCP_MISS/200 364 GET http://www.summerhamster.com/bcn? - HIER_DIRECT/52.27.8.169 image/gif
1505665605.489     24 192.168.0.4 TCP_MISS/304 446 GET http://dnn506yrbagrg.cloudfront.net/pages/scripts/0025/8842.js - HIER_DIRECT/54.230.11.179 -
1505665605.595     24 192.168.0.4 TCP_MISS/304 341 GET http://img.en25.com/i/elqCfg.min.js - HIER_DIRECT/23.207.182.156 application/x-javascript
1505665605.601     34 192.168.0.4 TCP_MISS/200 576 GET http://mpp.vindicosuite.com/bg/? - HIER_DIRECT/130.211.103.172 application/javascript
1505665605.615      3 192.168.0.4 TCP_DENIED/403 4350 POST http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665605.620      2 192.168.0.4 TCP_DENIED/403 4176 GET http://b.scorecardresearch.com/b? - HIER_NONE/- text/html
1505665605.679      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665605.686      4 192.168.0.4 TCP_DENIED/403 6207 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665605.734      6 192.168.0.4 TCP_DENIED/403 10875 GET http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665605.745      2 192.168.0.4 TCP_DENIED/403 3718 CONNECT googleads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665605.748    207 192.168.0.4 TCP_MISS/200 8677 GET http://uid1.vindicosuite.com/js/tm.js? - HIER_DIRECT/35.186.160.37 application/x-javascript
1505665605.776     38 192.168.0.4 TCP_MISS/200 619 GET http://uconnect.tealiumiq.com/ulog/_error? - HIER_DIRECT/54.72.62.235 image/gif
1505665605.882      3 192.168.0.4 TCP_DENIED/403 4062 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665605.887      3 192.168.0.4 TCP_DENIED/403 4062 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665605.892      3 192.168.0.4 TCP_DENIED/403 4446 GET http://r.skimresources.com/api/? - HIER_NONE/- text/html
1505665605.998    111 192.168.0.4 TCP_MISS/200 413 GET http://s1642912926.t.eloqua.com/visitor/v200/svrGP? - HIER_DIRECT/142.0.160.13 image/gif
1505665606.169      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665606.175      2 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665606.669    208 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665606.788     22 192.168.0.4 TCP_MISS/200 728 GET http://0914.global.ssl.fastly.net/ad2/script/x.js? - HIER_DIRECT/151.101.60.249 text/javascript
1505665606.813     22 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665606.817     24 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665607.000      3 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665607.003      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
^C
root@raspberrypi:/etc/squid3#

Looks like there is a bunch of them! To check let's pipe the access file through cordcount looking for TCP_DENIED

root@raspberrypi:/etc/squid3# cat /var/log/squid3/access.log | grep TCP_DENIED | wc
     44     440    5192
root@raspberrypi:/etc/squid3#

There we are - there have been 44 instances that have been blocked.

Eyeball the Websites

NME was totally successful - empty spaces in the screen where the blocked ads would've been seen. In this instance two ads have been blocked.

The native Android app Whisper was partially successful. The main ad shows a diagnostic that the ad failed to launch - I'm happy with that - but it failed to block the small ad at the bottom of the screen.

Automating the Fetching of the Blacklist

The list should be refreshed occasionally - like every week - in case there are new entries. The best way of achieving this is to put the earlier curl command in a shell script and then add a cron entry.
/etc/squid3/newads.sh

#!/bin/bash
curl -sS -L --compressed "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=0&mimetype=plaintext" > /etc/squid3/ad_block.txt 
 
## restart squid
/usr/sbin/squid3 restart

And now ad an entry to the root crontab - I have elected to download the script once a week just into Sunday morning.
crontab -e

# Squid3 fetch ad blocker blacklist
5 4 * * 1 /etc/squid3/newads.sh  > /dev/null 2>&1

blog terms

Squid Raspberry Pi Linux Hardware

Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox

nigel — Sat, 09 Sep 2017 21:43:47 +0000

Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox nigel Sat, 09/09/2017 - 22:43

This blog shows you how to build a Squid proxy server on an inexpensive Raspberry Pi machine to help to test web application development in a local VM sandbox.

So here's the problem. Say you are using a VirtualBox hosting Linux in a MacBook to develop your web application. This is practically an industry standard these days; certainly in the open source world anyway. You will have defined the IP address allocated to your VM in your /etc/hosts along with the domain name you have chosen.

You have elected a 'browser first' strategy for the markup, but how do you test your website against hand held devices? Option 1 is using the simulator in Chrome and other browsers which do a very good job of emulating most common devices, but they aren't perfect and frequently the results will differ between the simulator and the physical device. But since it isn't possible to edit the /etc/hosts file in hand held devices, how do you point them to your arbitrarily named website domain? Well the second option is to use an external servicessuch as localtunnel that can expose an IP address to the Internet - but a local service must be running constantly on the VM and firewalls have to be configured to allow incoming http requests which is not always desirable.

There is a third way - use a proxy server to redirect the urls tapped into the hand held devices to the VM in your host machine (Mac / Windows or Linux). The easiest way to achieve this is to use Squid - a mature and free proxy server used throughout the IT industry. It has immense capabilities - such as blacklisting genres of websites (e.g. porn / sex sites) so young family members aren't exposed to what they shouldn't be. But we really only need simple redirection.

So my choice is Squid which runs on Linux. Should I buy even more new hardware for my apartment? Or repurpose equipment from my hardware junkyard in the spare bedroom? Nope. I've always wanted a Raspberry Pi to play with, and now I have an excuse to buy one!

Bill of Materials

Device	Raspberry Pi 3 Model B Quad Core CPU 1.2 GHz 1 GB RAM Motherboard	Amazon.co.uk	£32.50
SD Card	SanDisk Ultra 32 GB microSDHC Class 10 Memory Card up to 48 Mb/s - White/Grey	Amazon.co.uk	£13.50
Box	Raspberry Pi 3 Case - Black/Grey	Amazon.co.uk	£5.50
Keyboard	Mofii Full-size Colorful Wired USB LED Backlit Gaming Keyboard for PC & MAC - UK Layout	Amazon.co.uk	£12.99
Power Cable	2M / 6.5ft USB 2.0 A To MICRO B Cable Data and Charging Lead Cable - Black	eBay.co.uk	£1.65
Wall Plug	DC5V 2A UK 3pin USB Charger Wall Plug Power Adapter	eBay.co.uk	£3.45
Mouse	USB Mouse - spare	n/a	£-
Network Cable	RJ45 CAT5 cable - spare	n/a	£-
	TOTAL		£68.80

So the total price was significantly less than buying a new desktop or laptop computer and obviously the USB keyboard can be reused on other projects. I could've reduced my exposure by sharing phone charging cables around my apartment rather than buy another one to be dedicated for the Raspberry Pi but I wanted a 5V 2A power supply so it made sense to spend a small amount of extra money.

Building the System

There are plenty of tutorials on the Internet for setting up the base Raspberry Pi operation system - so I won't duplicate it here, but please note you will need to install the Raspberry Pi version of Debian Linux (Raspbian) on the SD card.

Here's a list of useful links:

 Raspberry Pi Software Guide
 How to set up and use a Raspberry Pi 3 with a Mac
 SETTING WIFI UP VIA THE COMMAND LINE - Only needed if you are not going to use a static IP address through the LAN connection
 Use Ssh to Talk With Your Raspberry Pi.   

Once your Raspberry Pi is set up and you are ready to go, you can either ssh into the machine (as I did) or connect to it using a keyboard and mouse. If you choose SSH, you will have to enable that service first - see the link above if you are unsure how to do this.

Update the distribution

Always a good idea to update the box in case there are pending patches which weren't in the initial downloaded distribution. When I ran the commands below I got a welter of updates (which I've removed from the blow listing due to brevity reasons)

pi@raspberrypi:~ $ sudo apt-get update
pi@raspberrypi:~ $ sudo apt-get upgrade
pi@raspberrypi:~ $ sudo apt-get dist-upgrade

Set a Static IP Address

The Raspberry Pi 3 does come with WifI so theoretically it is possible to set it up to have a DHCP dynamically assigned IP address. However that would be saving up problems should the IP address change due to rebooting or power interruptions. It would mean a whole load of settings would need to be changed in our system each time that happens. Most routers come with a built-in DHCP server and the range of IP addresses can be set - see below. I have already set my range to be 192.168.0.2 to 192.168.0.199 so anything 200 and above is reserved for devices that need a static address.

I now start up the Raspberry Pi, connected to my LAN with a RJ45 cable, and it gets allocated a dynamic IP address by the DHCP server on my router. This can be checked through the router UI (see below) or by issuing ifconfig when logged on to the Pi.

Become the super user and edit the /etc/dhcpcd.conf file with vi or nano.

pi@raspberrypi:~ $ sudo su
root@raspberrypi:/home/pi# cd /etc/
root@raspberrypi:/etc# vi dhcpcd.conf

Now add the following text to the end of it. Note that I am just using the tail command to list the edits I have already made.

root@raspberrypi:/etc# tail -5 dhcpcd.conf
 
interface eth0
static ip_address=192.168.0.201/24        
static routers=192.168.0.1
static domain_name_servers=90.207.238.97
root@raspberrypi:/etc#

You will need to change the values according to your circumstances. I am using 201 for my Pi server, 1 is my router and the domain name server setting is pointing to my ISP's DNS address. Now reboot the Pi.

root@raspberrypi:/etc# init 6
Connection to 192.168.0.12 closed by remote host.
Connection to 192.168.0.12 closed.

Once the device has completed its startup sequence, go back to your router and check you've got an entry for your Pi - see screenshot below. My two servers can be seen highlighted - FreeNAS using 200 and the Pi at 201. We are good to go!

Install Squid

Ok let's install Squid

pi@raspberrypi:~ $ sudo apt-get install squid3
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libecap2 libmnl0 libnetfilter-conntrack3 squid-langpack squid3-common
Suggested packages:
  squidclient squid-cgi squid-purge smbclient ufw winbindd
The following NEW packages will be installed:
  libecap2 libmnl0 libnetfilter-conntrack3 squid-langpack squid3 squid3-common
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,370 kB of archives.
After this operation, 8,372 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://mirrordirector.raspbian.org/raspbian/ jessie/main libmnl0 armhf 1.0.3-5 [10.9 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ jessie/main libnetfilter-conntrack3 armhf 1.0.4-1 [40.0 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid-langpack all 20140506-1 [150 kB]
Get:4 http://mirrordirector.raspbian.org/raspbian/ jessie/main libecap2 armhf 0.2.0-3 [13.8 kB]
Get:5 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid3-common all 3.4.8-6+deb8u4 [259 kB]
Get:6 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid3 armhf 3.4.8-6+deb8u4 [1,897 kB]
Fetched 2,370 kB in 3s (634 kB/s)   
Selecting previously unselected package libmnl0:armhf.
(Reading database ... 115447 files and directories currently installed.)
Preparing to unpack .../libmnl0_1.0.3-5_armhf.deb ...
Unpacking libmnl0:armhf (1.0.3-5) ...
Selecting previously unselected package libnetfilter-conntrack3:armhf.
Preparing to unpack .../libnetfilter-conntrack3_1.0.4-1_armhf.deb ...
Unpacking libnetfilter-conntrack3:armhf (1.0.4-1) ...
Selecting previously unselected package squid-langpack.
Preparing to unpack .../squid-langpack_20140506-1_all.deb ...
Unpacking squid-langpack (20140506-1) ...
Selecting previously unselected package libecap2.
Preparing to unpack .../libecap2_0.2.0-3_armhf.deb ...
Unpacking libecap2 (0.2.0-3) ...
Selecting previously unselected package squid3-common.
Preparing to unpack .../squid3-common_3.4.8-6+deb8u4_all.deb ...
Unpacking squid3-common (3.4.8-6+deb8u4) ...
Selecting previously unselected package squid3.
Preparing to unpack .../squid3_3.4.8-6+deb8u4_armhf.deb ...
Unpacking squid3 (3.4.8-6+deb8u4) ...
Processing triggers for man-db (2.7.5-1~bpo8+1) ...
Processing triggers for systemd (215-17+deb8u7) ...
Setting up libmnl0:armhf (1.0.3-5) ...
Setting up libnetfilter-conntrack3:armhf (1.0.4-1) ...
Setting up squid-langpack (20140506-1) ...
Setting up libecap2 (0.2.0-3) ...
Setting up squid3-common (3.4.8-6+deb8u4) ...
Setting up squid3 (3.4.8-6+deb8u4) ...
Creating Squid HTTP proxy 3.x spool directory structure
2017/07/30 18:55:00 kid1| Set Current Directory to /var/spool/squid3
2017/07/30 18:55:00 kid1| Creating missing swap directories
2017/07/30 18:55:00 kid1| No cache_dir stores are configured.
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Processing triggers for systemd (215-17+deb8u7) ...
pi@raspberrypi:~ $

Configuring Squid

First change directory to the squid configuration Squid directory and save the original configuration file just in case it needs to revered sometime.

root@raspberrypi:~# cd /etc/squid3/
root@raspberrypi:/etc/squid3# cp squid.conf squid.conf.original
root@raspberrypi:/etc/squid3#

By default Squid doesn't allow any access to the Internet so that will need fixing. Like most ACL based systems, once Squid finds a rule that applies to a particular URL then it stops looking for more rules. So early in the config file you need to tell Squid to allow all HTTP traffic.

http_access allow all

You will also need to tell Squid about your local network. I am using the 192.168.0.0 range of IP addresses - find that in the configuration and uncomment it

acl localnet src 192.168.0.0/16	# RFC1918 possible internal network

Finally we don't want any caching since we will be using our Squid mainly for redirection purposes and when recent website changes need to be seen for testing purposes, so add this at the caching section

cache deny all

Below is a list of the changes in a form of a diff file which some may find more useful than the separate settings I've mentioned above.

root@raspberrypi:/etc/squid3# diff -Naur  squid.conf.original squid.conf
--- squid.conf.original	2017-09-02 19:22:23.804970154 +0000
+++ squid.conf	2017-09-03 10:41:48.190318779 +0000
@@ -1037,10 +1037,11 @@
 # should be allowed
 #acl localnet src 10.0.0.0/8	# RFC1918 possible internal network
 #acl localnet src 172.16.0.0/12	# RFC1918 possible internal network
-#acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
+acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
 #acl localnet src fc00::/7       # RFC 4193 local private network range
 #acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines
 
+
 acl SSL_ports port 443
 acl Safe_ports port 80		# http
 acl Safe_ports port 21		# ftp
@@ -1180,6 +1181,7 @@
 #Default:
 # Deny, unless rules exist in squid.conf.
 #
+http_access allow all
 
 #
 # Recommended minimum Access Permission configuration:
@@ -2999,6 +3001,9 @@
 # DISK CACHE OPTIONS
 # -----------------------------------------------------------------------------
 
+cache deny all
+
+
 #  TAG: cache_replacement_policy
 #	The cache replacement policy parameter determines which
 #	objects are evicted (replaced) when disk space is needed.
root@raspberrypi:/etc/squid3#

We are now done with the configuration in the conf file so we can restart the Squid service so these changes take effect

root@raspberrypi:/etc/squid3# service squid3 restart
root@raspberrypi:/etc/squid3#

Won't be the worst idea to copy the configuration just for safety.

root@raspberrypi:/etc/squid3# cp squid.conf squid.conf.local-vm
root@raspberrypi:/etc/squid3#

You are now probably wondering how does Squid possibly know where my Virtual Hosts in my Virtual Machine are!! This is the simple bit :) - you need to add all those sites you are working on into the /etc/hosts file on your Pi. You will already have these listed in your Macbook hosts file so simply copy and paste. My file ended up looking like this below:

root@raspberrypi:/etc/squid3# cat /etc/hosts
127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters
 
127.0.1.1	raspberrypi
 
192.168.0.14 badzilla-d8.localhost
192.168.0.14 meedjum.localhost
root@raspberrypi:/etc/squid3#

The last two entries - badzilla-d8.localhost and meedjum.localhost are the two sites I am working on. The migration of Badzilla from D6 to D8. Yey!

Handheld Device Client Configuration

We are now ready to test our solution, and for this I'll use screenshots from Android and iOS.

Android

Go to Settings->WiFi, then long tap on your network and tap 'Modify network config', then tap 'Show advanced options' and add the configuration for the proxy, then tap save.

Now point a web browser to your web development domain

iOS

Go to settings, tap on your WiFi, tap on the information icon, tap on 'Manual' under HTTP Proxy, and enter the values.

Now point a web browser to your web development domain - this is very early in the process as you can see. Somehow two seach boxes are on this early build!

blog terms

Raspberry Pi Squid Linux Drupal Development VirtualBox