Squid

Ad Blocker using Squid Proxy on Raspberry Pi

nigel — Sun, 17 Sep 2017 15:33:40 +0000

Ad Blocker using Squid Proxy on Raspberry Pi nigel Sun, 17/09/2017 - 16:33

As mentioned in my previous blog Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox it is possible (in fact quite usual) to use Squid to prevent user access to certain websites. This can include ad servers which serve up annoying ad content when trying to read articles. This tutorial shows how we can extend the functionality of the Squid server we have already set up to block ads.

The banning of ad content is achieved by downloading a blacklist of prohibited urls from a service provider, although it has to be mentioned that these lists depend upon the community to contribute any urls they discover and thus there can never be 100% confidence that such a list is ever complete.

Once the list is downloaded from the service provider, Squid needs to be informed of its presence in its configuration and the ban will take the form of an acl. Ok now we know what we need to achieve let's start the process.

Lets log into the Raspberry Pi, switch to superuser, and download the blacklist.

Nigels-MacBook-Pro:Projects nigel$ ssh pi@192.168.0.201
pi@192.168.0.201s password: 
 
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
 
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Sep 10 14:33:42 2017 from 192.168.0.7
pi@raspberrypi:~ $ sudo su
root@raspberrypi:/home/pi# cd /etc/squid3/
root@raspberrypi:/etc/squid3# ls -las
total 832
  4 drwxr-xr-x   2 root root   4096 Sep  9 08:44 .
  4 drwxr-xr-x 112 root root   4096 Sep  9 11:17 ..
  4 -rw-r--r--   1 root root   1547 Dec 24  2016 errorpage.css
  4 -rw-r--r--   1 root root    421 Dec 24  2016 msntauth.conf
272 -rw-r--r--   1 root root 277585 Sep  9 08:44 squid.conf
272 -rw-r--r--   1 root root 277565 Sep  3 11:06 squid.conf.local-vm
272 -rw-r--r--   1 root root 277526 Sep  2 19:22 squid.conf.original
root@raspberrypi:/etc/squid3# curl -sS -L --compressed "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=0&mimetype=plaintext" > ad_block.txt 
root@raspberrypi:/etc/squid3# ls -las
total 872
  4 drwxr-xr-x   2 root root   4096 Sep 17 15:20 .
  4 drwxr-xr-x 112 root root   4096 Sep  9 11:17 ..
 40 -rw-r--r--   1 root root  40042 Sep 17 15:20 ad_block.txt
  4 -rw-r--r--   1 root root   1547 Dec 24  2016 errorpage.css
  4 -rw-r--r--   1 root root    421 Dec 24  2016 msntauth.conf
272 -rw-r--r--   1 root root 277585 Sep  9 08:44 squid.conf
272 -rw-r--r--   1 root root 277565 Sep  3 11:06 squid.conf.local-vm
272 -rw-r--r--   1 root root 277526 Sep  2 19:22 squid.conf.original
root@raspberrypi:/etc/squid3#

The blacklist is now in place - you can see it in the directory listing as file ad_block.txt. The next job is to add this to the Squid configuration as an acl. This needs to be placed above the http_access allow all line we put in the config in the previous blog.

## disable ads ( http://pgl.yoyo.org/adservers/ )
acl ads dstdom_regex "/etc/squid3/ad_block.txt"
http_access deny ads

You can use your favourite editor for this - I used vi - and then reboot the server.

root@raspberrypi:/etc/squid3# vi ad_block.txt 
root@raspberrypi:/etc/squid3# service squid3 restart

We are now ready to test our solution by landing on an ad heavy website such as NME. There are two excellent methods of doing this - one is checking the access.log in real time for TCP_DENIED records, and the other is to eyeball the site and see if the ads really are missing. Let's go for option one first.

root@raspberrypi:/etc/squid3# tail -f /var/log/squid3/access.log
1505665437.619     38 192.168.0.4 TCP_MISS/200 630 GET http://pagead2.googlesyndication.com/activeview? - HIER_DIRECT/216.58.208.130 image/gif
1505665440.152   3632 192.168.0.4 TCP_MISS/200 4009 CONNECT bs.serving-sys.com:443 - HIER_DIRECT/82.199.68.72 -
1505665449.828   8361 192.168.0.4 TCP_MISS/200 630 CONNECT ml314.com:443 - HIER_DIRECT/34.252.181.159 -
1505665452.178     78 192.168.0.4 TCP_MISS/204 740 GET http://beacon.krxd.net/event.gif? - HIER_DIRECT/54.228.222.160 image/gif
1505665491.360  69091 192.168.0.4 TCP_MISS/200 746 CONNECT www.facebook.com:443 - HIER_DIRECT/31.13.90.36 -
1505665493.069  65946 192.168.0.4 TCP_MISS/200 5038 CONNECT match.adsrvr.org:443 - HIER_DIRECT/54.247.91.116 -
1505665502.280  65538 192.168.0.4 TCP_MISS/200 5552 CONNECT odr.mookie1.com:443 - HIER_DIRECT/35.157.149.45 -
1505665519.263  91582 192.168.0.4 TCP_MISS/200 3318 CONNECT bam.nr-data.net:443 - HIER_DIRECT/162.247.242.21 -
1505665544.650 120399 192.168.0.4 TCP_MISS/200 1227 CONNECT session.timecommerce.net:443 - HIER_DIRECT/54.192.2.48 -
1505665554.600      7 192.168.0.4 TCP_MISS/503 4491 GET http://pagead2.googlesyndication.com/activeview? - HIER_NONE/- text/html
1505665577.132      8 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665577.146      9 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665577.357    200 192.168.0.4 TCP_MISS/200 27785 GET http://www.nme.com/ - HIER_DIRECT/54.230.11.205 text/html
1505665577.396    249 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665577.495      7 192.168.0.4 TCP_DENIED/403 4577 GET http://pagead2.googlesyndication.com/activeview? - HIER_NONE/- text/html
1505665577.567      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665577.615      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665577.748    337 192.168.0.4 TCP_MISS/200 364 GET http://www.summerhamster.com/bcn? - HIER_DIRECT/52.27.8.169 image/gif
1505665579.364     58 192.168.0.4 TCP_MISS/304 446 GET http://dnn506yrbagrg.cloudfront.net/pages/scripts/0025/8842.js - HIER_DIRECT/54.230.11.179 -
1505665579.380      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665579.669     69 192.168.0.4 TCP_MISS/200 576 GET http://mpp.vindicosuite.com/bg/? - HIER_DIRECT/130.211.103.172 application/javascript
1505665579.676      4 192.168.0.4 TCP_DENIED/403 4350 POST http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665579.761      3 192.168.0.4 TCP_DENIED/403 4176 GET http://b.scorecardresearch.com/b? - HIER_NONE/- text/html
1505665579.785     79 192.168.0.4 TCP_MISS/304 341 GET http://img.en25.com/i/elqCfg.min.js - HIER_DIRECT/23.207.182.156 application/x-javascript
1505665579.794    197 192.168.0.4 TCP_MISS/200 8677 GET http://uid1.vindicosuite.com/js/tm.js? - HIER_DIRECT/35.186.160.37 application/x-javascript
1505665579.802     64 192.168.0.4 TCP_MISS/304 577 GET http://cdn.optimizely.com/js/103816086.js - HIER_DIRECT/2.21.189.139 text/javascript
1505665579.852      3 192.168.0.4 TCP_DENIED/403 6207 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665579.928     83 192.168.0.4 TCP_MISS/200 619 GET http://uconnect.tealiumiq.com/ulog/_error? - HIER_DIRECT/54.72.62.235 image/gif
1505665579.939      7 192.168.0.4 TCP_DENIED/403 10853 GET http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665579.989      2 192.168.0.4 TCP_DENIED/403 3718 CONNECT googleads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665580.067      3 192.168.0.4 TCP_DENIED/403 6209 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665580.696      3 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665580.728      3 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665580.756      3 192.168.0.4 TCP_DENIED/403 4061 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665580.758      2 192.168.0.4 TCP_DENIED/403 4061 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665580.763      2 192.168.0.4 TCP_DENIED/403 4446 GET http://r.skimresources.com/api/? - HIER_NONE/- text/html
1505665580.829      2 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665581.101    222 192.168.0.4 TCP_MISS/200 413 GET http://s1642912926.t.eloqua.com/visitor/v200/svrGP? - HIER_DIRECT/142.0.160.13 image/gif
1505665581.101    225 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665581.462     53 192.168.0.4 TCP_MISS/200 726 GET http://0914.global.ssl.fastly.net/ad2/script/x.js? - HIER_DIRECT/151.101.60.249 text/javascript
1505665581.492     44 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665581.496     43 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665581.661      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665581.666      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665581.732      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665582.039    204 192.168.0.4 TCP_MISS/204 657 GET http://gwiq.globalwebindex.net/gwiq/? - HIER_DIRECT/69.16.175.10 text/javascript
1505665582.099     90 192.168.0.4 TCP_MISS/200 8775 GET http://ksassets.timeincuk.net/wp/uploads/sites/55/2016/10/cropped-nme-site-icon-300x300.jpg - HIER_DIRECT/52.85.74.98 image/jpeg
1505665582.208      2 192.168.0.4 TCP_DENIED/403 4197 GET http://b.scorecardresearch.com/b? - HIER_NONE/- text/html
1505665582.230    114 192.168.0.4 TCP_MISS/304 534 GET http://ksassets.timeincuk.net/wp/uploads/sites/55/2016/10/cropped-nme-site-icon-300x300.jpg - HIER_DIRECT/52.85.74.98 -
1505665582.314    262 192.168.0.4 TCP_MISS/200 605 GET http://po.st/v1/status? - HIER_DIRECT/74.217.253.90 application/javascript
1505665582.414     71 192.168.0.4 TCP_MISS/200 410 GET http://p.po.st/p? - HIER_DIRECT/208.146.36.215 image/gif
1505665586.135      2 192.168.0.4 TCP_DENIED/403 3925 POST http://t.skimresources.com/api/link - HIER_NONE/- text/html
1505665592.588  13319 192.168.0.4 TCP_MISS/200 4157 CONNECT c.disquscdn.com:443 - HIER_DIRECT/104.16.77.166 -
1505665592.589  15014 192.168.0.4 TCP_MISS/200 154 CONNECT cdn.shopify.com:443 - HIER_DIRECT/151.101.62.110 -
1505665592.591  15018 192.168.0.4 TCP_MISS/200 154 CONNECT cdn.shopify.com:443 - HIER_DIRECT/151.101.62.110 -
1505665592.592  15460 192.168.0.4 TCP_MISS/200 213 CONNECT csi.gstatic.com:443 - HIER_DIRECT/216.58.201.163 -
1505665592.592  13326 192.168.0.4 TCP_MISS/200 3666 CONNECT disqus.com:443 - HIER_DIRECT/151.101.64.134 -
1505665601.000      4 192.168.0.4 TCP_DENIED/403 6210 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665601.173      4 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665601.173      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665603.359      8 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665603.359      3 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665603.557    150 192.168.0.4 TCP_MISS/200 27785 GET http://www.nme.com/ - HIER_DIRECT/54.230.11.205 text/html
1505665603.560    200 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665603.720      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665603.889    173 192.168.0.4 TCP_MISS/200 364 GET http://www.summerhamster.com/bcn? - HIER_DIRECT/52.27.8.169 image/gif
1505665605.489     24 192.168.0.4 TCP_MISS/304 446 GET http://dnn506yrbagrg.cloudfront.net/pages/scripts/0025/8842.js - HIER_DIRECT/54.230.11.179 -
1505665605.595     24 192.168.0.4 TCP_MISS/304 341 GET http://img.en25.com/i/elqCfg.min.js - HIER_DIRECT/23.207.182.156 application/x-javascript
1505665605.601     34 192.168.0.4 TCP_MISS/200 576 GET http://mpp.vindicosuite.com/bg/? - HIER_DIRECT/130.211.103.172 application/javascript
1505665605.615      3 192.168.0.4 TCP_DENIED/403 4350 POST http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665605.620      2 192.168.0.4 TCP_DENIED/403 4176 GET http://b.scorecardresearch.com/b? - HIER_NONE/- text/html
1505665605.679      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665605.686      4 192.168.0.4 TCP_DENIED/403 6207 GET http://as.casalemedia.com/cygnus? - HIER_NONE/- text/html
1505665605.734      6 192.168.0.4 TCP_DENIED/403 10875 GET http://timeinc.demdex.net/event? - HIER_NONE/- text/html
1505665605.745      2 192.168.0.4 TCP_DENIED/403 3718 CONNECT googleads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665605.748    207 192.168.0.4 TCP_MISS/200 8677 GET http://uid1.vindicosuite.com/js/tm.js? - HIER_DIRECT/35.186.160.37 application/x-javascript
1505665605.776     38 192.168.0.4 TCP_MISS/200 619 GET http://uconnect.tealiumiq.com/ulog/_error? - HIER_DIRECT/54.72.62.235 image/gif
1505665605.882      3 192.168.0.4 TCP_DENIED/403 4062 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665605.887      3 192.168.0.4 TCP_DENIED/403 4062 GET http://p.skimresources.com/px.gif? - HIER_NONE/- text/html
1505665605.892      3 192.168.0.4 TCP_DENIED/403 4446 GET http://r.skimresources.com/api/? - HIER_NONE/- text/html
1505665605.998    111 192.168.0.4 TCP_MISS/200 413 GET http://s1642912926.t.eloqua.com/visitor/v200/svrGP? - HIER_DIRECT/142.0.160.13 image/gif
1505665606.169      2 192.168.0.4 TCP_DENIED/403 3727 CONNECT securepubads.g.doubleclick.net:443 - HIER_NONE/- text/html
1505665606.175      2 192.168.0.4 TCP_DENIED/403 4014 POST http://as.casalemedia.com/headerstats? - HIER_NONE/- text/html
1505665606.669    208 192.168.0.4 TCP_MISS/200 252 GET http://uid1.vindicosuite.com/e/? - HIER_DIRECT/35.186.160.37 text/plain
1505665606.788     22 192.168.0.4 TCP_MISS/200 728 GET http://0914.global.ssl.fastly.net/ad2/script/x.js? - HIER_DIRECT/151.101.60.249 text/javascript
1505665606.813     22 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665606.817     24 192.168.0.4 TCP_MISS/200 759 GET http://0914.global.ssl.fastly.net/ad2/img/x.gif? - HIER_DIRECT/151.101.60.249 image/gif
1505665607.000      3 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
1505665607.003      2 192.168.0.4 TCP_DENIED/403 3709 CONNECT www.google-analytics.com:443 - HIER_NONE/- text/html
^C
root@raspberrypi:/etc/squid3#

Looks like there is a bunch of them! To check let's pipe the access file through cordcount looking for TCP_DENIED

root@raspberrypi:/etc/squid3# cat /var/log/squid3/access.log | grep TCP_DENIED | wc
     44     440    5192
root@raspberrypi:/etc/squid3#

There we are - there have been 44 instances that have been blocked.

Eyeball the Websites

NME was totally successful - empty spaces in the screen where the blocked ads would've been seen. In this instance two ads have been blocked.

The native Android app Whisper was partially successful. The main ad shows a diagnostic that the ad failed to launch - I'm happy with that - but it failed to block the small ad at the bottom of the screen.

Automating the Fetching of the Blacklist

The list should be refreshed occasionally - like every week - in case there are new entries. The best way of achieving this is to put the earlier curl command in a shell script and then add a cron entry.
/etc/squid3/newads.sh

#!/bin/bash
curl -sS -L --compressed "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=0&mimetype=plaintext" > /etc/squid3/ad_block.txt 
 
## restart squid
/usr/sbin/squid3 restart

And now ad an entry to the root crontab - I have elected to download the script once a week just into Sunday morning.
crontab -e

# Squid3 fetch ad blocker blacklist
5 4 * * 1 /etc/squid3/newads.sh  > /dev/null 2>&1

blog terms

Squid Raspberry Pi Linux Hardware

Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox

nigel — Sat, 09 Sep 2017 21:43:47 +0000

Raspberry Pi with Squid Proxy Server for Testing Hand-Held Devices when Developing in a Sandbox nigel Sat, 09/09/2017 - 22:43

This blog shows you how to build a Squid proxy server on an inexpensive Raspberry Pi machine to help to test web application development in a local VM sandbox.

So here's the problem. Say you are using a VirtualBox hosting Linux in a MacBook to develop your web application. This is practically an industry standard these days; certainly in the open source world anyway. You will have defined the IP address allocated to your VM in your /etc/hosts along with the domain name you have chosen.

You have elected a 'browser first' strategy for the markup, but how do you test your website against hand held devices? Option 1 is using the simulator in Chrome and other browsers which do a very good job of emulating most common devices, but they aren't perfect and frequently the results will differ between the simulator and the physical device. But since it isn't possible to edit the /etc/hosts file in hand held devices, how do you point them to your arbitrarily named website domain? Well the second option is to use an external servicessuch as localtunnel that can expose an IP address to the Internet - but a local service must be running constantly on the VM and firewalls have to be configured to allow incoming http requests which is not always desirable.

There is a third way - use a proxy server to redirect the urls tapped into the hand held devices to the VM in your host machine (Mac / Windows or Linux). The easiest way to achieve this is to use Squid - a mature and free proxy server used throughout the IT industry. It has immense capabilities - such as blacklisting genres of websites (e.g. porn / sex sites) so young family members aren't exposed to what they shouldn't be. But we really only need simple redirection.

So my choice is Squid which runs on Linux. Should I buy even more new hardware for my apartment? Or repurpose equipment from my hardware junkyard in the spare bedroom? Nope. I've always wanted a Raspberry Pi to play with, and now I have an excuse to buy one!

Bill of Materials

Device	Raspberry Pi 3 Model B Quad Core CPU 1.2 GHz 1 GB RAM Motherboard	Amazon.co.uk	£32.50
SD Card	SanDisk Ultra 32 GB microSDHC Class 10 Memory Card up to 48 Mb/s - White/Grey	Amazon.co.uk	£13.50
Box	Raspberry Pi 3 Case - Black/Grey	Amazon.co.uk	£5.50
Keyboard	Mofii Full-size Colorful Wired USB LED Backlit Gaming Keyboard for PC & MAC - UK Layout	Amazon.co.uk	£12.99
Power Cable	2M / 6.5ft USB 2.0 A To MICRO B Cable Data and Charging Lead Cable - Black	eBay.co.uk	£1.65
Wall Plug	DC5V 2A UK 3pin USB Charger Wall Plug Power Adapter	eBay.co.uk	£3.45
Mouse	USB Mouse - spare	n/a	£-
Network Cable	RJ45 CAT5 cable - spare	n/a	£-
	TOTAL		£68.80

So the total price was significantly less than buying a new desktop or laptop computer and obviously the USB keyboard can be reused on other projects. I could've reduced my exposure by sharing phone charging cables around my apartment rather than buy another one to be dedicated for the Raspberry Pi but I wanted a 5V 2A power supply so it made sense to spend a small amount of extra money.

Building the System

There are plenty of tutorials on the Internet for setting up the base Raspberry Pi operation system - so I won't duplicate it here, but please note you will need to install the Raspberry Pi version of Debian Linux (Raspbian) on the SD card.

Here's a list of useful links:

 Raspberry Pi Software Guide
 How to set up and use a Raspberry Pi 3 with a Mac
 SETTING WIFI UP VIA THE COMMAND LINE - Only needed if you are not going to use a static IP address through the LAN connection
 Use Ssh to Talk With Your Raspberry Pi.   

Once your Raspberry Pi is set up and you are ready to go, you can either ssh into the machine (as I did) or connect to it using a keyboard and mouse. If you choose SSH, you will have to enable that service first - see the link above if you are unsure how to do this.

Update the distribution

Always a good idea to update the box in case there are pending patches which weren't in the initial downloaded distribution. When I ran the commands below I got a welter of updates (which I've removed from the blow listing due to brevity reasons)

pi@raspberrypi:~ $ sudo apt-get update
pi@raspberrypi:~ $ sudo apt-get upgrade
pi@raspberrypi:~ $ sudo apt-get dist-upgrade

Set a Static IP Address

The Raspberry Pi 3 does come with WifI so theoretically it is possible to set it up to have a DHCP dynamically assigned IP address. However that would be saving up problems should the IP address change due to rebooting or power interruptions. It would mean a whole load of settings would need to be changed in our system each time that happens. Most routers come with a built-in DHCP server and the range of IP addresses can be set - see below. I have already set my range to be 192.168.0.2 to 192.168.0.199 so anything 200 and above is reserved for devices that need a static address.

I now start up the Raspberry Pi, connected to my LAN with a RJ45 cable, and it gets allocated a dynamic IP address by the DHCP server on my router. This can be checked through the router UI (see below) or by issuing ifconfig when logged on to the Pi.

Become the super user and edit the /etc/dhcpcd.conf file with vi or nano.

pi@raspberrypi:~ $ sudo su
root@raspberrypi:/home/pi# cd /etc/
root@raspberrypi:/etc# vi dhcpcd.conf

Now add the following text to the end of it. Note that I am just using the tail command to list the edits I have already made.

root@raspberrypi:/etc# tail -5 dhcpcd.conf
 
interface eth0
static ip_address=192.168.0.201/24        
static routers=192.168.0.1
static domain_name_servers=90.207.238.97
root@raspberrypi:/etc#

You will need to change the values according to your circumstances. I am using 201 for my Pi server, 1 is my router and the domain name server setting is pointing to my ISP's DNS address. Now reboot the Pi.

root@raspberrypi:/etc# init 6
Connection to 192.168.0.12 closed by remote host.
Connection to 192.168.0.12 closed.

Once the device has completed its startup sequence, go back to your router and check you've got an entry for your Pi - see screenshot below. My two servers can be seen highlighted - FreeNAS using 200 and the Pi at 201. We are good to go!

Install Squid

Ok let's install Squid

pi@raspberrypi:~ $ sudo apt-get install squid3
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libecap2 libmnl0 libnetfilter-conntrack3 squid-langpack squid3-common
Suggested packages:
  squidclient squid-cgi squid-purge smbclient ufw winbindd
The following NEW packages will be installed:
  libecap2 libmnl0 libnetfilter-conntrack3 squid-langpack squid3 squid3-common
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,370 kB of archives.
After this operation, 8,372 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://mirrordirector.raspbian.org/raspbian/ jessie/main libmnl0 armhf 1.0.3-5 [10.9 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ jessie/main libnetfilter-conntrack3 armhf 1.0.4-1 [40.0 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid-langpack all 20140506-1 [150 kB]
Get:4 http://mirrordirector.raspbian.org/raspbian/ jessie/main libecap2 armhf 0.2.0-3 [13.8 kB]
Get:5 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid3-common all 3.4.8-6+deb8u4 [259 kB]
Get:6 http://mirrordirector.raspbian.org/raspbian/ jessie/main squid3 armhf 3.4.8-6+deb8u4 [1,897 kB]
Fetched 2,370 kB in 3s (634 kB/s)   
Selecting previously unselected package libmnl0:armhf.
(Reading database ... 115447 files and directories currently installed.)
Preparing to unpack .../libmnl0_1.0.3-5_armhf.deb ...
Unpacking libmnl0:armhf (1.0.3-5) ...
Selecting previously unselected package libnetfilter-conntrack3:armhf.
Preparing to unpack .../libnetfilter-conntrack3_1.0.4-1_armhf.deb ...
Unpacking libnetfilter-conntrack3:armhf (1.0.4-1) ...
Selecting previously unselected package squid-langpack.
Preparing to unpack .../squid-langpack_20140506-1_all.deb ...
Unpacking squid-langpack (20140506-1) ...
Selecting previously unselected package libecap2.
Preparing to unpack .../libecap2_0.2.0-3_armhf.deb ...
Unpacking libecap2 (0.2.0-3) ...
Selecting previously unselected package squid3-common.
Preparing to unpack .../squid3-common_3.4.8-6+deb8u4_all.deb ...
Unpacking squid3-common (3.4.8-6+deb8u4) ...
Selecting previously unselected package squid3.
Preparing to unpack .../squid3_3.4.8-6+deb8u4_armhf.deb ...
Unpacking squid3 (3.4.8-6+deb8u4) ...
Processing triggers for man-db (2.7.5-1~bpo8+1) ...
Processing triggers for systemd (215-17+deb8u7) ...
Setting up libmnl0:armhf (1.0.3-5) ...
Setting up libnetfilter-conntrack3:armhf (1.0.4-1) ...
Setting up squid-langpack (20140506-1) ...
Setting up libecap2 (0.2.0-3) ...
Setting up squid3-common (3.4.8-6+deb8u4) ...
Setting up squid3 (3.4.8-6+deb8u4) ...
Creating Squid HTTP proxy 3.x spool directory structure
2017/07/30 18:55:00 kid1| Set Current Directory to /var/spool/squid3
2017/07/30 18:55:00 kid1| Creating missing swap directories
2017/07/30 18:55:00 kid1| No cache_dir stores are configured.
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Processing triggers for systemd (215-17+deb8u7) ...
pi@raspberrypi:~ $

Configuring Squid

First change directory to the squid configuration Squid directory and save the original configuration file just in case it needs to revered sometime.

root@raspberrypi:~# cd /etc/squid3/
root@raspberrypi:/etc/squid3# cp squid.conf squid.conf.original
root@raspberrypi:/etc/squid3#

By default Squid doesn't allow any access to the Internet so that will need fixing. Like most ACL based systems, once Squid finds a rule that applies to a particular URL then it stops looking for more rules. So early in the config file you need to tell Squid to allow all HTTP traffic.

http_access allow all

You will also need to tell Squid about your local network. I am using the 192.168.0.0 range of IP addresses - find that in the configuration and uncomment it

acl localnet src 192.168.0.0/16	# RFC1918 possible internal network

Finally we don't want any caching since we will be using our Squid mainly for redirection purposes and when recent website changes need to be seen for testing purposes, so add this at the caching section

cache deny all

Below is a list of the changes in a form of a diff file which some may find more useful than the separate settings I've mentioned above.

root@raspberrypi:/etc/squid3# diff -Naur  squid.conf.original squid.conf
--- squid.conf.original	2017-09-02 19:22:23.804970154 +0000
+++ squid.conf	2017-09-03 10:41:48.190318779 +0000
@@ -1037,10 +1037,11 @@
 # should be allowed
 #acl localnet src 10.0.0.0/8	# RFC1918 possible internal network
 #acl localnet src 172.16.0.0/12	# RFC1918 possible internal network
-#acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
+acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
 #acl localnet src fc00::/7       # RFC 4193 local private network range
 #acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines
 
+
 acl SSL_ports port 443
 acl Safe_ports port 80		# http
 acl Safe_ports port 21		# ftp
@@ -1180,6 +1181,7 @@
 #Default:
 # Deny, unless rules exist in squid.conf.
 #
+http_access allow all
 
 #
 # Recommended minimum Access Permission configuration:
@@ -2999,6 +3001,9 @@
 # DISK CACHE OPTIONS
 # -----------------------------------------------------------------------------
 
+cache deny all
+
+
 #  TAG: cache_replacement_policy
 #	The cache replacement policy parameter determines which
 #	objects are evicted (replaced) when disk space is needed.
root@raspberrypi:/etc/squid3#

We are now done with the configuration in the conf file so we can restart the Squid service so these changes take effect

root@raspberrypi:/etc/squid3# service squid3 restart
root@raspberrypi:/etc/squid3#

Won't be the worst idea to copy the configuration just for safety.

root@raspberrypi:/etc/squid3# cp squid.conf squid.conf.local-vm
root@raspberrypi:/etc/squid3#

You are now probably wondering how does Squid possibly know where my Virtual Hosts in my Virtual Machine are!! This is the simple bit :) - you need to add all those sites you are working on into the /etc/hosts file on your Pi. You will already have these listed in your Macbook hosts file so simply copy and paste. My file ended up looking like this below:

root@raspberrypi:/etc/squid3# cat /etc/hosts
127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters
 
127.0.1.1	raspberrypi
 
192.168.0.14 badzilla-d8.localhost
192.168.0.14 meedjum.localhost
root@raspberrypi:/etc/squid3#

The last two entries - badzilla-d8.localhost and meedjum.localhost are the two sites I am working on. The migration of Badzilla from D6 to D8. Yey!

Handheld Device Client Configuration

We are now ready to test our solution, and for this I'll use screenshots from Android and iOS.

Android

Go to Settings->WiFi, then long tap on your network and tap 'Modify network config', then tap 'Show advanced options' and add the configuration for the proxy, then tap save.

Now point a web browser to your web development domain

iOS

Go to settings, tap on your WiFi, tap on the information icon, tap on 'Manual' under HTTP Proxy, and enter the values.

Now point a web browser to your web development domain - this is very early in the process as you can see. Somehow two seach boxes are on this early build!

blog terms

Raspberry Pi Squid Linux Drupal Development VirtualBox