Web Content Filtering

From Linuxcentre Wiki

Jump to: navigation, search

Contents

Overview

We use a locally installed web content filter on the existing Squid proxy server using the following open-source software:

Filtering is enforced on all systems and the proxy.pac file will need modifying to force all web requests (and HTTPS) via the Dans Guardian proxy.

  • Direct squid access will need to be blocked as connections should only be allowed from Dans Guardian. (TODO)
  • A special authenticated proxy port could be setup to bypass some of the more restrictive rules for teachers. (TODO)
  • Filters are very conservative and use blocklists from:
* Shallalist URL block list
* URL block list
* Malware block list
* OpenDNS
  • Some filtering categories could be changed after school hours to allow games sites for example. (TODO)
  • Installation of some log analysis tools would be required to monitor the effectiveness of the filtering. (TODO)
  • Scheduled weekly cron jobs could be setup to refresh the filter lists as permitted. (TODO)
  • A simple web front-end would be required to allow teachers/admins to easily override the filtering using simple blacklists and whitelists. If this cannot be found a simple password-protected PHP script could be written. (TODO)
  • When it is available, to allow the more precise filtering of SSL/https website URLs we could use Dynamic SSL certificate generation of the Squid proxy. This will allow interception of all SSL connections. It will also require the installation of a school-wide root CA. Bad for privacy, good for filtering. (TODO)

Other Options

  • Outsource the filtering to an external proxy on the internet. This is never free and we can expect to pay in excess of £200pa for a solution that will not be as responsive and fast as the one proposed.
  • Install filtering software on each and every computer in the school. Although this is done by some schools it really is the most inefficient waste of computing and administrative resources.

Does it work?

Yes. I've even used similar configuration at home for my own children for several years. I've given it a lot of testing and as a parent I am happy with it. No filtering or blocking solution is perfect. However, Dans Guardian is actually a lot better than a lot of 'filtering' products because it actually inspects the contents of the page on-the-fly and doesn't entirely rely on a fixed list of bad websites.

OpenDNS configuration

The opendns.org free familyshield filtering has been used. The nameserver in the network was configured to forward all non-local DNS lookups to the opendns servers:

  • Add this forwarders line to the options section in the file '/etc/named.conf':
options {
        directory       "/var/named/data";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        allow-transfer  { 127.0.0.1; };
        version         "Go Away!";
        recursion       yes;
        forward only;
        #forwarders { 8.8.8.8; };
        # Using OpenDNS.org free familyshield to block adult content
        forwarders { 208.67.222.123; 208.67.220.123; };
};
  • Restart name server:
/etc/init.d/named restart

DansGuardian Configuration

These are the changes we made to the configuratiosn to tune it for local use.

  • /etc/dansguardian/dansguardian.conf:
# listen on port 8081
filterport = 8081
# squid on pot 8080
proxyport = 8080
# scan all pages < 10MB in size (required for large pages like google images)
maxcontentfiltersize = 0
maxcontentramcachescansize = 10240
# ensures that squid access log has source IPs
forwardedfor = on
# Make a few more threads start up
minchildren = 16
minsparechildren = 8
  • /etc/dansguardian/dansguardianf1.conf:
# use PICS ratings
enablepics = on
# also do scanning on embedded URLs
deepurlanalysis = on
  • /etc/dansguardian/lists/bannedextensionlist:
# Comment out all expect these:
.bat  # Batch file
.cmd  # Microsoft Windows NT Command script
.pif  # Shortcut to MS-DOS program
.reg  # Windows registry entries
.scr  # Screen saver
.sct  # Windows Script Component
.sh   # Shell script
.sys  # Windows system file
.vxd  # Windows system file
.wsc  # Windows Script Component
.wsf  # Windows Script file
.wsh  # Windows Script Host Settings file
.otf  # Font file - can be used to instant reboot 2k and xp
.ops  # Office XP settings
  • /etc/dansguardian/lists/bannedmimetypelist:
# Remove/comment out all mime-types from this file
  • /etc/dansguardian/lists/bannedphraselist:
# Add these
< callofduty >
< modernwarfare >
< call>,<of>,<duty >,< modern warfare>
< call>,<of>,<duty >,< world at war >
< call>,<of>,<duty >,< black ops>
  • /etc/dansguardian/lists/bannedregexpurllist:
# Added/uncommented these
(^|[\?+=&/])(.*\.google\..*/.*\?.*safe=(off|none))([\?+=&/]|$)
(^|[\?+=&/])(.*\.alltheweb.com/customize\?.*copt_offensive=off)([\?+=&/]|$)

# block autocomplete on google
(^.*\.google\.[a-z\.]+/complete/.*$)

# block bing explicit
(^.*explicit\.bing\.net\/.*$)

(meet|hook|mail........
(marr(y|i[ae])|........

# Call of duty
(call.?of.?duty|modern.?warfare)

# Violence etc
(suicid(e|al)|killing|violen(ce|t)|anorexi[ca]|pro\+ana|pro-ana|proana)
  • /etc/dansguardian/lists/bannedsitelist:
# Added these lines

### FROM urlblacklist.com ###

.Include</etc/dansguardian/lists/urlblacklist.com/abortion/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/ads/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/adult/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/aggressive/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/antispyware/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/artnudes/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/astrology/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/audio-video/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/banking/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/beerliquorinfo/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/beerliquorsale/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/blog/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/cellphones/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/chat/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/childcare/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/cleaning/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/clothing/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/contraception/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/culnary/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/dating/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/desktopsillies/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/dialers/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/drugs/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/ecommerce/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/entertainment/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/filehosting/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/frencheducation/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/gambling/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/games/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/gardening/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/government/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/guns/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/hacking/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/homerepair/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/hygiene/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/instantmessaging/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/jewelry/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/jobsearch/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/kidstimewasting/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/mail/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/marketingware/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/medical/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/mixed_adult/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/mobile-phone/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/naturism/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/news/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/onlineauctions/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/onlinegames/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/onlinepayment/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/personalfinance/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/pets/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/phishing/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/porn/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/proxy/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/radio/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/religion/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/ringtones/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/searchengines/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/sect/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/sexuality/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/shopping/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/socialnetworking/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/sportnews/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/sports/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/spyware/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/updatesites/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/vacation/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/violence/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/virusinfected/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/warez/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/weather/domains>
.Include</etc/dansguardian/lists/urlblacklist.com/weapons/domains>
#.Include</etc/dansguardian/lists/urlblacklist.com/webmail/domains>
####.Include</etc/dansguardian/lists/urlblacklist.com/whitelist/domains>

### FROM shallalist.de ###
# find shallalist.de/ -type f -name urls | perl -p -e 's|^(.*)$|.Include</etc/dansguardian/lists/$1>|g'

.Include</etc/dansguardian/lists/shallalist.de/spyware/domains>
.Include</etc/dansguardian/lists/shallalist.de/ringtones/domains>
#.Include</etc/dansguardian/lists/shallalist.de/imagehosting/domains>
#.Include</etc/dansguardian/lists/shallalist.de/hospitals/domains>
.Include</etc/dansguardian/lists/shallalist.de/anonvpn/domains>
#.Include</etc/dansguardian/lists/shallalist.de/jobsearch/domains>
.Include</etc/dansguardian/lists/shallalist.de/warez/domains>
.Include</etc/dansguardian/lists/shallalist.de/downloads/domains>
#.Include</etc/dansguardian/lists/shallalist.de/remotecontrol/domains>
#.Include</etc/dansguardian/lists/shallalist.de/homestyle/domains>
.Include</etc/dansguardian/lists/shallalist.de/hacking/domains>
.Include</etc/dansguardian/lists/shallalist.de/redirector/domains>
#.Include</etc/dansguardian/lists/shallalist.de/education/schools/domains>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/restaurants/domains>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/humor/domains>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/travel/domains>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/sports/domains>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/martialarts/domains>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/wellness/domains>
#.Include</etc/dansguardian/lists/shallalist.de/science/chemistry/domains>
#.Include</etc/dansguardian/lists/shallalist.de/science/astronomy/domains>
.Include</etc/dansguardian/lists/shallalist.de/tracker/domains>
#.Include</etc/dansguardian/lists/shallalist.de/library/domains>
#.Include</etc/dansguardian/lists/shallalist.de/forum/domains>
#.Include</etc/dansguardian/lists/shallalist.de/news/domains>
#.Include</etc/dansguardian/lists/shallalist.de/updatesites/domains>
#.Include</etc/dansguardian/lists/shallalist.de/webmail/domains>
.Include</etc/dansguardian/lists/shallalist.de/dating/domains>
#.Include</etc/dansguardian/lists/shallalist.de/military/domains>
#.Include</etc/dansguardian/lists/shallalist.de/models/domains>
.Include</etc/dansguardian/lists/shallalist.de/socialnet/domains>
.Include</etc/dansguardian/lists/shallalist.de/gamble/domains>
.Include</etc/dansguardian/lists/shallalist.de/violence/domains>
#.Include</etc/dansguardian/lists/shallalist.de/webradio/domains>
#.Include</etc/dansguardian/lists/shallalist.de/politics/domains>
#.Include</etc/dansguardian/lists/shallalist.de/podcasts/domains>
.Include</etc/dansguardian/lists/shallalist.de/costtraps/domains>
#.Include</etc/dansguardian/lists/shallalist.de/webtv/domains>
.Include</etc/dansguardian/lists/shallalist.de/dynamic/domains>
#.Include</etc/dansguardian/lists/shallalist.de/automobile/planes/domains>
#.Include</etc/dansguardian/lists/shallalist.de/automobile/cars/domains>
#.Include</etc/dansguardian/lists/shallalist.de/automobile/bikes/domains>
#.Include</etc/dansguardian/lists/shallalist.de/automobile/boats/domains>
.Include</etc/dansguardian/lists/shallalist.de/weapons/domains>
.Include</etc/dansguardian/lists/shallalist.de/searchengines/domains>
#.Include</etc/dansguardian/lists/shallalist.de/religion/domains>
#.Include</etc/dansguardian/lists/shallalist.de/government/domains>
#.Include</etc/dansguardian/lists/shallalist.de/webphone/domains>
#.Include</etc/dansguardian/lists/shallalist.de/fortunetelling/domains>
.Include</etc/dansguardian/lists/shallalist.de/adv/domains>
.Include</etc/dansguardian/lists/shallalist.de/alcohol/domains>
.Include</etc/dansguardian/lists/shallalist.de/chat/domains>
#.Include</etc/dansguardian/lists/shallalist.de/movies/domains>
.Include</etc/dansguardian/lists/shallalist.de/drugs/domains>
#.Include</etc/dansguardian/lists/shallalist.de/radiotv/domains>
#.Include</etc/dansguardian/lists/shallalist.de/domainshortener/domains>
.Include</etc/dansguardian/lists/shallalist.de/porn/domains>
.Include</etc/dansguardian/lists/shallalist.de/sex/education/domains>
.Include</etc/dansguardian/lists/shallalist.de/sex/lingerie/domains>
.Include</etc/dansguardian/lists/shallalist.de/hobby/games-online/domains>
#.Include</etc/dansguardian/lists/shallalist.de/hobby/gardening/domains>
.Include</etc/dansguardian/lists/shallalist.de/hobby/games-misc/domains>
#.Include</etc/dansguardian/lists/shallalist.de/hobby/pets/domains>
#.Include</etc/dansguardian/lists/shallalist.de/hobby/cooking/domains>
#.Include</etc/dansguardian/lists/shallalist.de/music/domains>
#.Include</etc/dansguardian/lists/shallalist.de/finance/banking/domains>
#.Include</etc/dansguardian/lists/shallalist.de/finance/other/domains>
#.Include</etc/dansguardian/lists/shallalist.de/finance/insurance/domains>
#.Include</etc/dansguardian/lists/shallalist.de/finance/realestate/domains>
#.Include</etc/dansguardian/lists/shallalist.de/finance/trading/domains>
#.Include</etc/dansguardian/lists/shallalist.de/finance/moneylending/domains>
#.Include</etc/dansguardian/lists/shallalist.de/shopping/domains>
#.Include</etc/dansguardian/lists/shallalist.de/isp/domains>
.Include</etc/dansguardian/lists/shallalist.de/aggressive/domains>
  • /etc/dansguardian/lists/bannedurllist:
# Added these lines

forums.moshimonsters.com

### FROM malware.com.br ###

.Include</etc/dansguardian/lists/malware.com.br/malware.txt>

### FROM urlblacklist.com ###

.Include</etc/dansguardian/lists/urlblacklist.com/abortion/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/ads/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/adult/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/aggressive/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/antispyware/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/artnudes/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/astrology/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/audio-video/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/banking/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/blog/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/cellphones/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/chat/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/childcare/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/cleaning/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/clothing/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/contraception/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/culnary/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/dating/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/desktopsillies/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/dialers/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/drugs/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/ecommerce/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/entertainment/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/filehosting/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/frencheducation/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/gambling/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/games/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/gardening/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/government/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/guns/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/hacking/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/homerepair/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/hygiene/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/instantmessaging/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/jewelry/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/jobsearch/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/kidstimewasting/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/mail/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/marketingware/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/medical/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/mixed_adult/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/mobile-phone/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/naturism/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/news/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/onlineauctions/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/onlinegames/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/onlinepayment/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/personalfinance/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/pets/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/phishing/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/porn/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/proxy/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/radio/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/religion/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/searchengines/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/sect/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/sexuality/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/shopping/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/socialnetworking/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/sportnews/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/sports/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/spyware/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/updatesites/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/vacation/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/violence/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/virusinfected/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/warez/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/weather/urls>
.Include</etc/dansguardian/lists/urlblacklist.com/weapons/urls>
#.Include</etc/dansguardian/lists/urlblacklist.com/webmail/urls>
####.Include</etc/dansguardian/lists/urlblacklist.com/whitelist/urls>


### FROM shallalist.de ###
# find shallalist.de/ -type f -name domains | perl -p -e 's|^(.*)$|.Include</etc/dansguardian/lists/$1>|g'

.Include</etc/dansguardian/lists/shallalist.de/spyware/urls>
.Include</etc/dansguardian/lists/shallalist.de/ringtones/urls>
#.Include</etc/dansguardian/lists/shallalist.de/imagehosting/urls>
#.Include</etc/dansguardian/lists/shallalist.de/hospitals/urls>
.Include</etc/dansguardian/lists/shallalist.de/anonvpn/urls>
#.Include</etc/dansguardian/lists/shallalist.de/jobsearch/urls>
.Include</etc/dansguardian/lists/shallalist.de/warez/urls>
.Include</etc/dansguardian/lists/shallalist.de/downloads/urls>
#.Include</etc/dansguardian/lists/shallalist.de/remotecontrol/urls>
#.Include</etc/dansguardian/lists/shallalist.de/homestyle/urls>
.Include</etc/dansguardian/lists/shallalist.de/hacking/urls>
.Include</etc/dansguardian/lists/shallalist.de/redirector/urls>
#.Include</etc/dansguardian/lists/shallalist.de/education/schools/urls>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/restaurants/urls>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/humor/urls>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/travel/urls>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/sports/urls>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/martialarts/urls>
#.Include</etc/dansguardian/lists/shallalist.de/recreation/wellness/urls>
#.Include</etc/dansguardian/lists/shallalist.de/science/chemistry/urls>
#.Include</etc/dansguardian/lists/shallalist.de/science/astronomy/urls>
.Include</etc/dansguardian/lists/shallalist.de/tracker/urls>
#.Include</etc/dansguardian/lists/shallalist.de/library/urls>
#.Include</etc/dansguardian/lists/shallalist.de/forum/urls>
#.Include</etc/dansguardian/lists/shallalist.de/news/urls>
#.Include</etc/dansguardian/lists/shallalist.de/updatesites/urls>
#.Include</etc/dansguardian/lists/shallalist.de/webmail/urls>
.Include</etc/dansguardian/lists/shallalist.de/dating/urls>
#.Include</etc/dansguardian/lists/shallalist.de/military/urls>
#.Include</etc/dansguardian/lists/shallalist.de/models/urls>
.Include</etc/dansguardian/lists/shallalist.de/socialnet/urls>
.Include</etc/dansguardian/lists/shallalist.de/gamble/urls>
.Include</etc/dansguardian/lists/shallalist.de/violence/urls>
#.Include</etc/dansguardian/lists/shallalist.de/webradio/urls>
#.Include</etc/dansguardian/lists/shallalist.de/politics/urls>
#.Include</etc/dansguardian/lists/shallalist.de/podcasts/urls>
.Include</etc/dansguardian/lists/shallalist.de/costtraps/urls>
#.Include</etc/dansguardian/lists/shallalist.de/webtv/urls>
.Include</etc/dansguardian/lists/shallalist.de/dynamic/urls>
#.Include</etc/dansguardian/lists/shallalist.de/automobile/planes/urls>
#.Include</etc/dansguardian/lists/shallalist.de/automobile/cars/urls>
#.Include</etc/dansguardian/lists/shallalist.de/automobile/bikes/urls>
#.Include</etc/dansguardian/lists/shallalist.de/automobile/boats/urls>
.Include</etc/dansguardian/lists/shallalist.de/weapons/urls>
#.Include</etc/dansguardian/lists/shallalist.de/searchengines/urls>
#.Include</etc/dansguardian/lists/shallalist.de/religion/urls>
#.Include</etc/dansguardian/lists/shallalist.de/government/urls>
#.Include</etc/dansguardian/lists/shallalist.de/webphone/urls>
#.Include</etc/dansguardian/lists/shallalist.de/fortunetelling/urls>
.Include</etc/dansguardian/lists/shallalist.de/adv/urls>
.Include</etc/dansguardian/lists/shallalist.de/alcohol/urls>
.Include</etc/dansguardian/lists/shallalist.de/chat/urls>
#.Include</etc/dansguardian/lists/shallalist.de/movies/urls>
.Include</etc/dansguardian/lists/shallalist.de/drugs/urls>
#.Include</etc/dansguardian/lists/shallalist.de/radiotv/urls>
#.Include</etc/dansguardian/lists/shallalist.de/urlshortener/urls>
.Include</etc/dansguardian/lists/shallalist.de/porn/urls>
.Include</etc/dansguardian/lists/shallalist.de/sex/education/urls>
.Include</etc/dansguardian/lists/shallalist.de/sex/lingerie/urls>
.Include</etc/dansguardian/lists/shallalist.de/hobby/games-online/urls>
#.Include</etc/dansguardian/lists/shallalist.de/hobby/gardening/urls>
.Include</etc/dansguardian/lists/shallalist.de/hobby/games-misc/urls>
#.Include</etc/dansguardian/lists/shallalist.de/hobby/pets/urls>
#.Include</etc/dansguardian/lists/shallalist.de/hobby/cooking/urls>
#.Include</etc/dansguardian/lists/shallalist.de/music/urls>
#.Include</etc/dansguardian/lists/shallalist.de/finance/banking/urls>
#.Include</etc/dansguardian/lists/shallalist.de/finance/other/urls>
#.Include</etc/dansguardian/lists/shallalist.de/finance/insurance/urls>
#.Include</etc/dansguardian/lists/shallalist.de/finance/realestate/urls>
#.Include</etc/dansguardian/lists/shallalist.de/finance/trading/urls>
#.Include</etc/dansguardian/lists/shallalist.de/finance/moneylending/urls>
#.Include</etc/dansguardian/lists/shallalist.de/shopping/urls>
#.Include</etc/dansguardian/lists/shallalist.de/isp/urls>
.Include</etc/dansguardian/lists/shallalist.de/aggressive/urls>
  • /etc/dansguardian/lists/exceptionsitelist:
# Added these lines

# SSL certificates
verisign.com

# safe-ish sites
twitter.com
twimg.com
ebay.com
ebay.co.uk
paypal.com
paypal.co.uk
linkedin.com
macromedia.com

# search sites (still want to ban certain URLs in these doamins so don't whitelist them)
# removed google.com/co.uk, bing.com/co.uk, yahoo.com from ./shallalist.de/searchengines/domains

# google sites
chatenabled.mail.google.com

# some educational safe sites
scratch.mit.edu
primarygames.com
bbc.co.uk
rm.com
myschoolwebsite.uk
picturetheuk.com
discoverybox.e2bn.org
cooltext.com
www.photoshop.com
photopin.com
wordle.net
linuxcentre.net
goanimate4schools.com
ocnmail.net
theocn.net
gov.uk
.gov
sch.uk

# hotmail login
live.com
hotmail.com

# amazon images
images-amazon.com

# used by itunes
phobos.apple.com

# cloud storage
dropbox.com
## Removed s3.amazonaws.com from /etc/dansguardian/lists/shallalist.de/downloads/domains
sourceforge.net

# miscategorised banking
mastercard.com
  • /etc/dansguardian/lists/pics:
# Changed these lines

ICRAnudityartistic = 0
ICRAnudityeducational = 0
ICRAnuditymedical = 0
RSACviolence = 0
RSACsex = 0
RSACnudity = 0
RSAClanguage = 0
evaluWEBrating = 0
CyberNOTsex = 0
CyberNOTother = 0
SafeSurfprofanity = 1
SafeSurfheterosexualthemes = 0
SafeSurfhomosexualthemes = 0
SafeSurfnudity = 0
SafeSurfviolence = 0
SafeSurfsexviolenceandprofanity = 0
SafeSurfintolerance = 0
SafeSurfdruguse = 0
SafeSurfotheradultthemes = 0
SafeSurfgambling = 0
SafeSurfagerange = 1
  • /etc/dansguardian/lists/urlregexplist:
# uncomment all predefined regex lines 
# also see below for search engine regex lines
  • /etc/dansguardian/lists/weightedphraselist:
# Added these lines
<suicid><100000>
< anorexia ><10000>
< anorex>,< pro>,<ana ><10000>
< pro-ana ><10000>
< pro ana ><10000>
< proana ><10000>
#.Include</etc/dansguardian/lists/phraselists/pornography/weighted_japanese> #ALPHA#
.Include</etc/dansguardian/lists/phraselists/drugadvocacy/weighted>
.Include</etc/dansguardian/lists/phraselists/illegaldrugs/weighted>
.Include</etc/dansguardian/lists/phraselists/illegaldrugs/weighted_portuguese>
.Include</etc/dansguardian/lists/phraselists/gore/weighted>
.Include</etc/dansguardian/lists/phraselists/gore/weighted_portuguese>
.Include</etc/dansguardian/lists/phraselists/violence/weighted>
.Include</etc/dansguardian/lists/phraselists/violence/weighted_portuguese>
.Include</etc/dansguardian/lists/phraselists/weapons/weighted>
.Include</etc/dansguardian/lists/phraselists/weapons/weighted_portuguese>
#.Include</etc/dansguardian/lists/phraselists/proxies/weighted>

Squid Configuration

The squid config is as follows:

http_port 8080

hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
maximum_object_size 32768 KB
# emulate_httpd_log off
# debug_options ALL,1
# log_fqdn off
negative_ttl 1 minute
positive_dns_ttl 1 hour
negative_dns_ttl 1 minute

### Send some requests to the OCN proxy server (due to either high upload requirements or intranet access)
cache_peer t9999.theocn.net parent 8080 0 no-query default
cache_peer_domain t9999.theocn.net .oxfordshire.gov.uk .ocnmail.net .theocn.net .s3.amazonaws.com
# always use parent for these domains even if non-cachable
nonhierarchical_direct off

### Only Allow These URLs or SSL domains ###
acl allowed_urls url_regex -i ^http://.*
acl blocked_urls url_regex -i ^http://.*wordle.net/(next|random|gallery).*$

acl whitelist_ssl_domains url_regex -i ^(.*accounts.google|www\.google\.com|www\.google\.co\.uk|mail\.google\.com|.*accounts\.youtube\.com).*$
acl blocked_ssl_domains url_regex -i ^(.*www.google\.|.*bing\.com|.*youtube|encrypted\.google\.).*$
acl allowed_ssl_domains dstdomain .

acl internal src 127.0.0.1/255.255.255.255
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl SSL_ports port 443
acl SSL_ports port 888
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 631         # https
acl Safe_ports port 888         # me
acl Safe_ports port 2812        # monit
acl Safe_ports port 8080        # http
acl CONNECT method CONNECT

# Logging
access_log /var/log/squid/access.log common
strip_query_terms off
log_uses_indirect_client on
follow_x_forwarded_for allow internal

# Only allow cachemgr access from localhost
http_access allow manager localhost
http_access deny manager

# Deny requests to unknown ports
http_access deny !Safe_ports

# Deny CONNECT to other than SSL ports
http_access deny CONNECT !SSL_ports

http_access allow whitelist_ssl_domains CONNECT
http_access deny blocked_ssl_domains CONNECT
http_access allow allowed_ssl_domains CONNECT

# Only allow access to defined URLs
http_access deny blocked_urls
http_access allow all
http_access deny !internal
http_access allow allowed_urls
http_access deny all


Search Related Filtering Setup

Enforcing Safesearch on Google and Bing

  • We setup additional URL rewriting rules on dansguardian as follows in file 'urlregexplist':
# Google - remove 'safe=...'
"(^http://[0-9a-z]+\.google\.[a-z]+[-/%.0-9a-z]*/(search|s|images)\?)(.*)(&?)(safe=[^&]*)"->"\1\3\4"
# ... and add 'safe=vss'
"(^http://[0-9a-z]+\.google\.[a-z]+[-/%.0-9a-z]*/(search|s|images)\?)"->"\1safe=vss&"

# BING Add &adlt=strict
"(^http://.+\.bing\.[a-z]+[-/%.0-9a-z]*/.*$)"->"\1&adlt=strict"
  • We additionally blocked unsafe access by adding these to the dansguardian file 'bannedregexpurllist':
#Block unfiltered options on various search engines
(^|[\?+=&/])(.*\.google\..*/.*\?.*safe=(off|none))([\?+=&/]|$)
(^.*explicit\.bing\.net\/.*$)

The Google SSL Search problem and Google Apps for Education

  • We use Google Apps for Education so it is not feasible to block https access to google's servers.
  • Since 2011, google automatically redirect logged-in users to the https search site and this defeats the filters.
  • Google have provided (a rather hard to implement) solution here.
  • We have implemented this using the nosslsearch.google.com solution for both www.google.co.uk and www.google.com on our BIND nameserver as follows:
    • Create a new zone file called '/var/named/www.google.com' and add the following contents ref:
@       IN      SOA     localhost.        root.localhost. (
                                        2012062000
                                        7200
                                        1800
                                        1209600
                                        300 )
        1800        IN        NS        localhost.
        1800        IN        A        216.239.32.20 ;nosslsearch.google.com.
    • Add this section to the file '/etc/named.conf':
zone "www.google.com." {
        type master;
        file "www.google.com";
};
zone "www.google.co.uk." {
        type master;
        file "www.google.com";
};
    • This was added to the squid.conf file to selectively block SSL on youtube and google international sites:
# whitelist these SSL domains for google apps and for the nosslsearch DNS feature
acl whitelist_ssl_domains url_regex -i ^(.*accounts.google|www\.google\.com|www\.google\.co\.uk|mail\.google\.com|.*accounts\.youtube\.com).*$

# explicitly block these on SSL only - catches all other SSL searches on other google international sites and youtube.
acl blocked_ssl_domains url_regex -i ^(.*www.google\.|.*bing\.com|.*youtube|encrypted\.google\.).*$

# allow all other SSL domains and leave the filtering to dansguardian
acl allowed_ssl_domains dstdomain .
...
http_access allow whitelist_ssl_domains CONNECT
http_access deny blocked_ssl_domains CONNECT   
http_access allow allowed_ssl_domains CONNECT  
...

Youtube Filtering

  • Google provides no obvious means to filter youtube videos.
  • There is a way to filter using a special parameter as long as you first setup a youtube account linked to a Google Apps account. See here for instructions and details.
  • We then setup URL rewriting rule on dansguardian as follows in file 'urlregexplist' (replace '123456789asdfghjkl' with your unique edufilter key):
# Youtube (use our unique Youtube for schools code)
# remove any existing edufilter param
"(^http://[^/]*\.youtube\.[^?]*\?)(.+&)?(edufilter=[^&]*)"->"\1\2"
# add in our edufilter param
"(^http://[^/]*\.youtube\.[^?]*\?)"->"\1edufilter=123456789asdfghjkl&"
# or if no params add ours in
"(^http://[^/]*\.youtube\.[^?]*$)"->"\1/?edufilter=123456789asdfghjkl"
  • Remember to block all SSL access to youtube (see above section)
Personal tools