Squid Caching Configuration

Caching Configuration

Squid proxy server supports two types of caching – RAM caching and hardware caching. While configuring you should keep in mind that caching may either accelerate query processing or slow it down, in case configuration parameters are wrong. Another important point is that caching results in additional load on server, e.g. excessive cache in RAM is able to paralyze sever causing lack of RAM.

Squid standard configuration provides RAM cache only with memory usage of 256 Mbyte. You may increase cache volume and set maximum volume of a cached object using the following directives:

cache_mem 1024 MB #volume of the memory available for casing
maximum_object_size_in_memory 512 KB #maximum volume of a cached object 

Also, one should take into account that cache is reset every time sever restarts or turns off, so the results of configuration change will be perceptible only over time. 

HDD cache usage is controlled by cache_dirdirective: 

cache_dir storage_type path_to_storage volume L1 L2

cache_dir ufs /var/squid_cache 1024 16 256 

Cache volume on hardware is counted in Mbytes. In the example below cache with maximum volume of 1 Gbyte is stored in /var/squid_cache. ufs storage type is standard. Parameters 16 and 256 denote the number of first and second level directories, these volumes are standard, too. 

You may also specify the cached object maximum volume:

maximum_object_size 2 MB

Credits

Install and Configure Squid Proxy Server, ClamAV, SquidClamav, C-ICAP Server – Debian 9 (minimal – standard system utilities, ssh server)

1. Install some needed dependencies.
$sudo apt-get install gcc make curl libcurl4-gnutls-dev rsync

2. Install and Configure Squid Proxy Server.
$sudo apt-get install squid3 calamaris

3.Edit config file /etc/squid/squid.conf:
$sudo cp squid.conf squid.conf.ORIG

Backup
$sudo cp /etc/squid/squid.conf /etc/squid/squid.conf.bak

To simplify the configuration file (squid.conf), we can remove everything that is comments or blank lines.
$sudo cat squid.conf.bak | egrep -v -e '^[[:blank:]]*#|^$' > squid.conf

$sudo nano /etc/squid/squid.conf

3.1 Change squid.conf options

Make sure the line is uncommented (#).
acl CONNECT method CONNECT

Create new access lists acl LAN to your internal network 192.168.0.0/24. or others internal network
acl LAN src 192.168.0.0/24
acl LAN src xxx.xxx.x.x/24

Additional access lists blacklist, whitelist, malware_block_list to block spam, commercials, malware, viruses…

acl malware_block_list url_regex -i "/etc/squid/malware_block_list"
acl blacklist dstdom_regex "/etc/squid/blacklist"
acl whitelist dstdom_regex "/etc/squid/whitelist"

Access new acl lists – order matters:

http_access allow whitelist
http_access deny blacklist
http_access deny malware_block_list
http_access allow LAN

I did not use: Inform users about blocked website. Blocked commercials will be displayed as empty transparent place, require http server.
deny_info http://YourServerName/error/dot-transparent.png blacklist
deny_info http://YourServerName/error/dot-transparent.png whitelist
deny_info http://YourServerName/error/error.html malware_block_list

Setup address IP and listening port. Transparent mean no caching.
http_port 192.168.0.1:3128

Additional  setup – Anonymizer. Blocking headers:
request_header_access Allow allow all
request_header_access Authorization allow all
request_header_access WWW-Authenticate allow all
request_header_access Proxy-Authorization allow all
request_header_access Proxy-Authenticate allow all
request_header_access Content-Encoding allow all
request_header_access Content-Length allow all
request_header_access Content-Type allow all
request_header_access Date allow all
request_header_access Expires allow all
request_header_access Host allow all
request_header_access If-Modified-Since allow all
request_header_access Last-Modified allow all
request_header_access Location allow all
request_header_access Pragma allow all
request_header_access Accept allow all
request_header_access Accept-Charset allow all
request_header_access Accept-Encoding allow all
request_header_access Accept-Language allow all
request_header_access Content-Language allow all
request_header_access Mime-Version allow all
request_header_access Retry-After allow all
request_header_access Title allow all
request_header_access Connection allow all
request_header_access Proxy-Connection allow all
request_header_access User-Agent allow all
request_header_access Cookie allow all
request_header_access Referer deny all
request_header_access X-Forwarded-For deny all
request_header_access Via deny all
request_header_access All deny all
request_header_access Cache-Control deny all
httpd_suppress_version_string on

Store cache ojects only in memory, cache (400MB)

## Cache options

cache_mem 512 MB
cache_dir ufs /var/spool/squid3 400 16 256
  ### cache_dir ufs /usr/local/squid/cache 51200 64 256

Disable cache for access list – LAN:
cache deny LAN

Hostname
visible_hostname YourServerName

Hiding IP
forwarded_for off

##———————- My test config ——————————————
acl LAN src 192.168.0.0/24
acl LAN src 192.168.100.0/24
acl LAN src 192.168.122.0/24

acl SSL_ports port 443
acl Safe_ports port 80        # http
acl Safe_ports port 21        # ftp
acl Safe_ports port 443        # https
acl Safe_ports port 70        # gopher
acl Safe_ports port 210        # wais
acl Safe_ports port 1025-65535    # unregistered ports
acl Safe_ports port 280        # http-mgmt
acl Safe_ports port 488        # gss-http
acl Safe_ports port 591        # filemaker
acl Safe_ports port 777        # multiling http
acl CONNECT method CONNECT

acl malware_block_list url_regex -i "/etc/squid/malware_block_list"
acl blacklist dstdom_regex "/etc/squid/blacklist"
acl whitelist dstdom_regex "/etc/squid/whitelist"
http_access allow whitelist
http_access deny blacklist
http_access deny malware_block_list

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow LAN
http_access allow localhost
http_access deny all

visible_hostname proxy
http_port 3128
coredump_dir /var/spool/squid

refresh_pattern ^ftp:        1440    20%    10080
refresh_pattern ^gopher:    1440    0%    1440
refresh_pattern -i (/cgi-bin/|\?) 0    0%    0
refresh_pattern .        0    20%    4320

request_header_access Allow allow all
request_header_access Authorization allow all
request_header_access WWW-Authenticate allow all
request_header_access Proxy-Authorization allow all
request_header_access Proxy-Authenticate allow all
request_header_access Content-Encoding allow all
request_header_access Content-Length allow all
request_header_access Content-Type allow all
request_header_access Date allow all
request_header_access Expires allow all
request_header_access Host allow all
request_header_access If-Modified-Since allow all
request_header_access Last-Modified allow all
request_header_access Location allow all
request_header_access Pragma allow all
request_header_access Accept allow all
request_header_access Accept-Charset allow all
request_header_access Accept-Encoding allow all
request_header_access Accept-Language allow all
request_header_access Content-Language allow all
request_header_access Mime-Version allow all
request_header_access Retry-After allow all
request_header_access Title allow all
request_header_access Connection allow all
request_header_access Proxy-Connection allow all
request_header_access User-Agent allow all
request_header_access Cookie allow all
request_header_access Referer deny all
request_header_access X-Forwarded-For deny all
request_header_access Via deny all
request_header_access All deny all
request_header_access Cache-Control deny all
httpd_suppress_version_string on

 

## Cache options

cache_mem 512 MB
cache_dir ufs /var/spool/squid 400 16 256
cache deny LAN
visible_hostname proxy
forwarded_for off

##———————————————–——–——–——–

3.2 Download files: blacklist and whitelist, unpack and save it to /etc/squid/.
$wget -c https://ffteixeira.net/blog/sites/default/files/blacklist.tar_.bz2 (rename blacklist.tar_.bz2 to blacklist.tar.bz2) || wget -c http://terminal28.com/wp-content/uploads/2013/10/blacklist.tar.bz2
$sudo tar -xvf blacklist.tar.bz2
$sudo mv blacklist whitelist /etc/squid

Before trying to start Squid, you should verify that your squid.conf file makes sense. This is easy to do. Just run the following command:
$sudo squid -k parse

Ignore this error, solved next step:
…/08/11 12:23:16| Processing: acl malware_block_list url_regex -i "/etc/squid/malware_block_list"
…/08/11 12:23:16| ERROR: Can not open file /etc/squid/malware_block_list for reading
…/08/11 12:23:16| Warning: empty ACL: acl malware_block_list url_regex -i "/etc/squid/malware_block_list"
…/08/11 12:23:16| Processing: acl blacklist dstdom_regex "/etc/squid/blacklist"
…/08/11 12:23:17| /etc/squid/squid.conf line 20: acl blacklist dstdom_regex "/etc/squid/blacklist"
…/08/11 12:23:17| WARNING: there are more than 100 regular expressions. Consider using less REs or use rules without expressions like 'dstdomain'.

Restart Squid.
$sudo /etc/init.d/squid restart

3.3 Download script malware_block_list to update domains and IP addresses , unpack and save it to /etc/squid
$wget -c https://ffteixeira.net/blog/sites/default/files/malware_block_list.tar_.bz2 (rename malware_block_list.tar_.bz2  to malware_block_list.tar.bz2) || wget -c http://terminal28.com/wp-content/uploads/2013/10/malware_block_list.tar.bz2
$sudo tar -xvf malware_block_list.tar.bz2
$sudo mv malware_block_list /usr/local/bin/
$sudo chmod +x /usr/local/bin/malware_block_list
$sudo touch  /var/log/malware_block_list.log

Add script malware_block_list to Cron.
$sudo crontab -e

add
    
@daily /usr/local/bin/malware_block_list

Logfile location: /var/log/malware_block_list.log.   Go to  MalwarePatrol.net, click tab: Block List. You should see subscription list: free and paid. Click Free/Subscribe. Subscribe the list. You should get password/receipt number on email. Log in to: https://www.malwarepatrol.net/login.php; and find Squid Web Proxy ACL and click Download. You will be redirected to website/text with malware list. Every subscription has unique receipt number receipt=f1234567890. https://lists.malwarepatrol.net/cgi/getfile?receipt=f1234567890&product=8&list=squid Copy URL and paste to script near link. Edit: link, user, pass.

$sudo nano /usr/local/bin/malware_block_list

link='PASTE_LINK_FROM_MALWAREPATROL.NET'
user='–http-user=USERNAME'
passwd='–http-passwd=PASSWORD'

Note: Change squid3 to squid

##————————————- My test config. —————————
#!/bin/sh
### ###
###
### Squid3 Installation and Configuration.
###
### Polish version
###
### http://man.sethuper.com/instalacja-squid-proxy-serwer-clamav-squidclamav-c-icap-serwer-debian-6-0-x
###
#=======================================================================================================================
###
### English version
###
### http://terminal28.com/how-to-install-and-configure-squid-proxy-server-clamav-squidclamav-c-icap-server-debian-linux/
###
### ###

# If you don't want to log wget debug output remove "$debug" in line (51) "fetchcmd"

## Setings
# Malware patrol URL with unique ID
# Change ID after receipt in link (..getfile?receipt=f138125701..)
link='https://lists.malwarepatrol.net/cgi/getfile?receipt=f1502379316&product=8&list=squid'

# HTTP USER
user='–http-user=<user>'

# HTTP PASSWORD
pass='–http-passwd=<passwd>'

# Checking certificate
cert='–no-check-certificate'

# File location for Squid
target='/etc/squid/malware_block_list'

# Reload Squid
reloadcmd='/usr/sbin/squid -k reconfigure'

# Temporary file
tmp="/tmp/.malware_block_list.$$"

# Wget debud
#debug="-nva /var/log/squid/malware_block_list.log"

# Command for download malware list
#I remove de debug because error
#fetchcmd="wget -q  –no-check-certificate  $link -O $tmp $user $pass $debug"
fetchcmd="wget -q  –no-check-certificate  $link -O $tmp $user $pass"

# ——-

# Log file
logs='/var/log/squid/malware_block_list.log'

## execution
##
echo "$(date -R) Downloading new malware_block_list" >> "$logs"

# Downloading new malware_block_list from Malware Patrol
$fetchcmd

# Checking temporary file – "OK" – before overwrite old malware list
if [ ! -s $tmp ]
then
echo "$(date -R) The temporary file '$tmp' does not exist or is empty; resignation" >> "$logs"
exit
fi

# moving malware_black_list to directory /etc/squid3/
cp  $tmp $target

# removing temporary file
rm $tmp

# restart Squid
$reloadcmd
##———————————————————————————

 
$sudo sh /usr/local/bin/malware_block_list

4. Install Clamav-server.
$sudo apt-get install clamav-daemon
$sudo mkdir install
$cd install
$sudo wget https://sourceforge.net/projects/c-icap/files/c-icap/0.5.x/c_icap-0.5.2.tar.gz/download -O c_icap-0.5.2.tar.gz
$sudo tar -xvf c_icap-0.5.2.tar.gz
$cd c_icap-0.5.2
$sudo ./configure
$sudo make
$sudo make install
$cd ..

Edit configfile  /usr/local/etc/c-icap.conf.
$sudo nano /usr/local/etc/c-icap.conf

Change:

Line 223: ServerAdmin root@localhost
Line 232: ServerName YourServerName

Add at line 708:

Service squidclamav squidclamav.so

4.1 C-ICAP server autostart script.
$wget -c  https://ffteixeira.net/blog/sites/default/files/c-icap-autostart.tar_.gz (rename c-icap-autostart.tar_.gz  to c-icap-autostart.tar.gz) || wget -c http://terminal28.com/wp-content/uploads/2013/10/c-icap-autostart.tar.gz
$sudo tar xvf c-icap-autostart.tar.gz
$sudo rsync -avh init.d default /etc
$sudo update-rc.d c-icap defaults

4.2 Create logrotate script for c-icap server.
$sudo cat << EOT > /etc/logrotate.d/c-icap

/usr/local/var/log/server.log /usr/local/var/log/access.log {
     daily
     rotate 4
     missingok
     notifempty
     compress
     create 0644 root root
     postrotate
     /etc/init.d/c-icap force-reload > /dev/null
     endscript
}
EOT

4.3 Change permission for c-icap logrotate script and server logs.
$sudo chmod 644 /etc/logrotate.d/c-icap
$sudo chown root:root /etc/logrotate.d/c-icap
$sudo chmod 644 /usr/local/var/log/ -R
$sudo chown root:root /usr/local/var/log/ -R
$sudo ln -s /usr/local/var/log/server.log /var/log/server.log
$sudo ln -s /usr/local/var/log/access.log /var/log/access.log

5. Install Squidclamav
$cd install
$wget -c https://sourceforge.net/projects/squidclamav/files/squidclamav/6.15/squidclamav-6.15.tar.gz/download -O squidclamav-6.15.tar.gz
$sudo tar zxvf squidclamav-6.15.tar.gz
$cd squidclamav-6.15
$sudo ./configure
$sudo make
$sudo make install
$cp -rf cgi-bin /usr/lib/
$chmod +x /usr/lib/cgi-bin/clwarn* -R
$chown www-data:www-data /usr/lib/cgi-bin/clwarn* -R
$cd ..
$sudo ldconfig

5.1 Configure squidclamav.
$sudo nano /usr/lcocal/etc/squidclamav.conf

Add redirect URL – default script – clwarn.cgi (en). You can choose diferent language: DE, FR, BR, RU.

Line 18: redirect http://YourServerName/cgi-bin/clwarn.cgi

Make sure the rule occurs in configfile.

Line 27: clamd_local /var/run/clamav/clamd.ctl

6. Checking config file – ClamAV, make sure the rule occurs in configfile.
$sudo nano /etc/clamav/clamd.conf

Line 4: LocalSocket /var/run/clamav/clamd.ctl

Configure Freshclam.
$sudo nano /etc/clamav/freshclam.conf

Line 22: SafeBrowsing true

6.1 Register on Securiteinfo.com: https://www.securiteinfo.com/clients/customers/signup
Subscribe basic list for clamav. You should get auto generated urls for clamav database under tab: Setup.
Download allowed from 1 IP address, limited to 24 downloads per day
Add generated URLS to freshclam.conf file at the end.

DatabaseCustomURL http://www.securiteinfo.com/get/signatures/3b4d0…5764/securiteinfo.hdb
DatabaseCustomURL http://www.securiteinfo.com/get/signatures/3b4b…eafd/securiteinfo.ign2
DatabaseCustomURL http://www.securiteinfo.com/get/signatures/3b4d0d…61eafd/javascript.ndb
DatabaseCustomURL http://www.securiteinfo.com/get/signatures/34d…81f/spam_marketing.ndb
DatabaseCustomURL http://www.securiteinfo.com/get/signatures/3b…61eafd/securiteinfohtml.hdb
DatabaseCustomURL http://www.securiteinfo.com/get/signatures/3b…365afd/securiteinfoascii.hdb

Restart ClamAV.
$sudo /etc/init.d/clamav-daemon restart

7. Configure Squid with C-ICAP. Configuration for Squid version – 3.1.20.
$sudo nano /etc/squid/squid.conf

Add at the end of the file

icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_header X-Authenticated-User
icap_service service_req reqmod_precache bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access service_req allow all
icap_service service_resp respmod_precache bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access service_resp allow all

Configuration for Squid version – 3.1.6.
$sudo nano /etc/squid/squid.conf

Add at the end of the file

icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Client-Username
icap_preview_enable on
icap_preview_size 1024
adaptation_service_set service_req
icap_service service_req reqmod_precache bypass=1 icap://127.0.0.1:1344/request
adaptation_access service_req allow all

adaptation_service_set service_resp
icap_service service_resp respmod_precache bypass=0 icap://127.0.0.1:1344/response
adaptation_access service_resp allow all

Run C-ICAP server.
$sudo /usr/local/bin/c-icap &

8. Restart Squid.
$sudo chown -R proxy:proxy /var/spool/squid
$sudo squid -z
$sudo service squid restart

9. Configure firewall – masquerade, prerouting.
Enable forwarding. Edit configfile sysctl.conf
$sudo nano /etc/sysctl.conf

Uncomment IPv4 i IPv6 and change to 1:

Line 28: net.ipv4.ip_forward = 1
Line 33: net.ipv6.conf.all.forwarding = 1

##——————- Not used ———————
9.1. Configure firewall – iptables.

$sudo nano /etc/iptables.up.rules

Add rules (Change address IP and network interface)

*nat

-A PREROUTING -p tcp -m tcp -i eth1 –dport 80 -j REDIRECT –to-ports 3128
-A POSTROUTING -s 192.168.0.0/24 -j MASQUERADE

 ##—————————————————–

10. Test.

If you have done it right then..
.. go to: http://www.eicar.org/85-0-Download.html and try to download file:

eicar.com
68 Bytes

Result:
You should be redirected to:

    http://YourServerName/cgi-bin/clwarn.cgi, http://YourServerName/error.html.

11. Sarg and squidguard

Credits

install SquidGuard! at debian 9

Install SquidGuard!
1. sudo apt-get install squidguard
2. sudo mkdir /opt/3rdparty
3. cd /opt/3rdparty    
We are going to use the list from shalalist.de for “testing”, since it’s 100% free for non-commerical.  For a bigger and much more through blacklist, I use http://urlblacklist.com/. It’s free to try once, and has different pricing tiers for person/school/business.
3. sudo wget http://www.shallalist.de/Downloads/shallalist.tar.gz
4. sudo tar xzf shallalist.tar.gz
5. sudo cp -a /opt/3rdparty/BL/porn /var/lib/squidguard/db
    sudo cp -a /opt/3rdparty/BL/adv /var/lib/squidguard/db
    sudo cp -a /opt/3rdparty/BL/spyware /var/lib/squidguard/db

6. Add this to  /etc/squid3/squid.conf , type “sudo nano /etc/squid3/squid.conf”
url_rewrite_program /usr/bin/squidGuard –c /etc/squidguard/squidGuard.conf
7. sudo squidGuard -C all
8. chown -R proxy:proxy /var/lib/squidguard/db

Edit the squidGuard.conf

Backup of your squidGuard.conf then making a new one..
1. sudo cp /etc/squidguard/squidGuard.conf /etc/squidGuard.conf.bak
2. sudo rm /etc/squidguard/squidGuard.conf
3.sudo nano /etc/suqidgurd/squidGuard.conf
Copy and paste this,

#—————————–squidGuard.conf—————————————————————
#
# CONFIG FILE FOR SQUIDGUARD
#
dbhome /var/lib/squidguard/db
logdir /usr/local/squidGuard/logs
    dest porn {
    domainlist porn/domains
    urllist porn/urls
}
dest adv {
    domainlist adv/domains    
    urllist adv/urls
}
dest spyware {
    domainlist spyware/domains
    urllist spyware/urls
}
acl {
    default {
        pass !porn !adv !spyware all
        redirect http://localhost/block.html
        }
}

#————————————————————————————————————

You can test your squidguard by doing a dry run
sudo echo "http://www.pornhub.com 10.50.55.10/- – GET" | squidGuard -c /etc/squidguard/squidGuard.conf –d

You should see,
squidGuard ready for requests
squidGuard stopped
If there are errors, it will tell you.. The most likely errors you’ll run into are permission issues.. If it gives you permission issues with your database, make sure that you set the user and group named “proxy” ownership. You can tell that by “sudo ls -l /var/lib/squidguard/db*”

You can now use the Firefox browser you setup to use with your proxy server to make sure you are blocking porn and ads. For better protection, I recommend using the blacklist from,  http://urlblacklist.com/

Finish with:
cd /var/lib/squidguard/db/; squidGuard -C all ; chown proxy:proxy -R /var/lib/squidguard/db/ ; squid -k reconfigure;  service squid restart

Credits