Install and Configure Squid Proxy Server, ClamAV, SquidClamav, C-ICAP Server – Debian 9 (minimal – standard system utilities, ssh server)

1. Install some needed dependencies.
$sudo apt-get install gcc make curl libcurl4-gnutls-dev rsync

2. Install and Configure Squid Proxy Server.
$sudo apt-get install squid3 calamaris

3.Edit config file /etc/squid/squid.conf:
$sudo cp squid.conf squid.conf.ORIG

Backup
$sudo cp /etc/squid/squid.conf /etc/squid/squid.conf.bak

To simplify the configuration file (squid.conf), we can remove everything that is comments or blank lines.
$sudo cat squid.conf.bak | egrep -v -e '^[[:blank:]]*#|^$' > squid.conf

$sudo nano /etc/squid/squid.conf

3.1 Change squid.conf options

Make sure the line is uncommented (#).
acl CONNECT method CONNECT

Create new access lists acl LAN to your internal network 192.168.0.0/24. or others internal network
acl LAN src 192.168.0.0/24
acl LAN src xxx.xxx.x.x/24

Additional access lists blacklist, whitelist, malware_block_list to block spam, commercials, malware, viruses…

acl malware_block_list url_regex -i "/etc/squid/malware_block_list"
acl blacklist dstdom_regex "/etc/squid/blacklist"
acl whitelist dstdom_regex "/etc/squid/whitelist"

Access new acl lists – order matters:

http_access allow whitelist
http_access deny blacklist
http_access deny malware_block_list
http_access allow LAN

I did not use: Inform users about blocked website. Blocked commercials will be displayed as empty transparent place, require http server.
deny_info http://YourServerName/error/dot-transparent.png blacklist
deny_info http://YourServerName/error/dot-transparent.png whitelist
deny_info http://YourServerName/error/error.html malware_block_list

Setup address IP and listening port. Transparent mean no caching.
http_port 192.168.0.1:3128

Additional  setup – Anonymizer. Blocking headers:
request_header_access Allow allow all
request_header_access Authorization allow all
request_header_access WWW-Authenticate allow all
request_header_access Proxy-Authorization allow all
request_header_access Proxy-Authenticate allow all
request_header_access Content-Encoding allow all
request_header_access Content-Length allow all
request_header_access Content-Type allow all
request_header_access Date allow all
request_header_access Expires allow all
request_header_access Host allow all
request_header_access If-Modified-Since allow all
request_header_access Last-Modified allow all
request_header_access Location allow all
request_header_access Pragma allow all
request_header_access Accept allow all
request_header_access Accept-Charset allow all
request_header_access Accept-Encoding allow all
request_header_access Accept-Language allow all
request_header_access Content-Language allow all
request_header_access Mime-Version allow all
request_header_access Retry-After allow all
request_header_access Title allow all
request_header_access Connection allow all
request_header_access Proxy-Connection allow all
request_header_access User-Agent allow all
request_header_access Cookie allow all
request_header_access Referer deny all
request_header_access X-Forwarded-For deny all
request_header_access Via deny all
request_header_access All deny all
request_header_access Cache-Control deny all
httpd_suppress_version_string on

Store cache ojects only in memory, cache (400MB)

## Cache options

cache_mem 512 MB
cache_dir ufs /var/spool/squid3 400 16 256
  ### cache_dir ufs /usr/local/squid/cache 51200 64 256

Disable cache for access list – LAN:
cache deny LAN

Hostname
visible_hostname YourServerName

Hiding IP
forwarded_for off

##———————- My test config ——————————————
acl LAN src 192.168.0.0/24
acl LAN src 192.168.100.0/24
acl LAN src 192.168.122.0/24

acl SSL_ports port 443
acl Safe_ports port 80        # http
acl Safe_ports port 21        # ftp
acl Safe_ports port 443        # https
acl Safe_ports port 70        # gopher
acl Safe_ports port 210        # wais
acl Safe_ports port 1025-65535    # unregistered ports
acl Safe_ports port 280        # http-mgmt
acl Safe_ports port 488        # gss-http
acl Safe_ports port 591        # filemaker
acl Safe_ports port 777        # multiling http
acl CONNECT method CONNECT

acl malware_block_list url_regex -i "/etc/squid/malware_block_list"
acl blacklist dstdom_regex "/etc/squid/blacklist"
acl whitelist dstdom_regex "/etc/squid/whitelist"
http_access allow whitelist
http_access deny blacklist
http_access deny malware_block_list

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow LAN
http_access allow localhost
http_access deny all

visible_hostname proxy
http_port 3128
coredump_dir /var/spool/squid

refresh_pattern ^ftp:        1440    20%    10080
refresh_pattern ^gopher:    1440    0%    1440
refresh_pattern -i (/cgi-bin/|\?) 0    0%    0
refresh_pattern .        0    20%    4320

request_header_access Allow allow all
request_header_access Authorization allow all
request_header_access WWW-Authenticate allow all
request_header_access Proxy-Authorization allow all
request_header_access Proxy-Authenticate allow all
request_header_access Content-Encoding allow all
request_header_access Content-Length allow all
request_header_access Content-Type allow all
request_header_access Date allow all
request_header_access Expires allow all
request_header_access Host allow all
request_header_access If-Modified-Since allow all
request_header_access Last-Modified allow all
request_header_access Location allow all
request_header_access Pragma allow all
request_header_access Accept allow all
request_header_access Accept-Charset allow all
request_header_access Accept-Encoding allow all
request_header_access Accept-Language allow all
request_header_access Content-Language allow all
request_header_access Mime-Version allow all
request_header_access Retry-After allow all
request_header_access Title allow all
request_header_access Connection allow all
request_header_access Proxy-Connection allow all
request_header_access User-Agent allow all
request_header_access Cookie allow all
request_header_access Referer deny all
request_header_access X-Forwarded-For deny all
request_header_access Via deny all
request_header_access All deny all
request_header_access Cache-Control deny all
httpd_suppress_version_string on

 

## Cache options

cache_mem 512 MB
cache_dir ufs /var/spool/squid 400 16 256
cache deny LAN
visible_hostname proxy
forwarded_for off

##———————————————–——–——–——–

3.2 Download files: blacklist and whitelist, unpack and save it to /etc/squid/.
$wget -c https://ffteixeira.net/blog/sites/default/files/blacklist.tar_.bz2 (rename blacklist.tar_.bz2 to blacklist.tar.bz2) || wget -c http://terminal28.com/wp-content/uploads/2013/10/blacklist.tar.bz2
$sudo tar -xvf blacklist.tar.bz2
$sudo mv blacklist whitelist /etc/squid

Before trying to start Squid, you should verify that your squid.conf file makes sense. This is easy to do. Just run the following command:
$sudo squid -k parse

Ignore this error, solved next step:
…/08/11 12:23:16| Processing: acl malware_block_list url_regex -i "/etc/squid/malware_block_list"
…/08/11 12:23:16| ERROR: Can not open file /etc/squid/malware_block_list for reading
…/08/11 12:23:16| Warning: empty ACL: acl malware_block_list url_regex -i "/etc/squid/malware_block_list"
…/08/11 12:23:16| Processing: acl blacklist dstdom_regex "/etc/squid/blacklist"
…/08/11 12:23:17| /etc/squid/squid.conf line 20: acl blacklist dstdom_regex "/etc/squid/blacklist"
…/08/11 12:23:17| WARNING: there are more than 100 regular expressions. Consider using less REs or use rules without expressions like 'dstdomain'.

Restart Squid.
$sudo /etc/init.d/squid restart

3.3 Download script malware_block_list to update domains and IP addresses , unpack and save it to /etc/squid
$wget -c https://ffteixeira.net/blog/sites/default/files/malware_block_list.tar_.bz2 (rename malware_block_list.tar_.bz2  to malware_block_list.tar.bz2) || wget -c http://terminal28.com/wp-content/uploads/2013/10/malware_block_list.tar.bz2
$sudo tar -xvf malware_block_list.tar.bz2
$sudo mv malware_block_list /usr/local/bin/
$sudo chmod +x /usr/local/bin/malware_block_list
$sudo touch  /var/log/malware_block_list.log

Add script malware_block_list to Cron.
$sudo crontab -e

add
    
@daily /usr/local/bin/malware_block_list

Logfile location: /var/log/malware_block_list.log.   Go to  MalwarePatrol.net, click tab: Block List. You should see subscription list: free and paid. Click Free/Subscribe. Subscribe the list. You should get password/receipt number on email. Log in to: https://www.malwarepatrol.net/login.php; and find Squid Web Proxy ACL and click Download. You will be redirected to website/text with malware list. Every subscription has unique receipt number receipt=f1234567890. https://lists.malwarepatrol.net/cgi/getfile?receipt=f1234567890&product=8&list=squid Copy URL and paste to script near link. Edit: link, user, pass.

$sudo nano /usr/local/bin/malware_block_list

link='PASTE_LINK_FROM_MALWAREPATROL.NET'
user='–http-user=USERNAME'
passwd='–http-passwd=PASSWORD'

Note: Change squid3 to squid

##————————————- My test config. —————————
#!/bin/sh
### ###
###
### Squid3 Installation and Configuration.
###
### Polish version
###
### http://man.sethuper.com/instalacja-squid-proxy-serwer-clamav-squidclamav-c-icap-serwer-debian-6-0-x
###
#=======================================================================================================================
###
### English version
###
### http://terminal28.com/how-to-install-and-configure-squid-proxy-server-clamav-squidclamav-c-icap-server-debian-linux/
###
### ###

# If you don't want to log wget debug output remove "$debug" in line (51) "fetchcmd"

## Setings
# Malware patrol URL with unique ID
# Change ID after receipt in link (..getfile?receipt=f138125701..)
link='https://lists.malwarepatrol.net/cgi/getfile?receipt=f1502379316&product=8&list=squid'

# HTTP USER
user='–http-user=<user>'

# HTTP PASSWORD
pass='–http-passwd=<passwd>'

# Checking certificate
cert='–no-check-certificate'

# File location for Squid
target='/etc/squid/malware_block_list'

# Reload Squid
reloadcmd='/usr/sbin/squid -k reconfigure'

# Temporary file
tmp="/tmp/.malware_block_list.$$"

# Wget debud
#debug="-nva /var/log/squid/malware_block_list.log"

# Command for download malware list
#I remove de debug because error
#fetchcmd="wget -q  –no-check-certificate  $link -O $tmp $user $pass $debug"
fetchcmd="wget -q  –no-check-certificate  $link -O $tmp $user $pass"

# ——-

# Log file
logs='/var/log/squid/malware_block_list.log'

## execution
##
echo "$(date -R) Downloading new malware_block_list" >> "$logs"

# Downloading new malware_block_list from Malware Patrol
$fetchcmd

# Checking temporary file – "OK" – before overwrite old malware list
if [ ! -s $tmp ]
then
echo "$(date -R) The temporary file '$tmp' does not exist or is empty; resignation" >> "$logs"
exit
fi

# moving malware_black_list to directory /etc/squid3/
cp  $tmp $target

# removing temporary file
rm $tmp

# restart Squid
$reloadcmd
##———————————————————————————

 
$sudo sh /usr/local/bin/malware_block_list

4. Install Clamav-server.
$sudo apt-get install clamav-daemon
$sudo mkdir install
$cd install
$sudo wget https://sourceforge.net/projects/c-icap/files/c-icap/0.5.x/c_icap-0.5.2.tar.gz/download -O c_icap-0.5.2.tar.gz
$sudo tar -xvf c_icap-0.5.2.tar.gz
$cd c_icap-0.5.2
$sudo ./configure
$sudo make
$sudo make install
$cd ..

Edit configfile  /usr/local/etc/c-icap.conf.
$sudo nano /usr/local/etc/c-icap.conf

Change:

Line 223: ServerAdmin root@localhost
Line 232: ServerName YourServerName

Add at line 708:

Service squidclamav squidclamav.so

4.1 C-ICAP server autostart script.
$wget -c  https://ffteixeira.net/blog/sites/default/files/c-icap-autostart.tar_.gz (rename c-icap-autostart.tar_.gz  to c-icap-autostart.tar.gz) || wget -c http://terminal28.com/wp-content/uploads/2013/10/c-icap-autostart.tar.gz
$sudo tar xvf c-icap-autostart.tar.gz
$sudo rsync -avh init.d default /etc
$sudo update-rc.d c-icap defaults

4.2 Create logrotate script for c-icap server.
$sudo cat << EOT > /etc/logrotate.d/c-icap

/usr/local/var/log/server.log /usr/local/var/log/access.log {
     daily
     rotate 4
     missingok
     notifempty
     compress
     create 0644 root root
     postrotate
     /etc/init.d/c-icap force-reload > /dev/null
     endscript
}
EOT

4.3 Change permission for c-icap logrotate script and server logs.
$sudo chmod 644 /etc/logrotate.d/c-icap
$sudo chown root:root /etc/logrotate.d/c-icap
$sudo chmod 644 /usr/local/var/log/ -R
$sudo chown root:root /usr/local/var/log/ -R
$sudo ln -s /usr/local/var/log/server.log /var/log/server.log
$sudo ln -s /usr/local/var/log/access.log /var/log/access.log

5. Install Squidclamav
$cd install
$wget -c https://sourceforge.net/projects/squidclamav/files/squidclamav/6.15/squidclamav-6.15.tar.gz/download -O squidclamav-6.15.tar.gz
$sudo tar zxvf squidclamav-6.15.tar.gz
$cd squidclamav-6.15
$sudo ./configure
$sudo make
$sudo make install
$cp -rf cgi-bin /usr/lib/
$chmod +x /usr/lib/cgi-bin/clwarn* -R
$chown www-data:www-data /usr/lib/cgi-bin/clwarn* -R
$cd ..
$sudo ldconfig

5.1 Configure squidclamav.
$sudo nano /usr/lcocal/etc/squidclamav.conf

Add redirect URL – default script – clwarn.cgi (en). You can choose diferent language: DE, FR, BR, RU.

Line 18: redirect http://YourServerName/cgi-bin/clwarn.cgi

Make sure the rule occurs in configfile.

Line 27: clamd_local /var/run/clamav/clamd.ctl

6. Checking config file – ClamAV, make sure the rule occurs in configfile.
$sudo nano /etc/clamav/clamd.conf

Line 4: LocalSocket /var/run/clamav/clamd.ctl

Configure Freshclam.
$sudo nano /etc/clamav/freshclam.conf

Line 22: SafeBrowsing true

6.1 Register on Securiteinfo.com: https://www.securiteinfo.com/clients/customers/signup
Subscribe basic list for clamav. You should get auto generated urls for clamav database under tab: Setup.
Download allowed from 1 IP address, limited to 24 downloads per day
Add generated URLS to freshclam.conf file at the end.

DatabaseCustomURL http://www.securiteinfo.com/get/signatures/3b4d0…5764/securiteinfo.hdb
DatabaseCustomURL http://www.securiteinfo.com/get/signatures/3b4b…eafd/securiteinfo.ign2
DatabaseCustomURL http://www.securiteinfo.com/get/signatures/3b4d0d…61eafd/javascript.ndb
DatabaseCustomURL http://www.securiteinfo.com/get/signatures/34d…81f/spam_marketing.ndb
DatabaseCustomURL http://www.securiteinfo.com/get/signatures/3b…61eafd/securiteinfohtml.hdb
DatabaseCustomURL http://www.securiteinfo.com/get/signatures/3b…365afd/securiteinfoascii.hdb

Restart ClamAV.
$sudo /etc/init.d/clamav-daemon restart

7. Configure Squid with C-ICAP. Configuration for Squid version – 3.1.20.
$sudo nano /etc/squid/squid.conf

Add at the end of the file

icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_header X-Authenticated-User
icap_service service_req reqmod_precache bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access service_req allow all
icap_service service_resp respmod_precache bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access service_resp allow all

Configuration for Squid version – 3.1.6.
$sudo nano /etc/squid/squid.conf

Add at the end of the file

icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Client-Username
icap_preview_enable on
icap_preview_size 1024
adaptation_service_set service_req
icap_service service_req reqmod_precache bypass=1 icap://127.0.0.1:1344/request
adaptation_access service_req allow all

adaptation_service_set service_resp
icap_service service_resp respmod_precache bypass=0 icap://127.0.0.1:1344/response
adaptation_access service_resp allow all

Run C-ICAP server.
$sudo /usr/local/bin/c-icap &

8. Restart Squid.
$sudo chown -R proxy:proxy /var/spool/squid
$sudo squid -z
$sudo service squid restart

9. Configure firewall – masquerade, prerouting.
Enable forwarding. Edit configfile sysctl.conf
$sudo nano /etc/sysctl.conf

Uncomment IPv4 i IPv6 and change to 1:

Line 28: net.ipv4.ip_forward = 1
Line 33: net.ipv6.conf.all.forwarding = 1

##——————- Not used ———————
9.1. Configure firewall – iptables.

$sudo nano /etc/iptables.up.rules

Add rules (Change address IP and network interface)

*nat

-A PREROUTING -p tcp -m tcp -i eth1 –dport 80 -j REDIRECT –to-ports 3128
-A POSTROUTING -s 192.168.0.0/24 -j MASQUERADE

 ##—————————————————–

10. Test.

If you have done it right then..
.. go to: http://www.eicar.org/85-0-Download.html and try to download file:

eicar.com
68 Bytes

Result:
You should be redirected to:

    http://YourServerName/cgi-bin/clwarn.cgi, http://YourServerName/error.html.

11. Sarg and squidguard

Credits

Explicit vs. Transparent Proxy

A proxy server is a server (a computer system or an application) that acts as an intermediary for requests from clients seeking resources from other servers. A client connects to the proxy server, requesting some service, such as a file, connection, web page, or other resource available from a different server and the proxy server evaluates the request as a way to simplify and control its complexity. Proxies were invented to add structure and encapsulation to distributed systems. Today, most proxies are web proxies, facilitating access to content on the World Wide Web and providing anonymity.1

In an explicit proxy configuration, the client (e.g. browser, desktop application etc.) is explicitly configured to use a proxy server, meaning the client knows that all requests will go through a proxy. The client is given the hostname/IP address and port number of the proxy service. When a user makes a request, the client connects to the proxy service and sends the request. The disadvantage to explicit proxy is that each client must be properly configured to use the proxy.

In a transparent proxy configuration, the proxy is typically deployed at the Internet gateway and the proxy service is configured to intercept traffic for a specified port. The client (e.g. browser, desktop application etc.) is unaware that traffic is being processed by a proxy. For example, a transparent HTTP proxy is configured to intercept all traffic on port 80/443. The typical benefits of a transparent proxy include a standard enterprise configuration where all clients routed to the internet will always be filtered and protected no matter what the end users do, or change, on their machines and the added benefit of reduction in typical user’s client-proxy configuration troubleshooting.

Credits