HTTP WAN ACCELERATOR - ZIPROXY JPEG 2000 + SQUID
Instructions on how to implement a WAN accelerator with Ziproxy, with JPEG 2000 and integration with Squid

1. - INTRODUCTION

This document describes a way to implement a HTTP WAN accelerator using Ziproxy, there are many ways this can be done and this one is just a specific way, one which was put in production in a certain network. Here it is described a real production environment.
In this document it is expected that the remote proxy to be configured as a transparent proxy, by the sole reason the structure this document is based on being that way. So, transparent proxy is not mandatory, if you prefer to chain all the steps with conventional proxying it's up to you (actually it's even better).

Basically the original structure used Ziproxy for JPEG and Gzip compression, all the work being done at the remote side and transparent as possible to the local sysadmin (thus, transparent proxy was used). There were some issues we wanted to improve/address:

You'll notice the Local Network runs two Ziproxy instances. Couldn't it be done with just one?
Yes, it's possible to do that with just one Ziproxy instance. The reason a 2-instance Ziproxy solution was chosen, is to improve Squid's cache performance. -- Storing small JPEG 2000 files is much more efficient than huge high-quality JPEG ones.

For performance reasons the hosts running Ziproxy should have, at least, 1MB of L2. The most, the better.
In certain cases with low traffic, 512kB L2 may be acceptable.

2. - Process description

Once implemented, the structure is expected to work the following way:
  1. The file is collected from the remote HTTP server by the Ziproxy-FAST. If it's a picture, it is recompressed to JPEG 2000 *. Otherwise, if it's html/text the file is gzipped.
  2. The file is transmitted through the low speed link to the Local Network.
  3. Once arrived at the Local Network, the html/text files are "de-Gzipped" by the Ziproxy-LINK. Other content types are left unmodified.
  4. Those files are transported to the Squid, which may cache them, apply ACLs etc.
  5. From Squid, those files are sent to Ziproxy-User, where the JPEG 2000 pictures are reconverted to conventional JPEG (with very high quality).
  6. From Ziproxy-User, the files go to the user's WWW browser.
* - The pictures will be recompressed to JPEG 2000 only if there are Ziproxies installed in the Local Network (auto-detected by Ziproxy-FAST). Otherwise, the pictures will be recompressed to conventional JPEG.


3. - Local Network requirements


THE CONFIGURATION FILES (BELOW) ASSUME CERTAIN PARAMETERS SUCH AS IP ADDRESSES AND PORTS, THOSE MAY NOT BE THE SAME AS THE ONES USED IN YOUR NETWORK
Change the relevant settings according to your network structure.

4. - Installing and configuring Ziproxy LINK

Create a file called /etc/ziproxy/ziproxy-link.conf with the following contents:

# Ziproxy LINK will listen to this port
Port = 8081

# the host's IP where Ziproxy LINK will listen for connections from Squid
Address = "127.0.0.1"

# if you want access logs, uncomment the following line
# AccessLogFileName = "/var/log/ziproxy/link-access.log"

TransparentProxy = false
ConventionalProxy = true
AllowMethodCONNECT = false
UseContentLength = false
Compressible = {
        "shockwave", "msword", "msexcel", "mspowerpoint", "rtf", "postscript",  
        "java", "javascript", "staroffice", "vnd.", "futuresplash", 
        "asp", "class", "font", "truetype-font", "php", "cgi", "executable",
        "shellscript", "perl", "python", "awk", "dvi", "css",
        "xhtml+xml", "rss+xml", "xml", "pdf", "tar"
}
ProcessJPG = false
ProcessPNG = false
ProcessGIF = false
ImageQuality = {20,15,15,15}
ModifySuffixes = false
ProcessJP2 = false
ProcessToJP2 = true
ForceOutputNoJP2 = false
AnnounceJP2Capability = true
5. - Installing and configuring Ziproxy USER

Create a file called /etc/ziproxy/ziproxy-user.conf with the following contents:

# Ziproxy USER will listen to this port
Port = 8080

# the local IP this host will use to listen for incoming connections from the users.
# this IP CANNOT BE LOCALHOST, it must be accessibe by the users' machines.
Address = "172.31.50.1"

# if you want access logs, uncomment the following line
# AccessLogFileName = "/var/log/ziproxy/user-access.log"

# this is the IP which Squid is listening for incoming connections
# (it may be the same as Ziproxy, if it's the same host, and that's what
# we're assuming here).
# ATTENTION: be sure that Squid is really listening through this IP!
NextProxy="127.0.0.1"

# this is the port which Squid is listening
NextPort=3128

# Is your proxy a transparent one?
# If yes, change this to "true"
# and disable transparent proxying in Squid
TransparentProxy = false

ConventionalProxy = true
AllowMethodCONNECT = false
UseContentLength = false
Compressible = {
        "shockwave", "msword", "msexcel", "mspowerpoint", "rtf", "postscript",  
        "java", "javascript", "staroffice", "vnd.", "futuresplash", 
        "asp", "class", "font", "truetype-font", "php", "cgi", "executable",
        "shellscript", "perl", "python", "awk", "dvi", "css",
        "xhtml+xml", "rss+xml", "xml", "pdf", "tar"
}
ProcessJPG = false
ProcessPNG = false
ProcessGIF = false
ImageQuality = {95,95,95,95}
ModifySuffixes = false
ProcessJP2 = true
ProcessToJP2 = false
ForceOutputNoJP2 = true
6. - Configuring Squid

Add the following lines to /etc/squid/squid.conf:
# "Hooks" Squid to Ziproxy LINK running in 'localhost', port '8081'.
# Note: Ziproxy does not support ICP nor cache querying
# since it is not a caching proxy.
cache_peer localhost parent 8081 0 no-query no-digest

# Prevents Squid trying to access directly the remote HTTP host
# if Squid is unable to connect to Ziproxy,
# otherwise you won't know whether Ziproxy is down/has_problems
# (the lack of gzip support is not obvious to the user).
never_direct allow all
7. - Starting the services

Start the two Ziproxy instances:
$ ziproxy -d -c /etc/ziproxy/ziproxy-link.conf
$ ziproxy -d -c /etc/ziproxy/ziproxy-user.conf

Insert the two lines above to /etc/rc.local in order to start the two Ziproxy instances during boot.
If your OS, while installing Ziproxy, automatically enabled Ziproxy execution during boot (with its initscript), disable that in order to avoid conflicts with Ziproxy-LINK and Ziproxy-User!

*** WIPE CLEAN SQUID'S CACHE ***
Otherwise the pictures with the former compression settings (or lack of) will appear, presenting to the user a mix of image qualities and degrading the user experience.

Restart Squid: 
$ /etc/init.d/squid restart

8. - Configuring the user's web browser

The users will connect to Ziproxy-User.
If the connections are done to Squid directly, the users will receive JPEG 2000 pictures and (most likely) won't be able to view those pictures!

Proxy (IP or proxy's hostname): 172.31.50.1
Port: 8080

Depending on the configuration, it may be necessary to use iptables (Linux, or equivalent in other OSes) and do port redirection and/or blocking.

9. - Configuring Ziproxy-FAST

Here is where the compression will be done.
The settings are pretty much conventional except for few options in /etc/ziproxy/ziproxy.conf:
This is not a complete configuration file, do not just copy/paste this!!!
## Port to listen for proxy connections
## We're using transparent proxy, so redir all HTTP/80 to this host and port.
Port = 3128

## Local address to listen for proxy connections
## If you have more than one network interface,
## it's useful for restricting to which interface you want to bind to.
## By default Ziproxy binds to all interfaces.
# Address = "127.0.0.1"

## Enabling logging is interesting to know how how the compression is going.
# AccessLogFileName = "/var/log/ziproxy/access.log"

## If you're using AccessLogFileName this may be interesting to enable.
# AccessLogUserPOV = true

## Do you have too many users and certain sites are complaining of too
## many access from the same IP? You may find this option useful.
# BindOutgoing = {"20.17.97.53", "20.17.97.46", "20.17.97.47", "20.17.97.48"}

## If you're using BindOutgoing, you may need (or not) the following options too.
# BindOutgoingExList="/etc/ziproxy/bo_exception.list"
# BindOutgoingExAddr="20.17.97.53"

## We're using transparent proxy!
TransparentProxy = true

## We're using transparent proxy ONLY!
## Conventional proxy requests are disabled.
ConventionalProxy = false

## Again, we're using transparent proxy.
## For security reasons, we disable the CONNECT method.
AllowMethodCONNECT = false

## Image quality when recompressing to conventional JPEG (not JPEG 2000)
ImageQuality = {30,25,25,25}

## Enable recompression to JPEG 2000
ProcessToJP2 = true

## Send JPEG 2000 ONLY if the client is a Ziproxy with JPEG 2000 support enabled
JP2OutRequiresExpCap = true

## Image quality when recompressing to JPEG 2000
JP2ImageQuality = {30,25,25,25}

## If you want an ad-blocker or something,
## this is the place to enable.
# URLReplaceData = "/etc/ziproxy/replace.list"

Last updated: 2009-02-22