Mobile browsing & content optimization

Vincent Bernat

Since my lap­top fea­tures WWAN con­nec­tiv­ity, I pur­chased a 3G mo­bile broad­band plan from SFR, a French mo­bile phone com­pany. I took the cheap­est one avail­able: 250 MiB per month for €10.

Optimizing transparent proxy by SFR#

While brows­ing, I was cu­ri­ous about the fact that there were mul­ti­ple ac­cesses to non-pub­lic IP, like 10.141.0.4. I was brows­ing the mo­bile ver­sion of Le Monde, a French daily news­pa­per. I en­abled Chromium de­vel­oper tools and watched what hap­pened in the “Net­work” tab. All im­ages were served from IP like 10.141.0.4. Look­ing at the HTML source code, I dis­cov­ered that <img> tags were rewrit­ten to point to these im­ages. The smaller ones were iden­ti­cal to the orig­i­nal one but the larger ones (more than 50 KiB) were of re­duced qual­ity (and there­fore re­duced size).

<img> tags were not the only ones to be rewrit­ten. <style> and <script> tags that were point­ing to ex­ter­nal re­sources (with a src at­tribute) were also rewrit­ten to in­clude the CSS or the Java­Script in­line. Before:

<script src="http://s1.lemde.fr/js/lib/jquery/1.5.1/jquery.js"></script>

After:

<script oldSrc="http://s1.lemde.fr/js/lib/jquery/1.5.1/jquery.js" NG="replaced">
/*!
 * jQuery JavaScript Library v1.5.1
 *
 * Copyright 2011, John Resig
 * Dual licensed under the MIT or GPL Version 2 licenses.
 * http://jquery.org/license
 * […] */
</script>

This means that each page that in­cludes jQuery, a pop­u­lar Java­Script li­brary, as an ex­ter­nal re­source, will be turned into a page that in­cludes the whole jQuery code which is about 85 KiB. Each page will weigh more than 85 KiB in­stead of re­ly­ing on an ex­ter­nal and cacheable resource.

Nowa­days, most browsers sup­port HTTP keep-alive which al­lows one to serve mul­ti­ple re­sources into a sin­gle re­quest. There are two cases:

  1. If the re­source is served from the same server as the HTML page, keep-alive would en­able us to get al­most the same re­sult with­out hin­der­ing the cache: the browser would first re­quest the HTML page, then the Java­Script re­source into the same TCP con­nec­tion. There is a small penalty of about 100 ms to let the browser re­quest the sec­ond re­source. The browser may de­cide to open an­other con­nec­tion be­cause the ini­tial doc­u­ment is not loaded yet. Some browsers may even ne­go­ti­ate this ad­di­tional con­nec­tion be­fore need­ing it. In this case, it can use this idle con­nec­tion at no cost. Oth­er­wise, we may fall into the sec­ond case.

  2. If the re­source is served from an­other server (like in the ex­am­ple above) or if the browser does not have an idle con­nec­tion ready for re­use, it would down­load the re­source in par­al­lel with the help of an­other con­nec­tion. It will only cost about 300 ms of ad­di­tional time due to the es­tab­lish­ment of a new TCP con­nec­tion with the server. These 300 ms are a very small price to pay com­pared to the 2 sec­onds needed to down­load 85 KiB for each page. More­over, other static re­sources will likely be down­loaded from the same server and there­fore, this is 300 ms the first time and about 100 ms for ad­di­tional re­sources (round-trip time for a new HTTP re­quest into the same TCP con­nection).

Up­date (2011-06)

In a pre­vi­ous ver­sion of this ar­ti­cle, I con­fused pipe­lin­ing and keep-alive mech­a­nisms. While keep-alive al­lows one to re­use an ex­ist­ing con­nec­tion to down­load a new re­source, only one re­quest at a time by con­nec­tion could be is­sued. Pipe­lin­ing al­lows one to push ad­di­tional re­quests in the same TCP con­nec­tion with­out wait­ing for an­swers. As noted by Éric Daspet in a com­ment, pipe­lin­ing is dis­abled for most browsers. How­ever, keep-alive is well supported.

Search­ing on the web for oldSrc and NG="replaced", I find very lit­tle in­for­ma­tion about what is used. It seems that such a fea­ture may also break some scripts, like TinyMCE, try­ing to get ad­di­tional re­sources rel­a­tive to the path where the script is in­stalled. It ap­pears that SFR is us­ing a prod­uct from Flash Net­works called Har­mony Gate­way. I have asked SFR more info about this and if it was pos­si­ble to dis­able the use of this fea­ture but got no an­swer so far.

I have re­tried a bit later and there was no rewrit­ing for Java­Script and CSS. Im­ages were still de­graded and served by a spe­cial server. The next day, my re­quests did not travel through a trans­par­ent proxy any­more (no more Via header in an­swers). Each try was done at a dif­fer­ent lo­ca­tion. The prox­ies may be in­stalled at some lo­ca­tions only and con­fig­ured dif­fer­ently. An­other ex­pla­na­tion is that SFR silently dis­abled this func­tion­al­ity for my account.

If it was done cor­rectly, such an ac­cel­er­a­tor ser­vice could be pro­posed as an opt-out op­tion since most users would ap­pre­ci­ate faster page down­loads. If you care about net­work neu­tral­ity, you can dis­able this op­tion, but it needs to be prop­erly ad­ver­tised, for ex­am­ple as a free op­tion in your mo­bile plan.

Setting up our own optimizing proxy#

Google pro­vides mod_​pagespeed as a so­lu­tion to op­ti­mize a web­site on the fly. This is an Apache mod­ule. It is tar­geted at op­ti­miz­ing your web­site but since Apache can act as a reg­u­lar proxy, it seems pos­si­ble to en­able mod_​pagespeed for any website.

That’s easy to set up if you don’t mind us­ing a pre­com­piled ver­sion of mod_​pagespeed. Down­load mod_​pagespeed. I have down­loaded the 64-bit ver­sion for Debian.

# touch /etc/default/mod-pagespeed
# dpkg -i ~/download/mod-pagespeed-beta_current_amd64.deb
Selecting previously deselected package mod-pagespeed-beta.
(Reading database ... 277699 files and directories currently installed.)
Unpacking mod-pagespeed-beta (from .../mod-pagespeed-beta_current_amd64.deb) ...
dpkg: dependency problems prevent configuration of mod-pagespeed-beta:
 mod-pagespeed-beta depends on apache2.2-common; however:
  Package apache2.2-common is not installed.
dpkg: error processing mod-pagespeed-beta (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 mod-pagespeed-beta
# apt-get install -f
[…]
Unpacking apache2.2-common (from .../apache2.2-common_2.2.19-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up apache2.2-common (2.2.19-1) ...
[…]
# apt-get install apache2-mpm-worker
[…]
Unpacking apache2-mpm-worker (from .../apache2-mpm-worker_2.2.19-1_amd64.deb) ...
Setting up apache2-mpm-worker (2.2.19-1) ...
Starting web server: apache2.
# a2enmod proxy_http
Considering dependency proxy for proxy_http:
Enabling module proxy.
Enabling module proxy_http.
To activate the new configuration, you need to run:
  /etc/init.d/apache2 restart

Then, we mod­ify /etc/​apache2/​mods-enabled/​proxy.conf:

ProxyRequests On
<Proxy *>
    AddDefaultCharset off
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
</Proxy>

This makes lit­tle sense to run the proxy on your lap­top since you do not want to down­load the non-op­ti­mized con­tent over your WWAN link. You need to run it on an­other server and there­fore al­low your lap­top to access it.

Also, we need to mod­ify /etc/​apache2/​mods-enabled/​pagespeed.​conf to add the di­rec­tive ModPagespeedDomain * to let mod_​pagespeed re­write any page. You should restart Apache af­ter that. Now, check if the proxy is working:

$ curl -six http://localhost:80 http://www.lemonde.fr/ | grep ^X-Mod-Pagespeed
X-Mod-Pagespeed: 0.9.17.7-716

All is work­ing! Let’s try a very sim­ple site, like De­bian Back­ports. It fea­tures one HTML file and two CSS files:

Debian Backports website without pagespeed
Debian Backports website without pagespeed

We down­loaded a to­tal of 8 KiB (com­pres­sion in­cluded). Let’s use our proxy and try again:

Debian Backports website with pagespeed
Debian Backports website with pagespeed

mod_​pagespeed suc­ceeds in re­duc­ing the size of the HTML file and com­bined the CSS files. More­over, it mini­fied the re­sult­ing CSS file. There seems to be a bug in Chromium that dis­plays in­cor­rect trans­fer sizes (in­clud­ing in the time­line rep­re­sen­ta­tion) when us­ing a proxy. I have checked with Wire­shark that the over­all quan­tity trans­ferred was about 8 KiB. The gain is very thin here but you can imag­ine that you will get bet­ter re­sults on larger sites.

mod_​pagespeed seems to han­dle well most sites. How­ever, it only op­ti­mizes pages that al­ready are in its cache. The first time you grab a page, you will get the un­op­ti­mized ver­sion. I do not know if it is the ex­pected behavior.