futtta's blog

As my Acer e110 doesn’t sync with Google, all of my precious contacts in the cloud did not automagically appear on my handset. That left me little but no choice to go the old-fashioned way; the export/import-dance.

Exporting from Google is easy, but it generates one vcard-file with all you contacts in it, which the contacts app in Android 1.5 can only import the first entry from. To split up the contacts-file, Scroogle pointed me vCard list-file splitter, vcf-split for short, a Perl script from back in the days when Windows linebreaks apparently were sufficient evidence of the end of a vcard. But times and technology have changed and linebreaks have lost their former glory, meaning I had to slightly alter the script to watch out for a cryptic “END:VCARD” line to indicate the end of a vcard.

And the script goes a little something like this:

#!/usr/bin/perl
#
# vcf-split - split a .list.vcf file into many small .vcf files.
# Copyright (C) 2004  Raphael J. Schmid.
# Tweaked by Frank Goossens ("futtta") in 2011
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 1, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
#
# -- raphael.schmid@gmx.de
# -- futtta@gmail.com

use File::Basename;

if ($ARGV[0] eq "") {
  print "Usage: vcard-split <file to split>\n\n";
  exit;
}

$input=$ARGV[0];
$counter=0;

print $input;

open INPUT, $input or die $!;

while (<INPUT>) {
  open OUTPUT, ">> ".$counter."-".basename($input) or die $!;

  if ($_ =~ /END:VCARD/ ) {
    print OUTPUT $_;
    $counter+=1;
    close OUTPUT;
  } else {
    print OUTPUT $_;
  }
}

close OUTPUT;
close INPUT;

Who knows one day Google will send someone this way who has some vcf-splitting to do?

If you want to avoid the learning curve of Squid and Varnish or the cost of a dedicated caching & proxying appliance, using Apache with mod_cache may seem like a good, simple and cheap solution. Rest assured, it can be -to some extent- but here are 3 gotchas I learned the hard way:

1. mod_cache ignores Cache-control if Expires is in the past (which it shouldn’t according to RFC2616), so you might have to unset the Expires-header.
2. mod_cache by default caches cookies! Let me repeat; cookies are cached! That might be a huge security-disaster waiting to happen; sessionid’s (that provide access for logged-on users) are generally stored in cookies. If a logged on user that request an uncached page, then that user’s cookie will get cached and sent to other users that request the same page. Do disable this by adding “CacheIgnoreHeader Set-Cookie” to your config
3. mod_cache by default treats all browsers like the one that triggered the caching of the object. In the field that approach can cause problems with e.g. CSS-files that are stored gzipped (because the first browser requested with header “Accept-Encoding: gzip, deflate”). If a browser that does not support gzipped content requests the same file, the CSS will be unreadable and thus not applied. The solution; make sure the “backend webserver” sends the “Vary: Accept-Encoding” header in the response (esp. for CSS-files). This will tell mod_cache to take different Accept-Encodings into account, storing and sending different versions of the same CSS-file.

For a proof of concept I was preparing at work I needed a jQuery templating solution. Although there is beta templating support (contributed by Microsoft) in jQuery, I decided to implement jQote2 instead. This alternative jQuery plugin is small (3,2Kb minimized, 1,7Kb compressed), versatile and most importantly very, very fast!

So what do you need to know about jQote2 to get it working? Well, there’s 3 ingredients; data, template and javascript-code to put the data in the template.

The data can be fetched from an external source, e.g. this call to the iRail-api for departures from Brussels North.

The template is basically just HTML with some placeholders for your data:

<script type="text/x-jqote-template" id="liveboard_tmpl">
 <tr>
  <td class="left">
   <%= this.station %>
  </td>
  <td class="right">
   <%= this.time %>
  </td>
  <td class="right">
   <%= this.platform %>
  </td>
 </tr>
</script>

The javascript fetches the data using jQuery’s getJson, parses all departures in the template and adds the resulting HTML to an element in your DOM (in this case #liveboard’):

<script type="text/javascript">
$(document).ready(
	function() {
		$.getJSON(
			'http://api.irail.be/liveboard/?format=json&station=Brussel%20Noord&lang=EN&arrdep=DEP&callback=?',
			function(data) {
					$('#liveboard').jqoteapp('#liveboard_tmpl', data.departures.departure);
			}
		)
	}
);
</script>

Off course the UNIX-timestamp in this.time isn’t really usable, but we can easily add some javascript to the template, just before outputting the time, to fix that;

<% this.time=((new Date((Number(this.time))*1000)).toLocaleTimeString()).substr(0,5); %>

That’s right, use “<%” instead of “<%=” and you can mingle javascript in the template. To only show trains that have not left and to show departures including delay, the template looks like this:

<script type="text/x-jqote-template" id="liveboard_tmpl">
<% if (this.left!="1") { %>
 <tr>
  <td class="left">
   <%= this.station %>
  </td>
  <td class="right">
   <% if (this.delay!="0") {
    this.time="<span class=\"delayed\">"+((new Date((Number(this.time)+Number(this.delay))*1000)).toLocaleTimeString()).substr(0,5)+"</span>";
   } else {
    this.time=((new Date((Number(this.time))*1000)).toLocaleTimeString()).substr(0,5);
   } %>
   <%= this.time %>
  </td>
  <td class="right">
   <%= this.platform %>
  </td>
 </tr>
<% }; %>
</script>

Add some CSS and you’ll quickly have something like the demo you can find here. Just look at the code, it’s pretty straightforward and check out the jQote2 reference for even more info.

Damn, Venus doesn’t love noscript!

You’ve got no clue what I’m rambling about, do you? Well, allow me to explain;

• <noscript> is the html tag that identifies alternative content for user agents (browsers, mainly) that don’t or won’t do javascript
• Venus is a planet-like web application that aggregates rss- and atom feeds from a community or about a specific subject (see e.g. planet mozilla, planet debian and the much-loved planet grep)

So now you know the context, let me reiterate; Venus doesn’t treat noscript the way it should! It not only strips out javascript as it should (are you listening tt-rss?) but it replaces noscript-tags and all HTML inside with escaped HTML (with HTML-entities actually). And that, my beloved ones, means that the HTML that WP YouTube Lyte generates, doesn’t work properly on Venus-based planets.

So I started looking at the Venus source and mailed with Planet Grep’s Wouter Verhelst to solve this issue. At first sight the solution seemed pretty straightforward; Venus shouldn’t ‘escape’ noscript but should instead just strip the opening and closing noscript-tag. Wouter installed a small sed-filter I wrote and added noscript to the whitelist of Venus’s sanitizer (which is based on Universal Feed Parser) and … it did not work.

The problem apperantly is with another sanitizing component in Venus; html5lib. Sam Ruby, the developer of Venus, wrote on the mailinglist;

There are multiple sanitization passes involved here. [...] The html5parser seems to think that noscript is to be parsed as text only, which would result in the behavior that you describe.  Looking at the current HTML5 spec, it appears that this does not match the expected behavior — so perhaps that changed too.

So I started looking at html5lib and … well, I’m stuck, html5lib is a pretty complex beast for a smalltime non-developer to dive into. So earlier today I turned to the html5lib discussion list to ask how sanitization can be configured not to escape noscript, let’s hope someone will enlighten me. Because until then those poor Planet Greppers won’t be able to see (a thumbnail of) Al Jarreau’s great version of Take Five way back in 1976:

Suppose you’re setting up a Drupal-based site for which you have to implement a caching reverse proxy and for reasons beyond your comprehension Varnish (or even Squid) are not an option. Oh no, you’re stuck with Apache’s mod_proxy and mod_cache! What should you do?

First of all, Drupal 6 doesn’t like reverse proxies. If you don’t want to wait for version 7, which should do better in this respect, you might want to look at Pressflow. This Drupal 6 “distro” has everything on board to work with reverse proxies. So install Pressflow (or try to apply this out of date diff to stock Drupal) and in the Performance-screen set “Caching Mode” to “External” and “Page Cache Maximum Age” to the number of minutes you consider a cached page valid. Voila, you’re done in Drupal (edit: almost, as you might also want to change the $base_url in sites/default/settings.php to reverse proxy URL after you configured Apache).

Next up: Apache! A simple configuration like this one should do the trick:

ProxyRequests Off
ProxyPass /rp_drupal http://localhost/pressflow
ProxyPassReverse /rp_drupal http://localhost/pressflow
CacheEnable disk /rp_drupal/
CacheRoot c:/TEMP/apacache
CacheDefaultExpire 3600

OK, this must surely work, no? Well it should, but it doesn’t! When setting your Apache-loglevel to debug you’ll see “not cached” entries in your error-log, with the following reason:

Expires header already expired, not cacheable

Expires in the past, what does Pressflow think it’s doing deep down in includes/bootstrap.inc?

// HTTP/1.0 proxies do not support the Vary header, so prevent any caching
// by sending an Expires date in the past. HTTP/1.1 clients ignores the
// Expires header if a Cache-Control: max-age= directive is specified (see RFC
// 2616, section 14.9.3).
drupal_set_header('Expires', 'Sun, 11 Mar 1984 12:00:00 GMT');
// [...]
$max_age = variable_get('cache', CACHE_DISABLED) == CACHE_AGGRESSIVE && (!isset($_COOKIE[session_name()]) || isset($hook_boot_headers['vary'])) ? variable_get('page_cache_max_age', 0) : 0;
$default_headers['Cache-Control'] = 'public, max-age=' . $max_age;

Darn, those Pressflow-guys seem to have read up on their RFC’s! And indeed, 2616 confirms that cache-control’s max-age overrules expires;

If a response includes both an Expires header and a max-age directive, the max-age directive overrides the Expires header, even if the Expires header is more restrictive. This rule allows an origin server to provide, for a given response, a longer expiration time to an HTTP/1.1 (or later) cache than to an HTTP/1.0 cache.

Mod_cache’s code seems to take a much simpler approach; at line 503 it decides not to cache based on an Expires-header in the past, totally dismissing the potential presence of cache-control’s max-age.

else if (exp != APR_DATE_BAD && exp < r->request_time)
    {
        /* if a Expires header is in the past, don't cache it */
        reason = "Expires header already expired, not cacheable";
    }

But you’re not interested in code which does or does not adhere to whatever RFC some spec-buffs came up with, you just want to cache your frigging’ Drupal-site! Well, fear not little hacker-boy, here’s some Apache-magic to cure your ailments, to be copy/pasted in the config before ProxyPass and ProxyPassReverse:

<Location /rp_drupal>
     SetEnvIf Request_Protocol "HTTP/1.1" expires_overrule
     # homework: add a SetEnvIf to see if cache-control max-age is present
     Header unset Expires env=expires_overrule
</Location>

So there you have it, a rudimentary caching setup for Drupal (in the guise of Pressflow) using nothing but Apache’s mod_proxy and mod_cache. Now go do your homework and test and do some finetuning and test some more. Happy caching!

Did your HTC Hero get lost somewhere along the way, unable to fix GPS-location after upgrading to VillainRom, FroydVillan or another Hero ROM? The solution, which is mentioned in the hilarious release-notes for FroydVillain 1.5, is as simple as it is obscure; go to Settings -> Wireless & Networks -> Mobile networtks -> Access Point Names, click on the selected APN there, go to APN type (the last item in the list) and change the value to “default,supl”. I rebooted (with GPS on, didn’t want to jinx things) and voila, I’ll never get lost again.

I’ve been experimenting with the PHP OAuth PECL extension over the last few days and initially ran into some small problems getting it to function correctly. So for the sake of making this world wide web an even better place, here are some error-messages you might encounter and what you could do to resolve them:

• “OAuth::getRequestToken() URL file-access is disabled in the server configuration” → the OAuth extension by default uses fopen (streams) to fetch data from the OAuth provider (server), but you’re probably slightly paranoid (or is it security- conscious?) and have allow_url_fopen = 0 in you php.ini. Just change that value to “1″ and reload your webserver and then see …
• “SSL: fatal protocol error“ → OAuth seems to be working fine, but for reasons unknown OAuth insists something fatal just happened. You could ignore this one, or try to lower the error-reporting value or do as I did and decide to use OAuth::setRequestEngine to switch from those doomed streams to almighty Curl, until …
• “setRequestEngine expects parameter to be long, string given” → so you installed (compiled) the OAuth-extension, but you didn’t have that libcurl-dev package installed, did you? No matter how polite you request that Curl-engine, it is just not going to happen unless you install libcurl-dev, and then re-installed OAuth.

And after successfully re-installing OAuth, this time with Curl-support, you could try to build a small application, accessing GMail maybe?