Secure your smartphone

Your smartphone probably contains a wealth of information of personal and professional nature, which you would not want others to have access to. This is why (after losing my HTC Hero a couple of months ago) I now try to follow 2 out of these 3 simple rules:

  1. don’t lose your smartphone.
  2. if you lose your smartphone, make sure you have something in place to locate it
  3. if you lose your smartphone and you can’t locate it, make sure you can wipe it remotely

There are multiple solutions to locate & wipe smartphones (including HTC’s Sense online offering), but for my Sense-less HTC Magic I installed “Lookout“. Lookout is a free application that provides device location, contacts backup & restore and apparently also malware protection. If you’re willing to pay $3/month, you also get remote wipe, remote lock and backup/ restore of pictures and call log. If you lose your Android-phone, you just log in to the Lookout-website to locate and optionally lock or wipe your handset.
I’m happy using the free version for now; I activated Android’s pattern lock-screen to avoid anyone from accessing my handset and deactivating Lookout. Remote wipe is great, but I guess I can activate my Lookout Premium account if ever I need that feature?

How to buy, upgrade, brick, rescue and generally enjoy a HTC Magic in just 14 days

Step 1: Buy
So you’re not happy with your cheapo rebound phone, pining for your lost HTC Hero and you start checking out bargain-sites for a good 2nd hand Android smartphone. After a week or so you spot an HTC Magic, on sale for €100 and 2 days later go buy that beauty for even €10 less because hey, there’s no SD-card.
Step 2: Upgrade
Your brand new Magic turns out to be that very first Android-phone Proximus started sellinng in May 2009, with Android 1.5, but without HTC Sense and tethering. Not really the smartphone you’d settle for, so you start looking around xda-developers for an upgrade path. You install flashrec, flash a new recovery image and in recovery flash MyHero 2.0.5 and after 1 month without it, you can finally boot into that beautiful HTC Sense UI again.
Step 3: Brick
HTC Sense, great, but still Android 1.5 and no tethering, seriously? No can do Sir, so you head back to xda-developers to figure out your next step. Late at night, after browsing millions of forum posts about perfected SPL’s, goldcards and recovery images, you find a thread with links to official HTC RUU’s. Easy-peasy and you download one of those boot your Windows PC and start flashing. HBOOT updates, radio updates, … all goes well and you doze off for a minute. But when you open your eyes, the upgrade process halted and you have a white screen with “invalid Customer ID” in red and no Android. You reboot, no go. You try to enter recovery mode, no go. Congratulations, you now have a shiny (semi-)bricked HTC Magic and you go to sleep feeling an utter moron for trying to flash an official RUU.
Step 4: Rescue
The next day you start looking for information on the secret craft of goldcard creation. You spend a couple of days trying to get your SD-card’s CID on your PC, but eventually ask a colleague to adb-shell into his device with your SD-card in it to get the job done (thanks Thomas!). You don’t bother downloading crypto-software to reverse the string for all the wrong reason, instead immediately heading over to the goldcard-manufacturing-webstie, write the disk-image to SD and you try to flash the RUU with the goldcard you just created. Damn, no go! You reverse the string manually, no go. You buy a new SD card (4Gb Sandisk), adb-shelling into your own cheapo Acer this time to get the CID and create a new goldcard, no go. Over a week goes by and you decide to have another stab at it and opnly then you see that the string should be hex-reversed, not reversed. You click the link to an online hexreverser, create a goldcard with that string and bingo, the RUU flashes!
cyanogenmodStep 5: Enjoy
It looks like you’re back where you were at step 2; Android 1.5 with HTC Sense UI and no tethering, so you decide to install Cyanogenmod by first downgrading your radio & hboot and then -finally- flashing Cyanogen’s Android mod.
And there you have it, after only 2 weeks you successfully turned that old HTC Magic into a modern, fast and reliable Android smartphone. Android 2.2.1 that is, with ADWlauncher, tethering and Exchange-integration. Time well spent, except … Vodaphone has an official Froyo update for the Magic out as well and there’s already a tweaked ROM for it on xda-developers. You really should try that one out as soon as possible, now shouldn’t you?!

Splitting up a vcard-file

As my Acer e110 doesn’t sync with Google, all of my precious contacts in the cloud did not automagically appear on my handset. That left me little but no choice to go the old-fashioned way; the export/import-dance.
Exporting from Google is easy, but it generates one vcard-file with all you contacts in it, which the contacts app in Android 1.5 can only import the first entry from. To split up the contacts-file, Scroogle pointed me vCard list-file splitter, vcf-split for short, a Perl script from back in the days when Windows linebreaks apparently were sufficient evidence of the end of a vcard. But times and technology have changed and linebreaks have lost their former glory, meaning I had to slightly alter the script to watch out for a cryptic “END:VCARD” line to indicate the end of a vcard.
And the script goes a little something like this:

#!/usr/bin/perl
#
# vcf-split - split a .list.vcf file into many small .vcf files.
# Copyright (C) 2004  Raphael J. Schmid.
# Tweaked by Frank Goossens ("futtta") in 2011
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 1, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
#
# -- raphael.schmid@gmx.de
# -- futtta@gmail.com
use File::Basename;
if ($ARGV[0] eq "") {
  print "Usage: vcard-split <file to split>\n\n";
  exit;
}
$input=$ARGV[0];
$counter=0;
print $input;
open INPUT, $input or die $!;
while (<INPUT>) {
  open OUTPUT, ">> ".$counter."-".basename($input) or die $!;
  if ($_ =~ /END:VCARD/ ) {
    print OUTPUT $_;
    $counter+=1;
    close OUTPUT;
  } else {
    print OUTPUT $_;
  }
}
close OUTPUT;
close INPUT;

Who knows one day Google will send someone this way who has some vcf-splitting to do?

3 Apache mod_cache gotchas

If you want to avoid the learning curve of Squid and Varnish or the cost of a dedicated caching & proxying appliance, using Apache with mod_cache may seem like a good, simple and cheap solution. Rest assured, it can be -to some extent- but here are 3 gotchas I learned the hard way:

  1. mod_cache ignores Cache-control if Expires is in the past (which it shouldn’t according to RFC2616), so you might have to unset the Expires-header.
  2. mod_cache by default caches cookies! Let me repeat; cookies are cached! That might be a huge security-disaster waiting to happen; sessionid’s (that provide access for logged-on users) are generally stored in cookies. If a logged on user that request an uncached page, then that user’s cookie will get cached and sent to other users that request the same page. Do disable this by adding “CacheIgnoreHeaders Set-Cookie” to your config
  3. mod_cache by default treats all browsers like the one that triggered the caching of the object. In the field that approach can cause problems with e.g. CSS-files that are stored gzipped (because the first browser requested with header “Accept-Encoding: gzip, deflate”). If a browser that does not support gzipped content requests the same file, the CSS will be unreadable and thus not applied. The solution; make sure the “backend webserver” sends the “Vary: Accept-Encoding” header in the response (esp. for CSS-files). This will tell mod_cache to take different Accept-Encodings into account, storing and sending different versions of the same CSS-file.

How to do jQuery templates with jQote2

For a proof of concept I was preparing at work I needed a jQuery templating solution. Although there is beta templating support (contributed by Microsoft) in jQuery, I decided to implement jQote2 instead. This alternative jQuery plugin is small (3,2Kb minimized, 1,7Kb compressed), versatile and most importantly very, very fast!
So what do you need to know about jQote2 to get it working? Well, there’s 3 ingredients; data, template and javascript-code to put the data in the template.
The data can be fetched from an external source, e.g. this call to the iRail-api for departures from Brussels North.
The template is basically just HTML with some placeholders for your data:

<script type="text/x-jqote-template" id="liveboard_tmpl">
 <tr>
  <td class="left">
   <%= this.station %>
  </td>
  <td class="right">
   <%= this.time %>
  </td>
  <td class="right">
   <%= this.platform %>
  </td>
 </tr>
</script>

The javascript fetches the data using jQuery’s getJson, parses all departures in the template and adds the resulting HTML to an element in your DOM (in this case #liveboard’):

<script type="text/javascript">
$(document).ready(
	function() {
		$.getJSON(
			'http://api.irail.be/liveboard/?format=json&station=Brussel%20Noord&lang=EN&arrdep=DEP&callback=?',
			function(data) {
					$('#liveboard').jqoteapp('#liveboard_tmpl', data.departures.departure);
			}
		)
	}
);
</script>

Off course the UNIX-timestamp in this.time isn’t really usable, but we can easily add some javascript to the template, just before outputting the time, to fix that;

<% this.time=((new Date((Number(this.time))*1000)).toLocaleTimeString()).substr(0,5); %>

That’s right, use “<%” instead of “<%=” and you can mingle javascript in the template. To only show trains that have not left and to show departures including delay, the template looks like this:

<script type="text/x-jqote-template" id="liveboard_tmpl">
<% if (this.left!="1") { %>
 <tr>
  <td class="left">
   <%= this.station %>
  </td>
  <td class="right">
   <% if (this.delay!="0") {
    this.time="<span class=\"delayed\">"+((new Date((Number(this.time)+Number(this.delay))*1000)).toLocaleTimeString()).substr(0,5)+"</span>";
   } else {
    this.time=((new Date((Number(this.time))*1000)).toLocaleTimeString()).substr(0,5);
   } %>
   <%= this.time %>
  </td>
  <td class="right">
   <%= this.platform %>
  </td>
 </tr>
<% }; %>
</script>

Add some CSS and you’ll quickly have something like the demo you can find here. Just look at the code, it’s pretty straightforward and check out the jQote2 reference for even more info.

Venus doesn’t love noscript

Damn, Venus doesn’t love noscript!
You’ve got no clue what I’m rambling about, do you? Well, allow me to explain;

So now you know the context, let me reiterate; Venus doesn’t treat noscript the way it should! It not only strips out javascript as it should (are you listening tt-rss?) but it replaces noscript-tags and all HTML inside with escaped HTML (with HTML-entities actually). And that, my beloved ones, means that the HTML that WP YouTube Lyte generates, doesn’t work properly on Venus-based planets.
So I started looking at the Venus source and mailed with Planet Grep’s Wouter Verhelst to solve this issue. At first sight the solution seemed pretty straightforward; Venus shouldn’t ‘escape’ noscript but should instead just strip the opening and closing noscript-tag. Wouter installed a small sed-filter I wrote and added noscript to the whitelist of Venus’s sanitizer (which is based on Universal Feed Parser) and … it did not work.
The problem apperantly is with another sanitizing component in Venus; html5lib. Sam Ruby, the developer of Venus, wrote on the mailinglist;

There are multiple sanitization passes involved here. […] The html5parser seems to think that noscript is to be parsed as text only, which would result in the behavior that you describe.  Looking at the current HTML5 spec, it appears that this does not match the expected behavior — so perhaps that changed too.

So I started looking at html5lib and … well, I’m stuck, html5lib is a pretty complex beast for a smalltime non-developer to dive into. So earlier today I turned to the html5lib discussion list to ask how sanitization can be configured not to escape noscript, let’s hope someone will enlighten me. Because until then those poor Planet Greppers won’t be able to see (a thumbnail of) Al Jarreau’s great version of Take Five way back in 1976:

Al Jarreau 1976 -Take Five

Drupal, mod_cache & RFC2616 caching

Suppose you’re setting up a Drupal-based site for which you have to implement a caching reverse proxy and for reasons beyond your comprehension Varnish (or even Squid) are not an option. Oh no, you’re stuck with Apache’s mod_proxy and mod_cache! What should you do?
First of all, Drupal 6 doesn’t like reverse proxies. If you don’t want to wait for version 7, which should do better in this respect, you might want to look at Pressflow. This Drupal 6 “distro” has everything on board to work with reverse proxies. So install Pressflow (or try to apply this out of date diff to stock Drupal) and in the Performance-screen set “Caching Mode” to “External” and “Page Cache Maximum Age” to the number of minutes you consider a cached page valid. Voila, you’re done in Drupal (edit: almost, as you might also want to change the $base_url in sites/default/settings.php to reverse proxy URL after you configured Apache).
Next up: Apache! A simple configuration like this one should do the trick:

ProxyRequests Off
ProxyPass /rp_drupal http://localhost/pressflow
ProxyPassReverse /rp_drupal http://localhost/pressflow
CacheEnable disk /rp_drupal/
CacheRoot c:/TEMP/apacache
CacheDefaultExpire 3600

OK, this must surely work, no? Well it should, but it doesn’t! When setting your Apache-loglevel to debug you’ll see “not cached” entries in your error-log, with the following reason:

Expires header already expired, not cacheable

Expires in the past, what does Pressflow think it’s doing deep down in includes/bootstrap.inc?

// HTTP/1.0 proxies do not support the Vary header, so prevent any caching
// by sending an Expires date in the past. HTTP/1.1 clients ignores the
// Expires header if a Cache-Control: max-age= directive is specified (see RFC
// 2616, section 14.9.3).
drupal_set_header('Expires', 'Sun, 11 Mar 1984 12:00:00 GMT');
// [...]
$max_age = variable_get('cache', CACHE_DISABLED) == CACHE_AGGRESSIVE && (!isset($_COOKIE[session_name()]) || isset($hook_boot_headers['vary'])) ? variable_get('page_cache_max_age', 0) : 0;
$default_headers['Cache-Control'] = 'public, max-age=' . $max_age;

Darn, those Pressflow-guys seem to have read up on their RFC’s! And indeed, 2616 confirms that cache-control’s max-age overrules expires;

If a response includes both an Expires header and a max-age directive, the max-age directive overrides the Expires header, even if the Expires header is more restrictive. This rule allows an origin server to provide, for a given response, a longer expiration time to an HTTP/1.1 (or later) cache than to an HTTP/1.0 cache.

Mod_cache’s code seems to take a much simpler approach; at line 503 it decides not to cache based on an Expires-header in the past, totally dismissing the potential presence of cache-control’s max-age.

else if (exp != APR_DATE_BAD && exp < r->request_time)
    {
        /* if a Expires header is in the past, don't cache it */
        reason = "Expires header already expired, not cacheable";
    }

But you’re not interested in code which does or does not adhere to whatever RFC some spec-buffs came up with, you just want to cache your frigging’ Drupal-site! Well, fear not little hacker-boy, here’s some Apache-magic to cure your ailments, to be copy/pasted in the config before ProxyPass and ProxyPassReverse:

<Location /rp_drupal>
     SetEnvIf Request_Protocol "HTTP/1.1" expires_overrule
     # homework: add a SetEnvIf to see if cache-control max-age is present
     Header unset Expires env=expires_overrule
</Location>

So there you have it, a rudimentary caching setup for Drupal (in the guise of Pressflow) using nothing but Apache’s mod_proxy and mod_cache. Now go do your homework and test and do some finetuning and test some more. Happy caching!

Fixed GPS-location in VillainRom

Did your HTC Hero get lost somewhere along the way, unable to fix GPS-location after upgrading to VillainRom, FroydVillan or another Hero ROM? The solution, which is mentioned in the hilarious release-notes for FroydVillain 1.5, is as simple as it is obscure; go to Settings -> Wireless & Networks -> Mobile networtks -> Access Point Names, click on the selected APN there, go to APN type (the last item in the list) and change the value to “default,supl”. I rebooted (with GPS on, didn’t want to jinx things) and voila, I’ll never get lost again.

PHP OAuth extension: trial, error and success

I’ve been experimenting with the PHP OAuth PECL extension over the last few days and initially ran into some small problems getting it to function correctly. So for the sake of making this world wide web an even better place, here are some error-messages you might encounter and what you could do to resolve them:

  • “OAuth::getRequestToken() URL file-access is disabled in the server configuration” → the OAuth extension by default uses fopen (streams) to fetch data from the OAuth provider (server), but you’re probably slightly paranoid (or is it security- conscious?) and have allow_url_fopen = 0 in you php.ini. Just change that value to “1” and reload your webserver and then see …
  • SSL: fatal protocol error → OAuth seems to be working fine, but for reasons unknown OAuth insists something fatal just happened. You could ignore this one, or try to lower the error-reporting value or do as I did and decide to use OAuth::setRequestEngine to switch from those doomed streams to almighty Curl, until …
  • “setRequestEngine expects parameter to be long, string given” → so you installed (compiled) the OAuth-extension, but you didn’t have that libcurl-dev package installed, did you? No matter how polite you request that Curl-engine, it is just not going to happen unless you install libcurl-dev, and then re-installed OAuth.

And after successfully re-installing OAuth, this time with Curl-support, you could try to build a small application, accessing GMail maybe?

Eclair on HTC Hero: going rogue with Villainrom

I flashed my HTC Hero again, this time with an Android 2.1 image from Villainrom. Why not wait for the official update? Well, we’ve been waiting for quite some time now, the HTC-update might not even work on a rooted device and a colleague of mine was running Villainrom 10.1 for over a month now and was quite pleased with it.
So if you want to go rogue as well, go Villainrom using this installation guide. And in case things happen that aren’t described in that detailed howto, here are 5 things I had to learn the hard way;

  1. Formatting your SDCARD when on a Modaco rom might not work, having a colleague nearby who can format for you on his handset or PC might help.
  2. If flashrec complains “Could not run command” but your device is already rooted then forget about flashrec, you’re a superuser now
  3. nandroid+ext might not work, just perform a normal nandroid backup
  4. market downloads sometimes don’t work, in my case this was solved by resetting privacy and location-sharing settings to their (permissive) default settings
  5. Villainrom provides OTA-updates, but these aren’t always applied, re-downloading and re-applying the update did the trick for me.

My first Android 2.1 impressions; clearly snappier (and tests show 2.2 to be a lot faster still), some great apps on the market that weren’t there for 1.5 (Google Goggles looks great!) and most importantly; one of the best mobile browsers around (as confirmed by Quirksmode Webkit tests). So yes, I love my Eclair!