Blank Yoast SEO sitemaps no more!

yoast_thumbsIf you’re using both Autoptimize and WordPress SEO by Yoast, you might have noticed blank sitemaps in your browser. With the help of Vance Hallman and Armand Hadife, I have been able to isolate and fix this rather interesting bug.
I pretty soon saw that It actually wasn’t the sitemap XML, but the XSL (XML stylesheets, browsers use those to make XML readable) that was wrecking havoc. This means that although you couldn’t see them in your browser, the sitemaps themselves were not broken and could still be used by search engines.
But then why was the XSL broken? Well, Yoast’s WordPress SEO generates the sitemap XML and XSL on the fly, hooking into WordPress’ template_redirect action, according to the source-code to “Hijack requests for potential sitemaps and XSL files”. Autoptimize hooks into the same action, finds JavaScript in the XSL (jQuery tablesorter actually) to optimize, and injects the minimized script with a boolean defer-attribute back in the HTML. The problem; it’s not HTML, it’s XSL with HTML in it. And a defer attribute on a script-tag is not valid XML, so the XSL does not validate so the browser can not render the sitemap, resulting in the blank page you might have seen.
Although fiddling with the defer-attribute could solve the issue, I think it’s not a good idea for Autoptimize to try to optimize non-HTML resources as it isn’t WordPress’ core functionality either, so I changed Autoptimize to not act on non-HTML content (i.e. without an HTML-tag or with an xsl:stylesheet-tag). This change is in the upcoming 1.7.3 release, it will be pushed live tomorrow.

WP default themes 2010-2014: performance going down

Although it is easy to download and install free or “premium” themes, WordPress does come with a default theme that is updated yearly. Last year I did performance tests of Twenty Twelve and there were some performance-issues, which led me to create a cleary faster child theme (2012.FFWD). But how does Twenty Thirteen fare, you might wonder? And how did the WordPress themes do before? And what about Twenty Fourteen? Just the questions I was asking myself as well, so here goes!
I decided to compare the raw performance of the default WordPress themes from 2010 to 2014, creating new blogs in my multi-site test-environment. All 5 test-blogs’ homepages were tested 9 times with webpagetest.org, using the Amsterdam node, IE9 as browser and with a DSL traffic shaping profile and the median test result was used. No performance enhancing substances (such as WP Super Cache or Autoptimize) were used in this test, a few bunnies might have been slightly injured though … The result? Well, Just look at this graph;
wordpress default theme 2010-2014 performance
As you can see performance (until document loaded) got worse with every release. Download time skyrocketed from 1.7s to 4.2s (and even 5.3s the upcoming 2014), mostly because download size went from 60Kb for Twenty Ten to a whopping 489Kb for Twenty Thirteen (and 659Kb for the non-final Twenty Fourteen). All details of the tests can be found in this Google Docs spreadsheet.
So what is the reason for this important performance degradation? Web Fat, that’s what. In 2013 jQuery & co were added (there was already one smallish JS-file in 2012), but even more damaging is the explosion in font-file downloads; 4 in 2012, 9 in 2013 and 15 in 2014! I’ve already expressed my dislike for webfonts and -although I think it can be immensely useful and I use it for the admin pages of my plugins- I think one should also try to avoid including jQuery where possible, especially for content-oriented sites such as the ones powered by WordPress? Unless you don’t care about performance off course.
(In case you’re wondering; the use WP Super Cache and Autoptimize or similar solutions will clearly improve performance, but not having JavaScript is better then having to optimize JavaScript and font-files cannot really be optimized, so the problem will be less but will remain.)

Do not donate to me!

Some people ask me if they can donate for my software. My answer invariably is they can’t, because I don’t want money. Instead I would like them to make a small donation to a good cause!
Why? Well, I’m a very lucky guy, living in the richest region of one of the richest countries of the world and having a full-time job that allows me and my family to live a comfortable life. Software is “just a hobby” which I hope is a small contribution to make the web (and by extension the world) a slightly better place.
So given that context and given the fact that there are many ways in which the world could be made a better place, I would like to ask you to donate any amount of money you think my little projects are worth to a good cause. Just pick one, click and donate!

  • Donate via GiveDirectly which helps people living in extreme poverty by making unconditional cash transfers to them via mobile phone.
  • Lend via Kiva, or send me a Kiva-voucher and I will lend with that. Kiva allows people to lend money via the Internet to low-income / underserved entrepreneurs and students in 70 countries.
  • Support La Quadrature du Net, which is one of Europe’s leading organizations promoting net neutrality, a positive reform of copyright and the protection of privacy (these guys were instrumental in defeating ACTA).
  • Mary’s Meals provides life-changing meals to some of the world’s poorest children every school day.
  • The Mozilla Foundation is responsible for Firefox and for the Mozilla Developer Network documentation, keeping the web open en other browsers honsest. Donate to them!

And there are many, many more great causes both locally and internationally that can use our help!

I want you for Autoptimize 1.7.0 (and beyond)!

I started work on Autoptimize 1.7.0, working my way through this feature list:

  • update CSS & JS optimization components to most recent versions
  • add CSS exclusion
  • remove legacy CDN & YUI support, replacing CDN with something simpler
  • clear cache of w3tc, hypercache, fast cache, faster cache, … when flushing AO cache
  • better layout of the admin-page
  • update translations

The goal is to release a first test-version in October (maybe as early as next week, things are progressing pretty well) and to push the final version (which might include one extra feature which could be kind of a killer one) out in November.
So why do I need help?

  1. For testing the subsequent “beta”-releases (based on trunk)
  2. To provide translations (based on a POT-file, using a gettext-compatible tool).

If you’re interested to help on this release (or, more in general, would like to support other users or co-develop future releases) hop to this wordpress.org support forum thread and give us a shout-out.
[cross-posted from Autoptimize support forum on wordpress.org]

Autoptimize; JavaScript in head and some deprecated functionality

Autoptimize 1.6.5 got just pushed out, with one new feature and one notification. The new option configures Autoptimize to output the aggregated JavaScript in the head-section instead of at the bottom of the HTML. This can be useful when some of the JavaScript needs to be loaded asap and might prove useful to make jQuery plugins behave.
The notification might be less welcome for some users; YUI compression and the CDN options are marked as deprecated in this release. YUI compression was pretty exotic and required the installation of JAVA and the YUI compression jar, so I doubt anyone was actually using this. CDN options are deprecated as well; some people reported issue that I could not reproduce or fix. As I consider CDN not to be core functionality and as it can better be accomplished using e.g. WP Super Cache (which is a must-have companion of Autoptimize anyhow), CDN will indeed also be removed from Autoptimize. Expect the deprecated parts to be removed in 1.7.0 (which isn’t planned yet).

WP Caching plugin vulnerability debrief

Now that both WP Super Cache and W3 Total Cache developers have released a new version of their respective plugins (upgrade first, continue reading after) it seems time for a small “post mortem“.
The problem was in the interpretation of dynamic snippets, that are contained inside a number of specific HTML-comment tags. These snippets allow both plugins (and their predecessor WP Cache) to cache pages while keeping a limited amount of dynamic, PHP-generated content in them that can be executed on the fly. Think ESI in e.g. Varnish.
The vulnerability, which was originally discovered by kisscsaby and reported 3 weeks ago on the wordpress.org plugins support forum, had multiple causes:

  1. Unlike ESI’s, dynamic snippets can not only be includes (mclude) but also PHP-code (mfunc). Whereas one could consider includes of known files more or less safe, inclusion of PHP-code introduces a risk.
  2. As WP Super Cache & W3 Total Cache keep entire pages in cache and as pages can contain comments, that user generated content is parsed for dynamic snippets as well.
  3. WordPress core by default only allows a limited set of HTML in comments (“a blockquote code em strong ul ol li”), but it also leaves HTML comments in place.

As a result, blogs with WP Super Cache (before version 1.3) and W3 Total Cache (before version 0.9.2.9) were at risk of PHP code injection. Blog comments could contain dynamic snippets (in HTML-comments) and WordPress core did not them filter out. Upon a such a malicious comment having been submitted, a new cached version of the page was created that included the injected PHP-code. Upon the first request of the cached page, that code was successfully executed.
I stumbled on the vulnerability report about a week and a half ago, while researching why dynamic snippets weren’t executing when Autoptimize was active (simple really, Autoptimize by default removes HTML comments, the upcoming 1.6.3 will leave mfunc/mclude in place). As this seemed like a pretty severe security hole and as there was no feedback from developers in the support thread, I created a small “stopgap plugin” to mitigate the threat on April 10th, mailed security@wordpress.org and plugins@wordpress.org and requested WP Safer Cache being published on wordpress.org on the 11th. A couple of hours later WP Super Cache’s Donncha O Caoimh contacted me and the same day he released a version (1.3) that fixed this vulnerability by parsing out potential exploits from comments as they are posted and as they are rendered. On April 12th W3 Total Cache’s Frederick Townes confirmed they were working on a fix. Version 0.9.2.9 got released on April 17th, disabling dynamic snippets by default and when these are enabled, they require a secret alphanumeric key to be included in the snippet which is checked against one that is defined in wp-config.php.
Conclusions; The fact that this didn’t generate any fuss (as opposed to W3 Total Cache’s widely published information disclosure vulnerability in December 2012) is surprising. PHP Code injection clearly is a more severe security risk that must have been there for a long time already. The fact that this only got discovered recently is baffling. And why WordPress core doesn’t filter out HTML-comments from submitted blog comments, others seem to understand, but to me that remains the biggest mystery of all.

Easter Bunny special: Autoptimize 1.6.0

Not sure if April Fools or the Easter Bunny are to blame, but Autoptimize 1.6.0 just got pushed out the gates.
The main new features:

  • You can now specify which JavaScript should not be Autoptimized. Just add the names (or part of the path) of the scripts in a comma-separated list and that JavaScript-file will remain untouched by Autoptimize.
  • Added support for ETag and LastModified (essentially for a better pagespeed score, as the files were already explicitly marked as cacheable for 1 year).
  • Autoptimizing for logged in users is enabled again.
  • Misc. bug fixes (see changelog)

As always; leave a reply on this post or create a new topic on the support forum if you encounter any problem.

Adopting an OSS-orphan: Autoptimize

I’m taking over support for the venerable Autoptimize WordPress plugin. Although I started out using CSS-JS-booster first, I switched to Autoptimize for my JS & CSS aggregating and minimizing pleasure somewhere in 2012, not in the least because it was the only plugin to offer conversion of background-images to data-uri’s. I hadn’t noticed back then that Autoptimize was already pretty old and that the developer announced he didn’t use WordPress any more and that he had lost most motivation. Fast-forward to December, when, while working on a Twenty Twelve child theme, I noticed that Autoptimize messed Twenty Twelve up severely. So I dug in, found some problems and fixed some others while I was at it:

  • leave html5.js be; aggregating it breaks HTML5-support in older IE versions
  • make sure the IE-specific CSS-files are loaded after the normal aggregated CSS
  • ensure both JPG and JPEG files are taken into account for conversion of background-images to data-uri’s
  • correct a bug that resulted in aggregated files not having a hash in them (having one or more autoptimize_.php can break things)
  • stop autoptimizing for logged in users (which broke the WordPress 3.5 admin bar again)

Based on feedback on the WordPress support forums, others were still using Autoptimize as well, needing bug-fixes and support so I contacted Turl a couple of days ago and proposed that I join his one-man team. He agreed, so I’ll be taking over Autoptimize as of now. The first update (1.4.1) with the fixes listed above will be pushed to SVN soon. I’ll provide support on the wordpress.org forums as well and release new bugfix-versions if needed. New features or other major updates however, are not on the roadmap (yet). I wouldn’t want my own children, WP YouTube Lyte and WP Donottrack, to feel neglected, now would I?

2012.FFWD; high performance Twenty Twelve child theme

Twenty Twelve is beautiful but slow, so I created 2012.FFWD. This Twenty Twelve child theme looks pretty much the same as the original, but comes with the following performance-boosting changes under the hood:

  • no webfont downloaded: loading fonts slow sites down, always. Fallback-fonts as defined in Twenty Twelve (Arial, Helvetica, Sans-Serif) are used instead.
  • minimized style.css: the amount of CSS is staggering, even minimized we’re still at 28KB!
  • minimized and inlined navigation.js: the overhead for doing a separate request for small javascript-files is bigger then the increase in filesize when inlining.

And these small changes indeed seem to have a pretty good impact on performance when comparing my previous test with Twenty Twelve with a new one for 2012.FFWD;

  • from 9 to 3 requests
  • from 142KB to 15KB downloaded size
  • loadtime from 2.457s to 0.937s

You can download the child theme here for now, but I might upload it to wordpress.org’s theme repository later.
Before applying 2012.FFWD on my own blog, I had to do some emergency bugfixing in Autoptimize. Autoptimize is the plugin I use to aggregate and minify JS and CSS (including data-uri magic on background images), but it is currently unmaintained. It messed Twenty Twelve (and hence 2012.FFWD) up severely when viewed in IE7 and IE8, aggregating html5.js into a javascript-file loaded at the bottom of the HTML and loading the IE-specific CSS before the aggregated generic CSS. If anyone is interested in my bugfixed version of Autoptimize (there’s some other fixes in there as well), drop me a line.

Dipping below the magical 1 second page load time

A couple of weeks ago I started looking into data-uri’s as a way to further optimize the performance of this personal playground of mine. Testing was easy enough; Autoptimize, the javascript/ css/ html opitimizing plugin I now use in conjunction with WP Super Cache, comes with support for data-uri’s in CSS and switching that option on indeed immediately resulted in less requests being made.
While testing I did find a small bug in Autoptimize (in /wp-content/autoptimize/classes/autoptimizeStyles.php) which caused jpeg images not to be taken into account. The regular expression in the code was ‘jpe?j’, while just below in a switch/case block ‘jpej’ and ‘jpg’ were referenced, resulting in neither ‘.jpg’ nor .’jpeg’ files ever being turned into data-uri’s. While reading the code I also noticed that the upper filesize limit for images to be turned into data-uri’s was set at 5120 bytes, but as base64-encoding does come with overhead, I decided to lower that limit to 2560 bytes.
So I made some minor changes in autoptimizeStyles.php on lines 207 and following:

if($path != false && preg_match('#\.(jpe?g|png|gif|bmp)$#',$path) && file_exists($path) && is_readable($path) && filesize($path) <= 2560) {
   switch(end(explode('.',$path))) {
      case 'jpeg':
         $dataurihead = 'data:image/jpeg;base64,';
         break;
      case 'jpg':
         $dataurihead = 'data:image/jpeg;base64,';
         break;
      case 'gif':
...

I also  switched my “subscribe” widget from a paragraph- to a bulleted list-approach with the rss- and mail-icons as background images and defined the adfreeblog and creative commons badges as background-images as well. And then I was ready to test the real impact of using data-uri’s on webpagetest.org. Behold the results for my current blog homepage (i.e. the one just before this post got published) with and without the use of data-uri’s (but excluding those dog-slow calls to stats.wordpress.com):

 No data-uri’sWith data-uri’s
webpagetest results URLhttp://www.webpagetest.org/result/121106_12_6FZ/http://www.webpagetest.org/result/121106_4H_71R/
#requests (full page)2619
Bytes in (full page)116KB114KB
Start render (median)0.615s0.634s
Doc Complete time (median)0.969s0.870s
Full page load time (median)1.932s1.332s

Let it be clear that the use of data-uri’s for background-images is not a silver bullet, but if you have images that are on every page of your site and they’re small in file-size, migrating those into your CSS as background-images with data-uri’s can result in an important performance improvement for your site.