Some HTML DOM parsing gotchas in PHP’s DOMDocument

Although I had used Simple HTML DOM parser for WP DoNotTrack, I’ve been looking into native PHP HTML DOM parsing as a possible replacement for regular expressions for Autoptimize as proposed by Arturo. I won’t go into the performance comparison results just yet, but here’s some of the things I learned while experimenting with DOMDocument which in turn might help innocent passers-by of this blogpost.

  • loadHTML doesn’t always play nice with different character encodings, you might need something like mb_convert_encoding to work around that.
  • loadHTML will try to “repair” your HTML to make sure an XML-parser can work with it. So what goes in will not come out the same way.
  • loadHTML will spit out tons of warnings or notices about the HTML not being XML; you might want to suppress error-reporting by prepending the command with an @ (e.g. @$dom->loadHTML($htmlstring);)
  • If you use e.g. getELementsByTagName to extract nodes into a seperate DomNodeList and you want to use that to change the DomDocument can result in … unexpected behavior as the DomNodeList gets updated when changes are made to the DomDocument. Copy the DomNodes from the DomNodeList into a new array (which will not get altered) and iterate over that to update the DomDocument as seen in the example below.
  • removeChild is a method of DomNode, not of DomDocument. This means $dom->removeChild(DomNode) will not work. Instead invoke removeChild on the parent of the node you want to remove as seen in the example below
// loadHTML from string, suppressing errors
$dom = new DOMDocument();
@$dom->loadHTML($html);
// get all script-nodes
$_scripts=$dom->getElementsByTagName("script");
// move the result form a DomNodeList to an array
$scripts = array();
foreach ($_scripts as $script) {
   $scripts[]=$script;
}
// iterate over array and remove script-tags from DOM
foreach ($scripts as $script) {
   $script->parentNode->removeChild($script);
}
// write DOM back to the HTML-string
$html = $dom->saveHTML();

Now chop chop, back to my code to finish that performance comparison. Who know what else we’ll learn 😉

How to keep Autoptimize’s cache size under control (and improve visitor experience)

Update 2016: since AO 2.0, inline JS is not aggregated by default any more, which should prevent cache-size problems from occurring. Easiest solution for cache size issues is to make sure “aggregate inline JS” (and CSS) option is disabled. Below HowTo remains relevant in case you decide to enable the aggregation of inline code.


Confession time: Autoptimize does not have its proper cache purging mechanism. There are some good reasons for that (see below) but in most cases this is not something to worry about.
Except when it is something to worry about off course. Because in some cases the amount of cache-files generated by Autoptimize can grow to several Gigabytes. Why, you might wonder? Well, for each page being loaded Autoptimize aggregates all JS (and CSS) calculates the hash of that string and checks if an optimized version is in cache using that hash. If there is a difference (even if just a comma), the hash is not the same and the aggregated CSS/ JS is cached seperately. This behavior typically is caused by plugins that generate javascript-variables (or CSS-selectors) that are specific for each page (or even worse, for each page request). That does not only lead to a huge amount of files in the cache, but also impacts visitors as their browsers will have to request a different optimized CSS- or JS-file for each page instead of reusing the same file for several pages.
This is what you can do if you want a healthier cache both from a server- and visitor-perspective (based on JavaScript, but the same principle applies to CSS);

  1. Open two similar pages (posts).
  2. View source of the optimized JavaScript in those two pages.
  3. Copy the source of each to a seperate file and replace all semi-colons (“;”) with semi-colon+linefeed (“;\n”) in both files.
  4. Execute an automatic comparison between the two using e.g. diff (or “compare” in Notepad++), this should give you one or more lines that will probably be almost the same, but not exactly (e.g. with a different nonce or a postid in them).
  5. Now disable JS optimization and look for similar strings in the inline and the external JavaScript.
  6. If you find it in the inline JavaScript, try to identify a unique string in there (the name of a specific variable, probably) and write that down. If the variable JS is in a file, jot down the filename.
  7. Go to the autoptimize settings page and make sure the advanced settings are shown.
  8. Now add the strings or filenames from (6) to “Exclude scripts from Autoptimize:” (which is a comma-seperated list).
  9. Re-enable JS optimization.
  10. Save settings & clear cache.

This does require some digging, but the advantages are clear; a (much) smaller cache-size on disk and better performance for your visitors. Everyone will be so happy, people will want to hug you and there will be much rejoicing, generally.
So why doesn’t Autoptimize have automatic cache pruning? Well, the problem is a page caching layer (which could be a browser, a caching reverse proxy or a wordpress page caching plugin) contains pages that refer to the aggregated JS/CSS-files. If those optimized files were to be automatically removed while the page would remain in the page caching layer, people would get the cached page without any JS- or CSS-files being available. And as I don’t want Autoptimize to break your pages, I didn’t include a automatic cache purging mechanism. But if you have a bright idea of how this problem could be tackled, I’d be happy to reconsider, off course!

Should you inline or defer blocking CSS?

CSS codeYou care about web performance and so you dutifully aggregate and minify your CSS. But then a couple of months ago Google PageSpeed Insights (for mobile) started identifying CSS as a render blocking resource and so you wonder if you should you inline your CSS, or even defer loading it. Based on tests executed on a multi-site WordPress installation, deferring CSS is not the best idea just yet, but inlining might be worthwhile! Read on for the hard numbers and other details.

The test-setup

  • 4 test-blogs on a multi-site WordPress instance
  • using the Expound theme (which is interesting because its main stylesheet imports 2 other CSS-files)
  • using Lite Cache for page caching (new WordPress page caching plugin by the Hyper Cache author)
  • the same content was imported on all 4 blogs
  • all 4 had Autoptimize HTML & JS optimization active
  • the difference was in the Autoptimize CSS settings, where:
    • blog 1 had no CSS optimization at all (baseline )
    • blog 2 had standard CSS aggregation and minification, linked in head
    • blog 3 inlined the optimized CSS
    • blog 4 deferred the optimized CSS
  • each blog was tested on webpagetest.org‘s Amsterdam node on a DSL-profile using IE9 and doing 9 test-runs on one specific blogpost with contained a 16KBimage (I excluded favicon.ico as it seemed to pollute results)
  • each blog was analyzed by Google PageSpeed Insights for both mobile & desktop

The results

test report urlfirst bytestart renderdoc completefully loadedmobile pagespeeddesktop pagespeed
1. no css optimization140212_Z1_MKN0.299s2.246s2.221s2.221s7992
2. optimized CSS linked140212_H7_MKP0.239s0.608s1.390s1.390s9197
3. optimized CSS inlined140212_A3_NJA0.232s0.348s0.658s0.658s9999
4. optimized CSS deferred 140212_8J_P1G0.248s0.357s1.034s1.034s9995

The conclusions

Based on these tests (your mileage may vary, always test your results):

  • deferring all CSS is useless, performance is worse, desktop PageSpeed score is (slightly) lower and there is a “flash of unstyled content” between the rendering of the page and the application of the CSS.
  • Inlining CSS yields the best results both from a page speed and PageSpeed perspective. Although the base HTML is larger as it has the CSS payload, this has almost no impact in this specific context and rendering is almost instantaneous. Off course in a context where multiple other pages from the same site, with the exact same CSS would be loaded, the impact would be significant. Hence inlining CSS is especially interesting for sites with a low pageviews/ visitor ratio.

The future; inline + defer

Deferring CSS may seem pretty useless, but the sweet spot may just be inlining base CSS (everything needed for initial rendering above the fold) and deferring everything else. This is what CSS optimizing tools should focus on in 2014 and you can certainly expect something along these lines in one of the next major Autoptimize releases (although diehards can already test this approach).

Irregular Expressions have your stack for lunch

I love me some regular expressions (problems), but have you ever seen one crash Apache? Well I have! This regex is part of YUI-CSS-compressor-PHP-port, the external CSS minification component in Autoptimize, my WordPress JS/CSS optimization plugin:
/(?:^|\})(?:(?:[^\{\:])+\:)+(?:[^\{]*\{)/)/
yo regex dawgExecuting that on a large chunk of CSS (lots of selectors for one declaration block, which cannot be ripped apart) triggers a stack overflow in PCRE, which crashes Apache and shows up as a “connection reset”-error in the browser.
Regular expression triggered segfaults are no exception in the PHP bugtracker and each and every of those tickets gets labeled “Not a bug” while pointing the finger at PCRE, which in their man-pages and in their bug tracker indeed confirm that stack overflows can occur. This quote from that PCRE bug report says it all, really;

If you are running into a problem of stack overflow, you have the
following choices:
  (a) Work on your regular expression pattern so that it uses less
      memory. Sometimes using atomic groups can help with this.
  (b) Increase the size of your process stack.
  (c) Compile PCRE to use the heap instead of the stack.
  (d) Set PCRE's recursion limit small enough so that it gives an error
      before the stack overflows.

Are you scared yet? I know I am. But this might be a consolation; if you test your code on xampp (or another Apache on Windows version), you’re bound to detect the problem early on, as the default threadstacksize there is a mere 1MB instead of the whopping 8MB on Linux.
As for the problem in YUI-CSS-compressor-PHP-port; I logged it on their Github issue-list and I think I might just have a working alternative which will be in Autoptimize 1.8.

Looking at 2013 disappearing fast in the rear view mirror

rearview mirror by Alvaro on FlickrAnother year behind us, another overview in numbers (as done previously for 2011 and 2012).

On a personal note, 2013 has not been the easiest of years, but our lovely daughter’s “lentefeest” (a non-religious rite of passage for 6-year olds) and our holiday in Italy were great highlights though.

Coming up: Autoptimize 1.8 with API and inlined CSS

I’ve been working on a new Autoptimize version over the last couple of weeks and consider it largely finished now. The following features are in:

  • Optionally inline all CSS as suggested by Hamed (warning: can result in improved pagespeed-score but lower page speed! I’ll write a blogpost about when to use and not to use this one shortly)
  • Simple API Set of filters to provide a simple API (including example code) to change Autoptimize behavior to
    • conditionally disable Autoptimize on a per request-basis
    • change the CSS- and JS-excludes
    • change the limit for CSS background-images to be inlined in the CSS
    • define what JS-files are moved behind the aggregated one
    • change the defer-attribute on the aggregated JS script-tag
  • Improvement: updated upstream CSS minifier
  • Improvement: switch default delivery of optimized CSS/JS-files from dynamic PHP to static files
  • Improvement (force gzip of static files) and bugfix (force expiry for dynamic files, thanks to Willem Razenberg) in .htaccess
  • Bugfix: fail gracefully when things go wrong (e.g. the CSS import aggregation gone haywire resulting in empty aggregated CSS-files that was reported by Danka)
  • Bugfix: stop import-statements in CSS comments to be taken into acccount as seen by Joseph from blog-it-solutions.de
  • Bugfix: fix for blur in CSS breaking as reported by Chris of clickpanic.com
  • Updated translations

Some more testing and a couple of translations are still to be updated and we’re good to go for a release early January. You’re welcome to join in on the fun off course; download the test-version here and let me know what works and -more importantly- what is broken!

Blank Yoast SEO sitemaps no more!

yoast_thumbsIf you’re using both Autoptimize and WordPress SEO by Yoast, you might have noticed blank sitemaps in your browser. With the help of Vance Hallman and Armand Hadife, I have been able to isolate and fix this rather interesting bug.
I pretty soon saw that It actually wasn’t the sitemap XML, but the XSL (XML stylesheets, browsers use those to make XML readable) that was wrecking havoc. This means that although you couldn’t see them in your browser, the sitemaps themselves were not broken and could still be used by search engines.
But then why was the XSL broken? Well, Yoast’s WordPress SEO generates the sitemap XML and XSL on the fly, hooking into WordPress’ template_redirect action, according to the source-code to “Hijack requests for potential sitemaps and XSL files”. Autoptimize hooks into the same action, finds JavaScript in the XSL (jQuery tablesorter actually) to optimize, and injects the minimized script with a boolean defer-attribute back in the HTML. The problem; it’s not HTML, it’s XSL with HTML in it. And a defer attribute on a script-tag is not valid XML, so the XSL does not validate so the browser can not render the sitemap, resulting in the blank page you might have seen.
Although fiddling with the defer-attribute could solve the issue, I think it’s not a good idea for Autoptimize to try to optimize non-HTML resources as it isn’t WordPress’ core functionality either, so I changed Autoptimize to not act on non-HTML content (i.e. without an HTML-tag or with an xsl:stylesheet-tag). This change is in the upcoming 1.7.3 release, it will be pushed live tomorrow.

WP default themes 2010-2014: performance going down

Although it is easy to download and install free or “premium” themes, WordPress does come with a default theme that is updated yearly. Last year I did performance tests of Twenty Twelve and there were some performance-issues, which led me to create a cleary faster child theme (2012.FFWD). But how does Twenty Thirteen fare, you might wonder? And how did the WordPress themes do before? And what about Twenty Fourteen? Just the questions I was asking myself as well, so here goes!
I decided to compare the raw performance of the default WordPress themes from 2010 to 2014, creating new blogs in my multi-site test-environment. All 5 test-blogs’ homepages were tested 9 times with webpagetest.org, using the Amsterdam node, IE9 as browser and with a DSL traffic shaping profile and the median test result was used. No performance enhancing substances (such as WP Super Cache or Autoptimize) were used in this test, a few bunnies might have been slightly injured though … The result? Well, Just look at this graph;
wordpress default theme 2010-2014 performance
As you can see performance (until document loaded) got worse with every release. Download time skyrocketed from 1.7s to 4.2s (and even 5.3s the upcoming 2014), mostly because download size went from 60Kb for Twenty Ten to a whopping 489Kb for Twenty Thirteen (and 659Kb for the non-final Twenty Fourteen). All details of the tests can be found in this Google Docs spreadsheet.
So what is the reason for this important performance degradation? Web Fat, that’s what. In 2013 jQuery & co were added (there was already one smallish JS-file in 2012), but even more damaging is the explosion in font-file downloads; 4 in 2012, 9 in 2013 and 15 in 2014! I’ve already expressed my dislike for webfonts and -although I think it can be immensely useful and I use it for the admin pages of my plugins- I think one should also try to avoid including jQuery where possible, especially for content-oriented sites such as the ones powered by WordPress? Unless you don’t care about performance off course.
(In case you’re wondering; the use WP Super Cache and Autoptimize or similar solutions will clearly improve performance, but not having JavaScript is better then having to optimize JavaScript and font-files cannot really be optimized, so the problem will be less but will remain.)

Do not donate to me!

Some people ask me if they can donate for my software. My answer invariably is they can’t, because I don’t want money. Instead I would like them to make a small donation to a good cause!
Why? Well, I’m a very lucky guy, living in the richest region of one of the richest countries of the world and having a full-time job that allows me and my family to live a comfortable life. Software is “just a hobby” which I hope is a small contribution to make the web (and by extension the world) a slightly better place.
So given that context and given the fact that there are many ways in which the world could be made a better place, I would like to ask you to donate any amount of money you think my little projects are worth to a good cause. Just pick one, click and donate!

  • Donate via GiveDirectly which helps people living in extreme poverty by making unconditional cash transfers to them via mobile phone.
  • Lend via Kiva, or send me a Kiva-voucher and I will lend with that. Kiva allows people to lend money via the Internet to low-income / underserved entrepreneurs and students in 70 countries.
  • Support La Quadrature du Net, which is one of Europe’s leading organizations promoting net neutrality, a positive reform of copyright and the protection of privacy (these guys were instrumental in defeating ACTA).
  • Mary’s Meals provides life-changing meals to some of the world’s poorest children every school day.
  • The Mozilla Foundation is responsible for Firefox and for the Mozilla Developer Network documentation, keeping the web open en other browsers honsest. Donate to them!

And there are many, many more great causes both locally and internationally that can use our help!