My Adventures on OpenShift

openshiftI have always been a fan of Red Hat, even if have never used their products extensively. They were one of the original movers in Linux-market back when Slackware was big and when InfoMagic CD-rom boxes with multiple distro’s were popular. And I have remained a fan because they succeeded in building a solid company built on and around open source & services.
So I was very happy to read that Red Hat had entered the PAAS-market with OpenShift, that that platform was built on open source(d) solutions and that a small-timer like me could deploy apps for free on their application cloud. I signed up, installed the WordPress instant application, added some tried & tested plugins and imported my content. Half an hour works, tops and performance proved to be great. Everything was just peachy, until I received this message in my mailbox;

We believe your use of OpenShift violates the Services Agreement and Acceptable Use Policy both of which can be found here: https://openshift.redhat.com/app/legal/
Infected file(s):
/var/lib/openshift53bcc3fd5973cabac00000d1/.tmp/53bcc3fd5973cabac00000d1/just_test_bc: Perl.Shellbot-8

And ZAP, my application was removed. As I had no idea how “just_test_bc” ended up in a temp-folder, the only possibility was a successful hack-attempt, so I contacted the security team to get more information. It took some time (and an escalation via the Customer Enablement Team), but I eventually got in touch with Stefanie at Red Hat, who was able to provide me with more information:

It looks like we had a one-off error in the script that emailed you. Your application was still flagged, but on a different file than we emailed about. This is the actual file:
/var/lib/openshift/53bd21435973cad637000080/mysql/data/ib_logfile0: PHP.ShellExec
So there was something in the mysql database log that set off the scan. […] It looks like mysql may have logged someone’s attempt to inject some bad PHP code into your app.

ib_logfiles are MySQL’s innodb replay log files and as Stefanie provided me with a tarball with my entire application, I extracted ib_logfile0 and used “strings” to extract readable information from the binary file. The result (from my mail to Stefanie);

Although php’s exec (and similar functions) can be found [in the logfile], this is always due to … blogposts about web security and specifically this one; http://blog.futtta.be/2007/12/02/php-security-eval-is-evil/. The content of that article was inserted in the DB and [thus] added to ib_logfile. Your scanner finds the content [in that innodb replay logfile] and flags this as a problem. I would think the OpenShift scanner needs some finetuning, [as now] anyone is at risk of having their app auto-removed if the mysql-redo-logfile happens to contain vaguely “offending” strings such as shell_exec?

OpenShift confirmed this analysis;

You’re absolutely right that our scanner needs work. So what I’m going to do is get you onto a whitelist so this thing doesn’t flag you again. […] All takedowns are currently on hold until I can implement pre-removal notifications [and] improve our standard operating procedure for this kind of thing. That should give people a chance to tell us that their apps are not malicious, so that we can whitelist others too, if needed. As long as they notice an email saying “OpenShift Terms Of Service Violation” within a few days, I think they should be safe. If they do get flagged as a false positive like your app did, they’ll email us back and let us know it’s a mistake, and then they’ll be added to the whitelist too.

Now wasn’t that an interesting adventure? If ever you get a notification-mail from OpenShift related to security issues, check if the problem isn’t with benign content being inserted in the database and if so be sure to contact OpenShift so they can add you to their whitelist.

Dude, where’s my WordPress session?

WordPress is a favourite hackers target. Some say that is because it is inherently insecure, but in reality WordPress is mainly a target because of its popularity, because of people not keeping their installations up to date or using easy to guess usernames and passwords and because of vulnerabilities in plugins rather then WordPress itself.
There is, however, one security-related shortcoming in WordPress from a design point of view: sessions are not stored server-side. If someone logs in, a cookie is set in the browser containing username, a session expiration timestamp and a hash. With every new request to WordPress that cookie (and specifically the hash) is checked to validate the session, but there is no check to see if there indeed was such a session.
This can be considered mainly a theoretical shortcoming, not an immediately exploitable vulnerability, because;

  1. session-cookies are set with the HTTPOnly-flag so XSS should not be an issue
  2. in an ideal world all traffic, once logged in, would be over HTTPS, securing against network sniffing.

But there are other (albeit less obvious) ways to steal cookies or even create create new ones to gain unauthorized access, as demonstrated in this very detailed blogpost. As explained in that article, there is no way to block “fake” session-cookies from gaining access (your OTP plugin won’t protect you either) and there is no functionality to monitor and if needed delete sessions.
So … I wrote a small proof-of-concept plugin that gets triggered upon login, logout and upon session verification (i.e. each request) and which stores sessions server-side, automatically logging out unknown sessions. With that in place, lots of other optional features could easily be added;

  • display a list of all known current sessions
  • allow one or more sessions to be removed
  • compare IP address at session verification against the one at session creation and notify or logout if no match
  • compare User Agent (and optionally some HTTP accept-headers) at session verification against the one at session creation and notify or logout if no match
  • create an audit log

But … I don’t want to do this on my own. I have 3 plugins already, 2 of which are semi-popular and for which I try to do regular releases and provide great support (and I have a daytime-job and a wife and daughter with whom I love to spend quality time as well). Moreover I really don’t want the plugin to “just” be open source, but I want it to be developed in an open source, collaborative manner as well.
So if you’re a WordPress coder, a security consultant or just an innocent passer-by and you are willing to code, review code, translate or document, then do drop me a line. Fame (but not fortune) will be yours!

WP Caching plugin vulnerability debrief

Now that both WP Super Cache and W3 Total Cache developers have released a new version of their respective plugins (upgrade first, continue reading after) it seems time for a small “post mortem“.
The problem was in the interpretation of dynamic snippets, that are contained inside a number of specific HTML-comment tags. These snippets allow both plugins (and their predecessor WP Cache) to cache pages while keeping a limited amount of dynamic, PHP-generated content in them that can be executed on the fly. Think ESI in e.g. Varnish.
The vulnerability, which was originally discovered by kisscsaby and reported 3 weeks ago on the wordpress.org plugins support forum, had multiple causes:

  1. Unlike ESI’s, dynamic snippets can not only be includes (mclude) but also PHP-code (mfunc). Whereas one could consider includes of known files more or less safe, inclusion of PHP-code introduces a risk.
  2. As WP Super Cache & W3 Total Cache keep entire pages in cache and as pages can contain comments, that user generated content is parsed for dynamic snippets as well.
  3. WordPress core by default only allows a limited set of HTML in comments (“a blockquote code em strong ul ol li”), but it also leaves HTML comments in place.

As a result, blogs with WP Super Cache (before version 1.3) and W3 Total Cache (before version 0.9.2.9) were at risk of PHP code injection. Blog comments could contain dynamic snippets (in HTML-comments) and WordPress core did not them filter out. Upon a such a malicious comment having been submitted, a new cached version of the page was created that included the injected PHP-code. Upon the first request of the cached page, that code was successfully executed.
I stumbled on the vulnerability report about a week and a half ago, while researching why dynamic snippets weren’t executing when Autoptimize was active (simple really, Autoptimize by default removes HTML comments, the upcoming 1.6.3 will leave mfunc/mclude in place). As this seemed like a pretty severe security hole and as there was no feedback from developers in the support thread, I created a small “stopgap plugin” to mitigate the threat on April 10th, mailed security@wordpress.org and plugins@wordpress.org and requested WP Safer Cache being published on wordpress.org on the 11th. A couple of hours later WP Super Cache’s Donncha O Caoimh contacted me and the same day he released a version (1.3) that fixed this vulnerability by parsing out potential exploits from comments as they are posted and as they are rendered. On April 12th W3 Total Cache’s Frederick Townes confirmed they were working on a fix. Version 0.9.2.9 got released on April 17th, disabling dynamic snippets by default and when these are enabled, they require a secret alphanumeric key to be included in the snippet which is checked against one that is defined in wp-config.php.
Conclusions; The fact that this didn’t generate any fuss (as opposed to W3 Total Cache’s widely published information disclosure vulnerability in December 2012) is surprising. PHP Code injection clearly is a more severe security risk that must have been there for a long time already. The fact that this only got discovered recently is baffling. And why WordPress core doesn’t filter out HTML-comments from submitted blog comments, others seem to understand, but to me that remains the biggest mystery of all.

WP Safer Cache: stopgap for WordPress Cache plugins vulnerability

[UPDATE April 18th 2013: this vulnerability has been fixed in both WP Super Cache and W3 Total Cache. You can find more information in this “post mortem” blogpost]
[UPDATE April 11th to reflect that WP Super Cache version 1.3 fixed this issue]
There was a pretty severe vulnerability in WordPress installations that had WP Super Cache (until version 1.2, 1.3 fixed this issue) or W3 Total Cache (up until version 0.9.2.8) plugins activated. This security bug would, under certain circumstances, allow attackers to inject and execute arbitrary PHP code in comments.
The vulnerability could have been handled in WordPress core or in WP Super Cache and W3 Total Cache separately (with my preference being a fix in  comment sanitization in core). On April 11th WP Super Cache version 1.3 was released, fixing this issue and W3 Total Cache released a fix on April 18th. If you are on an older version of WP Super Cache or W3 Total Cache (do upgrade!), you might be interested in installing this little plugin that cleans out malicious … stuff from comments being posted.
As always; comments, bugs & improvements are welcome in the comment-field below or via the contact form.

On WordPress GET floods, plugin fingerprinting & keeping safe

flood sign, by alan in belfast from flickr under ccInfosec consultant and blogger Xavier Mertens suffered from a GET flood last week. The would-be DDOS originated from WordPress blogs that seemed non-related both geographically and content-wise, were using different versions of WordPress and seemed not to be compromised.
So what gives? WordPress blogs autonomously trying to DDOS other WordPress blogs? My best guess; a WordPress plugin gone rogue. The great WordPress HTTP API makes It easy enough to create a plugin that fetches targets from a control server and issues requests to those targets at a given time. It’s only a matter of hiding that behavior inside a plugin that seems useful and getting people to install that. The easiest way; finding an older plugin with an existing userbase and taking over development from the original developer (as i did with Autoptimize) is the easiest route to create your own little DDOS-ing botnet.

But all of this is pure speculation (although the UA matches the one used by WordPress HTTP API) off course. The only way to know for sure is to, for at least a sample of the flooding blogs, check what plugins they have in common. Doing so is frightfully easy using the NMAP HTTP WordPress Pugins script and if I am not mistaking Xavier is indeed looking into this.

But given the ease with which the NMAP-script can scan for WordPress plugins (there are similar scripts for e.g. Drupal modules), you might want to stop this from happening? I for one added this line in my WordPress .htaccess:

RedirectMatch 404 ^/wp-content/plugins/[^\/]*/$

Maybe you would even want to return a 404 for plugin readme.txt and index.html files as well, but I’ll consider that an assignment for you guys to chew over 😉

And now you can even have my WordPress password!

Being slightly obsessed with security, I was delighted to discover that two factor-authentication (OTP) using Google Authenticator client is not restricted to Google applications, but is fully standardized and as such can be implemented without dependency on Google services on any system. There is code (off course varying in quality and scope) available for PHP, .NET, Java and Python (and I’m sure there are others).
As you might expect after reading the title, there is a great Google Authenticator WordPress plugin which I installed in 5 minutes time earlier today. For the Drupal-heads; Antwerp-based Attiks have a module that implements Google Authenticator OTP which looks worth checking out as well (and I’m interested in your experiences with it, actually).

Learning from my Apache/ CDN mistakes

Earlier this year I configured this here little blog to offload static resources to MaxCDN. I made some mistakes in the process, which I documented in a blogpost that has been in my drafts for too long. So here’s the gist;

  1. Etags misconfiguration:
    • Problem: By default Apache uses the file’s inode to calculate the Etag, but that inode is unknown to the CDN, so Etags can never match.
    • Solution: change the Apache config not to use inode, e.g. “FileETag MTime Size”
  2. Cookies & domains issues:
    • Problem: I created my CDN-domain as a sub-domain of my main one, which led to (small) performance and (potentially huge) security issues.
    • Solution: Make sure your CDN-domain is different from the domain on which your cookies are set. So for example for main domain blog.futtta.be don’t use static-cdn.blog.futtta.be, but rather blog-cdn.futtta.be (except if cookies would be set on futtta.be, in which case I would need cdn-futtta.be).
    • Todo: it might make sense to add some Apache magic to make sure that for requests from the CDN:

CSP: doing unsafe-inline the Firefox-way

A couple of weeks ago I sobbed because of the lack of support for “unsafe-inline” in Firefox. There’s some Mozillians working on that (for CSS, at least), but given the release-train, that’ll probably only appear around Firefox 19. While perusing CSP-related tickets in Bugzilla however, I came across an interesting comment:

Firefox expects “options inline-script eval-script” instead of “script-src ‘unsafe-inline’ ‘unsafe-eval'” which it should be per spec. Also, Firefox expects “xhr-src” instead of “connect-src”.

Come again? So I can tell Firefox to execute inline script even without support for CSP 1.0 after all? I opened up my development-version of WP DoNotTrack to rework the “proof of concept”-code into this:

function wp_donottrack_csp() {
 global $listmode;
 if ($listmode==="1")
  $whitelist=wp_donottrack_getWhiteList(true);
  $csp="default-src 'self' 'unsafe-inline' ";
  if (is_array($whitelist)) {
   foreach ($whitelist as $white) {
    $csp.=" *.".$white;
   }
  }
 // old-style options inline-script for firefox
 $csp.="; options inline-script;";
 header("X-Content-Security-Policy: " . $csp);
 header("Content-Security-Policy: ". $csp);
 // needed for chrome, but safari 5 (latest version on windows) might be broken?!
 header("X-WebKit-CSP: " . $csp);
 }
}

Based on limited testing, it indeed seems to work great this way. So maybe -if this also turns out to work in IE10 and on Safari for Windows- a next version of WP DoNotTrack can ship with CSP-support after all?

Content Security Policy; Great! or Wait?

A couple of days ago I had another look at Content Security Policy, a technology that allows a site to tell a browser resources are allowed to be loaded to protect against XSS and some other types of web application vulnerabilities. CSP was originally devised by the Firefoxians, but is in the process of being standardized by the W3C with support in Firefox, Chrome, Safari and even the upcoming Internet Explorer 10.
Great!
The functionality offered by CSP (blocking requests that are not allowed) is pretty close to what WP DoNotTrack tries to do, so I decided I’d try to integrate CSP in my plugin, based on the following assumptions:

  • CSP-mode will only work for WP DoNotTrack if it is configured to use a whitelist
  • As most WordPress+plugins installations are bound to have pages with at least inline JavaScript and/or style, I have to add “unsafe-inline” to allow those to continue to work (which indeed limits the level of protection against XSS-attacks)
  • Given that a lot (most?) WordPress installations implement WP Super Cache of W3 Total Cache, it will -at least in a first stage- only kick in if WP  DoNotTrack is configured to filter unconditionally
  • Ideally the JavaScript-based component of WP DoNotTrack would “see” that CSP was activated and would not perform those nifty JavaScript AOP trickery

The “proof of concept”-quality code I ended up adding to wp-donottrack.php was pretty simple:

function wp_donottrack_csp() {
 global $listmode;
 if ($listmode==="1") {
  $whitelist=wp_donottrack_getWhiteList(true);
  $csp="default-src 'self' 'unsafe-inline'";
  if (is_array($whitelist)) {
   foreach ($whitelist as $white) {
    $csp.=" *.".$white;
   }
  }
  header("X-Content-Security-Policy: " . $csp); //FF & MSIE10
  header("Content-Security-Policy: ". $csp); //new standard
  header("X-WebKit-CSP: " . $csp); //chrome & safari
 }
}
add_action('init', 'wp_donottrack_csp', 10, 0);

Wait?
With this code on my testblog I started playing around in a couple of browsers. Based on that experience I found the following limitations:

So in this particular context (and specifically the absolute need for “unsafe-inline”), I’ve decided to hold off implementing CSP (I might implement iFrame sandboxing as support for that is coming with IE10 and will probably also land in Firefox 17). But if you have full control over a particular website or -application (meaning you can remove all inline JavaScript and CSS and all instances of evals in insourced JavaScript) and you want to harden your installation to stop cross-site scripting, you really should start thinking about implementing CSP (as Twitter seems to have done already)!