My Adventures on OpenShift

openshiftI have always been a fan of Red Hat, even if have never used their products extensively. They were one of the original movers in Linux-market back when Slackware was big and when InfoMagic CD-rom boxes with multiple distro’s were popular. And I have remained a fan because they succeeded in building a solid company built on and around open source & services.

So I was very happy to read that Red Hat had entered the PAAS-market with OpenShift, that that platform was built on open source(d) solutions and that a small-timer like me could deploy apps for free on their application cloud. I signed up, installed the WordPress instant application, added some tried & tested plugins and imported my content. Half an hour works, tops and performance proved to be great. Everything was just peachy, until I received this message in my mailbox;

We believe your use of OpenShift violates the Services Agreement and Acceptable Use Policy both of which can be found here:

Infected file(s):
/var/lib/openshift53bcc3fd5973cabac00000d1/.tmp/53bcc3fd5973cabac00000d1/just_test_bc: Perl.Shellbot-8

And ZAP, my application was removed. As I had no idea how “just_test_bc” ended up in a temp-folder, the only possibility was a successful hack-attempt, so I contacted the security team to get more information. It took some time (and an escalation via the Customer Enablement Team), but I eventually got in touch with Stefanie at Red Hat, who was able to provide me with more information:

It looks like we had a one-off error in the script that emailed you. Your application was still flagged, but on a different file than we emailed about. This is the actual file:

/var/lib/openshift/53bd21435973cad637000080/mysql/data/ib_logfile0: PHP.ShellExec

So there was something in the mysql database log that set off the scan. [...] It looks like mysql may have logged someone’s attempt to inject some bad PHP code into your app.

ib_logfiles are MySQL’s innodb replay log files and as Stefanie provided me with a tarball with my entire application, I extracted ib_logfile0 and used “strings” to extract readable information from the binary file. The result (from my mail to Stefanie);

Although php’s exec (and similar functions) can be found [in the logfile], this is always due to … blogposts about web security and specifically this one; The content of that article was inserted in the DB and [thus] added to ib_logfile. Your scanner finds the content [in that innodb replay logfile] and flags this as a problem. I would think the OpenShift scanner needs some finetuning, [as now] anyone is at risk of having their app auto-removed if the mysql-redo-logfile happens to contain vaguely “offending” strings such as shell_exec?

OpenShift confirmed this analysis;

You’re absolutely right that our scanner needs work. So what I’m going to do is get you onto a whitelist so this thing doesn’t flag you again. [...] All takedowns are currently on hold until I can implement pre-removal notifications [and] improve our standard operating procedure for this kind of thing. That should give people a chance to tell us that their apps are not malicious, so that we can whitelist others too, if needed. As long as they notice an email saying “OpenShift Terms Of Service Violation” within a few days, I think they should be safe. If they do get flagged as a false positive like your app did, they’ll email us back and let us know it’s a mistake, and then they’ll be added to the whitelist too.

Now wasn’t that an interesting adventure? If ever you get a notification-mail from OpenShift related to security issues, check if the problem isn’t with benign content being inserted in the database and if so be sure to contact OpenShift so they can add you to their whitelist.

Music from Our Tube; Fela Kuti live in Germany

I Guess I first heard Fela Kuti a couple of years ago on a Gilles Peterson show. Just now KCRW played My Morning Jacket’s version of “Trouble Sleep”, there’s also a nice version by Taj Mahal & Baaba Maal and the original version is here. But eventually I stumbled on this video of a Fela concert in Germany from 1978, which I think you should really see and hear;

Fela Kuti Live in Berlin – Berliner Jazztage 1978

Watch this video on YouTube or on Easy Youtube.

Uploaded to Our Tube; IDRchitecture remixed by Nathan Fake

I love to listen to DJ-sets which are made for the radio (or the web) instead of the dancefloor. The music tends to be a lot more diverse if the DJ isn’t focused on keeping the crowd in the groove. Soundcloud is a treasure trove for such DJ sets and a couple of months ago I bookmarked Nathan Fake’s 6 mix as aired on BBC Radio 6 in December 2012.

Great stuff, but there was one track that I specifically enjoyed listening to; an unreleased remix of (Sign of the Fish) by IDRchitecture. I didn’t know the band, didn’t know the song -which reminds me of The Chills, somehow-, but the clicks and ticks in this remix are just mesmerizing.

So now it is on YouTube as well;

idrchitecture – (sign of the fish) – nathan fake remix

Watch this video on YouTube or on Easy Youtube.

WP DoNotTrack whitelist & WordPress/ Jetpack stats

Although the number of pageviews of this blog already decreased from approx. 2100 pageviews per week before mid May to 1300 pv/week after (I never thought I’d ever be hit by a Panda), yesterday was an absolute disaster. Turns out that Automattic changed the domain of the Jetpack stats tracking pixel to, which WP DoNoTrack (for which I pushed out a small update in May) blocked as that domain was not whitelisted. The downside of white- instead of blacklisting.

Terug uit Kreta

zwembad, zon en boekenErg van een weekje verlof in Kreta genoten. Het was er wel heel erg warm, er zaten tientallen oorverdovende krekels rond het zwembad en het landschap was betrekkelijk dor.

Maar ik heb een fantastisch boek gelezen (“Het Puttertje” van Donna Tartt), heb me op mijn 45-ste eindelijk over mijn angst om in een zwembad te duiken heen kunnen zetten (al was het nog geen schoonspringen) en genoot er van om elke dag naar de winkel te fietsen; bijna 4km kilometer bergaf, een dubbele espresso aan de zee, wat eten kopen en dan weer 4km zwetend, puffend maar uiteindelijk zegevierend bergop.

En met vrouw en dochter in en aan het zwembad hangen en ‘s avonds samen lekker gaan eten, hoe ontspannend kan een vakantie zijn?

WordPress-as-a-service tip: Flywheel

flywheelAt work I was asked to provide advice on WordPress hosting. As we don’t have in-house LAMP-experience and as I didn’t want to have to take care of server operations myself (been there, done that), I decided to look into WordPress as a service solutions. To make things a tad more complicated, hosting had to be in a European data-center as we wanted optimal performance for our local customers and as our Privacy Officer requires all company data to be in Europe.

I contacted several US companies, but eventually Flywheel came out on top; they confirmed they could host in Europe (Amsterdam), seemed pretty eager, had a great package and they could provide me with a test-account to play around with their solution. And so I did; I set up a stock WordPress 3.9.x with Autoptimize and WP YouTube Lyte (call me prejudiced, but I like my own plugins), imported a bunch of posts from this blog and had WebPageTest be the judge.

The results were quite impressive;

Document CompleteFully Loaded
Load timeFirst byteStart renderDOM elemsTimeReqsBytes InTimeReqsBytes In
First View (Run 3)0.457s0.120s0.292s9260.457s473 KB1.008s12152 KB

0.120s until first byte, 0.292s start render and 0.457s doc complete? Sweet! So yeah, given those numbers, their offering and the fact they can deploy to a datacenter in Europe I do think Flywheel is a great choice for those who are looking for WordPress-as-a-service (well, PAAS really) solution!