My Adventures on OpenShift

openshiftI have always been a fan of Red Hat, even if have never used their products extensively. They were one of the original movers in Linux-market back when Slackware was big and when InfoMagic CD-rom boxes with multiple distro’s were popular. And I have remained a fan because they succeeded in building a solid company built on and around open source & services.
So I was very happy to read that Red Hat had entered the PAAS-market with OpenShift, that that platform was built on open source(d) solutions and that a small-timer like me could deploy apps for free on their application cloud. I signed up, installed the WordPress instant application, added some tried & tested plugins and imported my content. Half an hour works, tops and performance proved to be great. Everything was just peachy, until I received this message in my mailbox;

We believe your use of OpenShift violates the Services Agreement and Acceptable Use Policy both of which can be found here: https://openshift.redhat.com/app/legal/
Infected file(s):
/var/lib/openshift53bcc3fd5973cabac00000d1/.tmp/53bcc3fd5973cabac00000d1/just_test_bc: Perl.Shellbot-8

And ZAP, my application was removed. As I had no idea how “just_test_bc” ended up in a temp-folder, the only possibility was a successful hack-attempt, so I contacted the security team to get more information. It took some time (and an escalation via the Customer Enablement Team), but I eventually got in touch with Stefanie at Red Hat, who was able to provide me with more information:

It looks like we had a one-off error in the script that emailed you. Your application was still flagged, but on a different file than we emailed about. This is the actual file:
/var/lib/openshift/53bd21435973cad637000080/mysql/data/ib_logfile0: PHP.ShellExec
So there was something in the mysql database log that set off the scan. […] It looks like mysql may have logged someone’s attempt to inject some bad PHP code into your app.

ib_logfiles are MySQL’s innodb replay log files and as Stefanie provided me with a tarball with my entire application, I extracted ib_logfile0 and used “strings” to extract readable information from the binary file. The result (from my mail to Stefanie);

Although php’s exec (and similar functions) can be found [in the logfile], this is always due to … blogposts about web security and specifically this one; http://blog.futtta.be/2007/12/02/php-security-eval-is-evil/. The content of that article was inserted in the DB and [thus] added to ib_logfile. Your scanner finds the content [in that innodb replay logfile] and flags this as a problem. I would think the OpenShift scanner needs some finetuning, [as now] anyone is at risk of having their app auto-removed if the mysql-redo-logfile happens to contain vaguely “offending” strings such as shell_exec?

OpenShift confirmed this analysis;

You’re absolutely right that our scanner needs work. So what I’m going to do is get you onto a whitelist so this thing doesn’t flag you again. […] All takedowns are currently on hold until I can implement pre-removal notifications [and] improve our standard operating procedure for this kind of thing. That should give people a chance to tell us that their apps are not malicious, so that we can whitelist others too, if needed. As long as they notice an email saying “OpenShift Terms Of Service Violation” within a few days, I think they should be safe. If they do get flagged as a false positive like your app did, they’ll email us back and let us know it’s a mistake, and then they’ll be added to the whitelist too.

Now wasn’t that an interesting adventure? If ever you get a notification-mail from OpenShift related to security issues, check if the problem isn’t with benign content being inserted in the database and if so be sure to contact OpenShift so they can add you to their whitelist.

2 thoughts on “My Adventures on OpenShift”

  1. Migrating from bluehost to open-shift was a true pain. And after I managed to overcome all problems, fix all errors, get all the settings work properly; They just banned my blog for no reason. They just provide me some backup files without any explanation. I couldn’t manage to restore my wordpress application with those files, kept getting errors with rhc command. And had to do every step of migration from scratch. I also asked whether this could happen again and got no response. So your adventure seems much nicer to me. I hope they don’t ban my humble programming related (in Turkish) blog again.

    Reply
    • It’s not reassuring for those who consider moving to openshift to see this still is an issue Harun. I would indeed expect them to at least provide some sort of explanation! Do let me know how this continues (as I am or was actually considering moving to Openshift for real this time) …

      Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.