Blogging Techstacks

Mitigating the Apache Range Header DoS on Ubuntu Apache 2.2

If you are running apache 2.2 (the current version in Ubuntu Server on my recently created VM being 2.2.17) and you wish to mitigate your web server against exploitation by the Apache Range Header Denial of Service vulnerability and the killapache exploit, below is the mitigation steps that worked best for me.

Ubuntu's Apache 2 implementation utilizes httpd.conf for user specific server configuration changes. The mitigation suggestion from the Apache Software Foundation that had the most favorable impact was scrubbing the Range Header if an incoming request contains more than 5 ranges within that specific request. Please note that this fix may not help with future versions of the exploit.

The workaround makes use of mod_headers, which, on a stock ubuntu server apache2 installation, is not enabled. You can enable it using the command line:

sudo a2enmod headers

Once that is complete, modify your servers httpd.conf and add the following directives:

SetEnvIf Range (?:,.*?){5,5} bad-range=1
RequestHeader unset Range env=bad-range

If you want to enable logging of hosts sending you those range requests, modify the configuration file for your virtual server—if you add it to the httpd.conf, the log file will get created but nothing will get logged. For example, if you are using the instance, modify the /etc/apache2/sites-available/default configuration file and add the following line somewhere that makes sense:

CustomLog /var/log/apache2/range-CVE-2011-3192.log common env=bad-range

That's all you need to do. Pretty straightforward workaround but hopefully a more formal fix will be made available by the Ubuntu Server team soon.

Tomcat Crashes Soon After Successful Startup - Common Causes*

( * plus the one that burned us...)

Perhaps you are seeing this, too.  You start tomcat up, tailing the logs for that Tomcat Started message.  All looks good.  Monitors are happy.  Smoke tests are working fine.  Log off the server and the next thing you know, your NOC is calling you because tomcat is down.  You log back onto the server, run a "ps -ef | grep tomcat" and sure enough, no tomcat process.

The tomcat logs don't seem to show anything.  There are no heap dumps getting written.  No stack traces, no exceptions.  Hoping its not going to be one of those unpleasant mysteries that take hours to fix, you start tomcat successfully again to restore service.  Everything is working fine again.  Hoping that shrugging your shoulders and commenting to the management on the complexities of tomcat classpath loading and bootstrapping, you end the call while silently muttering to yourself about these java "crapplications".  Seconds to minutes later, tomcat is down again...seemingly crashing as soon as you ended your ssh session!

Below are the three most common causes of tomcat failures after successful startup with no exceptions being logged:

1. You Suspect:  There is something wrong with the logging configuration.  Maybe you've got logging disabled or are writing to a directory you don't have permissions to write to.  Maybe the log level is wrong.
2. You Hope:  Someone on the development team has written, somewhere in the application, code that calls System.exit(0).  This would cause tomcat to terminate when called.  This has a few benefits to the systems administrator.  First, there is the timeless "wtf" that you can heave against the developers.  "WTF! Who would intentionally call System.exit(0) in production code!?!?"  Secondly, unless you like decompiling java .class files, you can get development to spend a lot of time pouring through their source to find the offending piece of code as this problem isn't necessarily yours to fix.
3. You Fear:  The JVM is crashing.  The JVM should be writing crash logs but if you think the jvm is crashing and you're not seeing crash logs getting generated, add the -XX:HeapDumpPath option to your JVM startup parameters, including a path to a directory tomcat can write to.  My Unofficial First Law of JEE System Administration dictates however that this will never be the cause unless you had dinner plans.

On to the reason that caused our failure, since, in our case, it wasn't any of the above.

More careful perusal of all the app server logs unearthed the following error:

X connection to localhost:10.0 broken (explicit kill or server shutdown).

Interestingly, in our case, one of the sysadmins starting tomcat up was connecting to the Linux host as the user that tomcat was running as with an ssh cllient that had X11 Port Forwarding enabled.  Once connected, the local DISPLAY environment variable was changed to point to his host.  Upon ending his session, the DISPLAY session that one of the third party graphing libraries apparently used if set, cleared, causing tomcat to die.  

A simple workaround was to add "unset DISPLAY" to tomcat's setenv.sh file, since, again in our case, the third party library worked just fine if loaded without DISPLAY being set.

Browsers: Is SSL Really Working?

Every time I think I have a fairly good understanding of how SSL works, something weird comes along to knock that understanding back a few notches.  Case in point:  Certificate Chains.  IBM has a nice, short article called "How Certificate Chains Work" that describes what they are so I'm linking to that in order to save some space for this post.  

With almost any type of certificate one purchases from Verisign today, and I use Verisign as an example because I am a Verisign certificate user, two intermediate certificates sit between the root certificate and the server certificate: a Primary Intermediate and a Secondary Intermediate.  The Primary Intermediate is the same regardless of the type of server certificate that was purchased.  The Secondary Intermediate varies according to the type of server certificate purchased.  If you purchased one of their SecureSite with EV certificates, the secondary intermediate is different from the one that is issued along with their SecureSite certificates but the Primary Intermediate is the same. 

Browsers have a feature in that they, I thought, displayed the full certificate chain, also known as the certification path.  They would display the certificate hierarchy, so you can see the root, the intermediate(s) and the server cert.  Problem is, they seem to have stopped doing this.  Take Safari 5, which is displaying the certification path for the Extended Validation cert securing www.verisign,com:

Safari 5 shows the Primary Intermediate as if it is the root certificate, followed by the secondary intermediate, followed by the server cert.  What's missing is the actual root cert as this is supposed to be a 4 way chain.  

Firefox 3.6.13 exhibits the same behavior.  Before you think, "Oh, this must be a Mac thing...", Firefox 3.6.13 running on Ubuntu 10.10 shows the same thing, too.  So then I thought, well, maybe this is what is supposed to happen but two peculiar additional discoveries are the cause of my confusion.

Safari 3 shows the full certification path:

I found in my System Roots keychain that Apple has imported the Class 3 Public Primary Certification Authority - G5 cert, (the Primary Intermediate).  Perhaps that is why it is displaying a 4 way chain as a 3 way chain?  Well, that's what me and a buddy thought before connecting to one of my sites secured with a non EV cert but still utilizing the same 4-way chain.  In that case, all four certs in the certification path are displayed in Safari 5 (portions of the image redacted to protect the innocent):

So...what's going on browser makers?  I'm assuming that there is a bug somewhere but where??  Is my understanding of how this should be working the bug?