Blogging Techstacks

TLS Renegotiation Remediation is Going to be Unpleasant

I was contacted by my colleagues in network engineering late last week to investigate what could possibly be causing a production issue that had started earlier that same week. An in-house client application making soap calls over an ssl connection periodically for data provided by a third party financial services firm began failing and they suspected a change somewhere in the network being the cause since "nothing changed in the application". The very same app continued operating just fine going to the exact same web service in an in-house test environment. Access to the service at the financial services firm was over ssl/tls-only and restricted by client certificate.

Below is a sample of the log for this web services client application:

16-Jul-2010 14:54:28 org.apache.commons.httpclient.HttpMethodDirector authenticateHost
WARNING: Required credentials not available for BASIC @redacted.host.tld:443

16-Jul-2010 14:54:28 org.apache.commons.httpclient.HttpMethodDirector authenticateHost
WARNING: Preemptive authentication requested but no default credentials available

16-Jul-2010 14:54:28 org.apache.commons.httpclient.HttpMethodBase readResponseBody
INFO: Response content length is not known
javax.net.ssl.SSLException: HelloRequest followed by an unexpected  handshake message

Later, the systems administrator informed me that they were able to workaround the issue by setting a system property at initialization of the jvm:

 Dsun.security.ssl.allowUnsafeRenegotiation=true

Ultimately, what this turned out being was an update of the JVM on the production boxes from whatever version was there previously to version 1.6.0_20. As we all know, java updates for the most part are cumulative and update 19 added what Oracle calls an "interim fix" for the TLS renegotiation vulnerability. I won't go into detail about the TLS Renegotiation vulnerability here—there is an excellent write up about it on the Educated Guesswork blog that I'm linking to, if you want to know more.

In a nutshell, version 1.6.0_19 disables TLS Renegotation by default. I think in the case of an ssl or tls connection that requires authentication by client certificate, the initial connection is made to the server by the client, then the server responds with a 403 looking for a client certificate. The client then renegotiates the session presenting the client certificate. The change to the jvm behavior due to the TLS Renegotiation vulnerability breaks this. In the case of a java application using the apache httpclient, it fails with a javax.net.ssl.SSLException: HelloRequest followed by an unexpected  handshake message exception.

Unfortunately, many vendors are utilizing this type of workaround while waiting for the TLS protocol to be fixed. As in Oracle's case, vendors, like Microsoft, are providing "patches" that simply disable auto renegotiation by default and letting support staff discover on their own when applications are broken. So, in case anyone finds this doing a search on the web, and because many vendors are not really providing examples of the types of applications and connections that _could_ fail should someone implement one of these patches, below is my draft list detailing the types of TLS/SSL connections that could fail. I have not personally tested or verified this list as it is an attempt to brainstorm potential issues so if anyone has any experience with them and knows that a particular item is incorrect, let me know and I'll update this post.

• SSL or TLS connections to servers that require client certificates.
• In apache specifically, if you upgrade encryption ciphers for specific sections of your site using Location directives, you could see problems.For example, if you specify Medium grade encryption as acceptable for most of the site and upgrade the cipher to a High grade one for login pages or Order Completion pages, this forces a renegotiation of the connection, which could break.
• FTPS (also known as FTP-SSL or simply FTP over SSL) connections in Explicit Mode.
• Secured LDAP connections where a STARTTLS operation is invoked and the existing unencrypted connection is upgraded to a secured connection.
• IMAP, POP3, NNTP, and SMTP connections that step up to a secured connection. These also invoke a STARTTLS operation to renegotiate the existing connection using TLS.
• I don't know of anything supporting it yet but connections supporting RFC 2817 - Upgrading to TLS Within HTTP/1.1 could potentially be impacted as well.

Does Technorati Work Any More?

If I ever need to know when I switched this blog over to its present-day domain name of 'blog.techstacks.com', Technorati has a great service for small-time bloggers like me called "Last Ping".  Last Ping, gives you the number of days since you last pinged technorati telling their crawlers that you have published or posted something new on your blog.  

As of the time of this writing, even though I ping technorati EVERY TIME I post something new or update something on the site, Last Ping is always there ready to let me know that it has been 185 days since my last ping (and counting!).  185 days ago is when I switched domain names from 'allsortsandnotions.blogspot.com' to 'blog.techstacks.com'.  

It's almost cruel.  It doesn't seem to matter where or how I ping.  Ping from their "Ping Us" page:  "Thanks for the ping!  You last pinged us 185 days ago!"  Ping through Ping-O-Matic?  "Ping sent!"  I've written this primitive groovy script below that pings using their XML-RPC API.  "Thanks for the Ping!" 

#!/usr/bin/env groovy

import groovy.net.xmlrpc.*

def server = new XMLRPCServerProxy("http://rpc.technorati.com/rpc/ping")

def result = server.weblogUpdates.ping("blogging techstacks" , "http://blog.techstacks.com/")

if (result != null)
 println "Thanks for the ping!" 

And still, Last Ping is always there to remind me that all my subsequent ping attempts are really cute and funny.  So, I have written a new script for other bloggers out there called "Last Ping" that logs into Technorati and reminds you how long it has been since your last worthy ping.  Uses Perl and the WebService::Technorati module, (gosh--CPAN really does have everything).  This script is a real time-saver!  Below is the source code and it may soon wind up available for download on the Downloads page.  Update:  Technorati API calls began failing on October 25th, 2009, due to the launch of the 'new' Technorati, so this script has been removed from the Downloads page.  Plug in your blog url and your technorati api key in the corresponding $url and $apikey variables and you too can see at a glance when your last 'real' ping was received.  Output for me is as follows:

    The last compelling thing I wrote was on: 2009-01-20 06:56:38 GMT

Perl and Groovy Source:

#!/usr/bin/env perl

use strict;
use warnings;

use WebService::Technorati;

my $apikey = 'enter_your_API_KEY_here';
my $url = 'your_BLOG_URL_goes_here';
my $t = WebService::Technorati->new( key => $apikey );
my $q = $t->getBloginfoApiQuery($url);
$q->execute;

my $lastping = $q->getSubjectBlog();
print "The last compelling thing I wrote was on: " . $lastping->{lastupdate} . "\n";

I'm hoping that soon Technorati will build some web services out of their support site so that I can then script something that will track how long it has been since I opened my first and second cases with them about this particular issue and maybe even calculate the delta between the case open date and a response.

The groovy version of the script above is:

#!/usr/bin/env groovy

import groovyx.net.http.RESTClient
import groovy.util.slurpersupport.GPathResult
import static groovyx.net.http.ContentType.*

client = new RESTClient( 'http://api.technorati.com/bloginfo?key=INSERT_YOUR_API_KEY_HERE&url=INSERT_YOUR_URL_HERE' )

resp = client.get( contentType:XML, headers: [Accept: 'application/xml'] )
  resp.data instanceof GPathResult

println "The last compelling thing I wrote was on: ${resp.data.document.result.weblog.lastupdate}"

b7xqdu7i7e

5K4HH6YPHF77

Gripe: Marketing, Development, and URLs

Just wanted to rant about another pet peeve of mine. As far as I am concerned, the disconnect between Development and the Business is no more pronounced as when the URL that the business advertises is different from the url of the application. For example, Development recently deployed a tomcat web application to www.site.com/webapp (not the actual name) and the next day, Marketing blasts the URL out to 20 or 30 thousand prospects as www.site.com/WEBAPP.

I mean really....haven't these two teams been working closely for at least the past few months developing this application? Has Microsoft's case-insensitive URL's in IIS completely lowered the bar when it comes to basic web knowledge? URLs are supposed to be case sensitive.

Obviously, it's relatively easy to fix with redirects, rewrites, whatever but how many requests wound up failing before the engineering folks got involved?