There is a danger lurking inside the configuration files of our java application servers, threatening our uptimes and taunting our users with inexplicable slowdowns and timeouts. This bugbear has a fairly harmless sounding name for something that can cause hours and hours of pain and frustration: http.proxyHost.
Some of you reading this post are like me and work on a network where we still need to designate a proxy server within our work machine's browser in order to get Internet access. In your browser, check to see if "Automatic Proxy Configuration" is enabled or whether anything appears to be set on that proxy settings page.
If you do have a proxy set in your browser, it is highly likely that it will also need to be set in your code or within the system properties of your JVM. The latter is one of the most common methods I've come across—setting a system-wide proxy using a JVM system startup property like: -Dhttp.proxyHost=oldskool_webcache.domain.com -Dhttp.proxyPort=8080. This is a great way to set it because it alleviates the developer from having to think about unpleasant things like "Wait...what am I doing?" and concentrate on coding but the preceding system property is the timebomb I am referring to writing this post.
Why? Well, I am sure that the intent of using -Dhttp.proxyHost=oldskool_webcache.domain.com -Dhttp.proxyPort=8080 as a JVM startup option, or stored in a property or configuration file, was for outbound Internet connection to an external web site or service but in actuality, all of your application's HTTP connections will go through that proxy. Even if your application is sending http connections to itself they will instead go to the proxy first. Setting the third system property http.nonProxyHosts may help but you do have to try and make that list as generic as possible because anything you may miss means a detour to the proxy. Also, consider that some developers are wily and are going to use an IP instead of a hostname in their code, so using http.nonProxyHosts is no panacea.
What's the danger? Well, if you're setting this is a system-wide proxy and your proxy server is down, then for all intents and purposes your application server is down. The scope of the disaster being entirely related to how busy your site is. If everyone in Sales and Marketing is flooding the proxy bandwidth trying to find out who said what on someone's Facebook wall or they're all trying to find out who Ashton Kutcher's 1 billionth follower on twitter will be, then your application servers' response times will also be affected. A proxy should only be used for a request that actually needs it because there is no other way to get to that URI without it. Why is this listed as my top availability killer? Because any system-wide JVM proxy setting is going to be forgotten about—by every one—over time. New developers and engineers may not even realize that it is even there!
What's the fix? Well, it's dreadfully unsexy but setting a proxy on a per-connection basis in your code (using System.setProperty()) is probably the best way to do it. Yes, it can be a maintenance nightmare as the number of external URLs your site interacts with for data increases but that's why we're supposed to document code with comments! The funny thing about outages or problems caused by setting a system-wide proxy is that at the end-of-the-day, management and the customer don't really care about the inconvenience of setting a per-connection proxy in code—management cares much more about any lost revenue and the customer just wants to be able to log in. If you do have a system-wide proxy set as a jvm startup option or in some config file, consider getting it fixed as soon as possible before you have a disaster on your hands.
I would love to know your opinions on this. I've been thinking about this for a few days and it seems like setting a system-wide proxy using a system property at the jvm command line places a burden on the network/application/system engineer and the web application for relatively little overall benefit to the application developer.
