14 posts categorized "loadbalancing"


A Slightly More Resilient JSESSIONID Persistence iRule

One of the nice features built into mod_jk is an automatic retry of a request if a worker is down.  It will not necessarily do this by default but it is a configurable option that a lot of people utilize in the event an upstream tomcat, jboss, or glassfish instance is timing out on a socket connection attempt.

The standard irule out there for jsessionid persistence that many of us use does not necessarily have this level of resiliency.  Unfortunately, servers and jvm's crash and the health check (monitor) intervals that we set on our bigip's may not catch an error quickly enough for higher volume sites. Taking the standard http_20_sec monitor built into the bigip as an example, one would need to wait 61 seconds before the bigip were to disable the server failing that check.  On a high volume site, 61 seconds worth of errors when a node or service fails would result in a lot of unhappy users of our sites.

 The following iRule will resend a request to another load-balanced node in the event that the node is not responding.  Please note that this rule also includes functionality when a jsessionid is set and it is empty.  (One mechanism I've come across in my system administration travels is that a quick and dirty way a developer will use to log a user out of a java web application is to clear the session ID.  This rule also handles these cases.)

  set mypool [LB::server pool]
  set lb_fails 0


   # Check if the JSESSIONID cookie is present in the request and has a non-null value
   if { [HTTP::cookie "JSESSIONID"] ne "" }{

      # Persist on the JSESSIONID cookie value for X seconds
      persist uie [HTTP::cookie "JSESSIONID"] 2700

   } else {

      # Cookie wasn't set or didn't have a value, so check for the session ID in the URI
      set jsess [findstr [HTTP::uri] "JSESSIONID" 11 ";"]
      if { $jsess != "" } {
         # Persist on the JSESSIONID URI value for X seconds
         persist uie $jsess 2700

   # Check if the JSESSIONID cookie is present in the response and has a non-null value
   if { [string map {\" ""} [HTTP::cookie "JSESSIONID"]] ne "" }{
      log local0. "JSessionID in Response: [HTTP::cookie "JSESSIONID"]"
      log local0. "Set-Cookie: [HTTP::header values Set-Cookie]" 
      # Persist on the JSESSIONID cookie value for X seconds
      persist add uie [HTTP::cookie "JSESSIONID"] 2700

when LB_FAILED {
  if { $lb_fails < [active_members $mypool] } {
    LB::mode rr
    LB::reselect pool $mypool

The new additions to this rule are the when CLIENT_ACCEPTED section and the when LB_FAILED section. These are also standard irule code examples used when a load-balanced selection fails. The irule simply selects another node using a round-robin algorithm when a connection failure occurs. A downed server or offline jvm will issue a connection reset back to the BigIP, which is why this irule works, and it will issue those resets quicker than it will take for the health monitors to disable the node.

Next up on my more-resilient jsessionid-persistence irule to do list is a mechanism to handle those cases where a jvm is paused due to long garbage collection times.  This rule will continue to route requests to application servers that are hung or paused.  The solution there might be to lower the accept count or listen queue size on the container's http connector instead of trying to fix this programmatically but there might be some concerns with that from the development teams.


Tomcat Connectors v1.2.31 Released

The Apache Tomcat Connectors team has released v 1.2 31 today.  Tomcat Connectors are also commonly known as mod_jk (for the apache web server), isapi_redirect.dll (for IIS), and nsapi_redirector.so (for iplanet).  The changelog indicates a relatively small number of fixes and enhancements but the three that stood out for me are:

  • Apache 2.3/2.4 support was enabled in mod_jk.
  • IIS adds apache rotatelogs-style support to generated log files, (no more multi-gigabyte jk logs).
  • For those of us interested in this level of detail, we can now log local and remote ports.

Download it from a mirror near you!


Updated Howto: Setting up BigIP for JSESSIONID-Based Persistence

I updated the HOWTO: Set Up JSessionID-Based Persistence on a BigIP article recently to reflect some additional changes/discoveries made while running this in a production environment for a while.  The howto covers migrating off of apache/mod_jk to apache/mod_proxy and moving persistence based on jsessionid to a bigip ltm.  

If you are a developer reading this and wondering why in the world anyone would want to move away from mod_jk, the rationale behind the migration is pretty straightforward:  When you have more than a couple web servers front-ending your tomcat, glassfish, or jboss application servers, or even if you only have a couple of web servers but a lot of workers defined within your workers.properties files, disabling servers using the jk-status handler page can be a real hassle.  Training 1st or 2nd level support personnel within your network operations center on disabling servers in a bigip is much easier than showing them how to disable workers within a jk-status page. 

In addition, the BigIP enables the DevOp or Application Delivery Engineer to create more sophisticated health checks to assess app server health over mod_jk's cping/cpong.  A well written health monitor on the bigip is much better able to assess if an app server is hung than mod_jk's health checks at present.  A BigIP health check can assess almost instantly when a java application server is hung where requests using the traditional mod_jk approach must wait until the ajp backlog threshold has been reached, which could result in a lot of request hitting a hung application server unnecessarily.

Changes to the document include information on why oneconnect is necessary as well as mod_proxy-specific environment variables that should be set in order to avoid 502-Proxy Errors showing up in your apache error log during periods of low site activity.