BigIP: Botkilling iRule

Below is an irule for when you need to quickly kill connections from nasty robots (or scripters) hitting your site. We use something like this to kill connections from folks who have scripted POST requests to data search features on our sites to keep them from mining data from the site.

class badbots {
  ultron
  sentinels
  doombots
  spider slayers
  nimrod
  deathlok
  master mold
}

when CLIENT_ACCEPTED {
 set myPool [LB::server pool]
}
when HTTP_REQUEST {
 if { [matchclass [string tolower [HTTP::header User-Agent]] contains $::badbots] } {
 drop
 log -noname local0. "Robot '[HTTP::header User-Agent]' blocked. IP of agent is [IP::client_addr]"
 } else {
 pool $myPool
# log -noname local0. "Robot '[HTTP::header User-Agent]' Allowed."
 }
}

The badbots class is a simple datagroup containing user-agents from your web server logs that one would consider abusive. When creating the class, I created the datagroup using all lowercase values for the user-agent. The set myPool [LB::server pool] statement sets the rule to use the default pool assigned to your VIP so you can refer to it later on in the rule. The [matchclass [string tolower [HTTP::header User-Agent]] line converts the incoming user-agent to lowercase and then matches that agent to the list of bad 'bots in the datagroup. If it gets a match, issue a tcp reset on the connection; otherwise, let it through. Uncomment the second log statement temporarily for debugging purposes but fair warning: it will fill up your ltm log very quickly because that second statement logs all the user-agents that were allowed to pass by the irule.

The main advantage to an irule like this is that it is quick, easy, and effective but the main disadvantage is that anyone knowing LWP::UserAgent (or something similar) could cook up their own modified user-agent and bypass the rule fairly easily. However, they would need to realize that the reset they are being issued is due to the user-agent that they are using and not something like sniffing their IP address. I'm using this type of rule on some of my sites while I try to figure out how to get something more elaborate and cool working like a resource obfuscation irule.