Drupal: One Fix for 403-Forbidden Errors from RSS Aggregators

While working with a colleague today on one of his personal sites, we came across an issue with a Drupal-managed RSS feed where some RSS Aggregators were receiving 403-Forbidden status codes. After QUITE A BIT of poring through all the configuration and .htaccess files included with Centos and Drupal, our focus kept returning to the following section in his httpd.conf, which turned out to be the culprit (bolded below):

<IfModule mod_rewrite.c>
 RewriteEngine on
 RewriteRule ^/$ /home/ [R]
 RewriteCond %{HTTP_USER_AGENT} http://.*\.com
 RewriteRule . - [F]
 RewriteCond %{REQUEST_METHOD} !^(GET|POST|HEAD)$ 
 RewriteRule .* - [F]
 RewriteRule ^/$ home/$1 [R=301,L]
</IfModule>

What this states is that if the User-Agent connecting to a site contains a .com URL, forbid access. Examples of User-Agents that stuff a URL inside the user-agent string are: msnbot/2.0b (+http://search.msn.com/msnbot.htm) or in the case of my friend: Yahoo's FeedSeeker, which uses the following user-agent string: YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide).

Neither one of us could figure out why this particular Rewrite Condition was in place—it seems like it is a bit too generic as a badbot killer. Getting rid of this condition and rule fixes the problem with RSS Aggregators getting a 403-Forbidden but does wind up leaving the site exposed to those bad robots and site rippers so it seems to make sense to replace the rule with something like the following:

RewriteEngine on
 RewriteRule ^/$ /home/ [R]
 RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:[email protected] [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Custo [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^HMView [OR] 
 RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR] 
 RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR] 
 RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^larbin [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Wget [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Widow [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR] 
 RewriteCond %{HTTP_USER_AGENT} ^Zeus 
 RewriteRule ^.* - [F,L]
 RewriteCond %{REQUEST_METHOD} !^(GET|POST|HEAD)$ 
 RewriteRule .* - [F]
 RewriteRule ^/$ home/$1 [R=301,L]

The Rewrite Conditions above come from many, many different SEO-related links that resulted from a google search. Search your favorite search engine for "RewriteCond %{HTTP_USER_AGENT}" and you should get pages and pages of results.