Apache: Error writing to log file. XXXXXXX messages lost

This particular error showed up recently over the past couple of weeks on one of the sites that I run. We have two apache web servers supporting this site that had been generating about 1.5GB worth of log file data on an average day, slowly growing over time but never appearing to climb anywhere above 1.8GB per day per server. Then one day recently while trying to troubleshoot a problem, doing a long directory listing on the directory showed that the log files were now between 300 and 500 MB for the most recent days. Opening the log, the first time-stamped request on one of these truncated daily logs showed that the first request occurred between 5:00PM and 6:00PM on that day but the first line in the log showed a message similar to the following:

Error writing to log file. XXXXXXX messages lost

Where XXXXXXX in our case was always some number in the 5.2 million to 5.3 million range. Disk space utilization on the volume was fine. In our case, the problem was load. Load increased on the site to the point where our daily logs files are now larger than 2GB per web server and rotatelogs on a Linux system will only write a 2GB log. Once the file hits 2GB, the log file is truncated. This only seems to occur on apache 1.3 and apache 2.0 running on Linux.

There are a couple of workarounds. The first is to rotate the log file more frequently. For example, set rotatelogs to rotate ever 43200 seconds (12 hours) instead of 86400. Alternatively, you can rotate based on filesize instead, like WebSEAL does by default. Another option is to take the rotatelogs executable from an apache 2.2 installation and use that instead. rotatelogs from apache 2.2 doesn't seem to have this problem.

I haven't used cronolog, so I can't say whether it has the same limitation or not.