ModSecurity Failed Deleting Collection

Houston, we have a problem.  Looking at my ModSecurity audit log, I found quite a few log entries similar to the following (I have obfuscated the actual IP address for privacy reason):

Failed deleting collection (name "ip", key “X.X.X.X_"): Internal error

I was concerned that if the collection file was not being cleaned up automatically by Apache or ModSecurity, the file may grow to an astronomical size over time causing other problems.  While I wasn't able to find a definitive cause for the error by searching online, I stumbled upon 3 proposed workaround solutions:

1. Use memcache for collections
2. Set SecCollectionTimeout to a small value such as 600 (default is 3600 seconds)
3. Install and run modsec-sdbm-util in a separate process to clean up the collection file regularly

Option 1 requires ModSecurity 3.0 or git-clone the memcache_collections branch, a choice I didn't want to make hastily.  There are reports that option 2 may not always work for everybody.  Just to be on the safe side, I implemented option 3 in addition to option 2.

For the record, I took the steps below to install modsec-sdbm-util on Amazon Linux AMI running Apache 2.4 with ModSecurity 2.8.

First, install all pre-requisite libraries and tools:

sudo su
yum install libtool autoreconf autoheader automake autoconf apr-util-devel

Then, download and install modsec-sdbm-util to a directory.

git clone https://github.com/SpiderLabs/modsec-sdbm-util.git
cd modsec-sdbm-util
./autogen.sh
./configure
make install

Check that it’s installed successfully by running (assuming ip.pag file is in /tmp):

/usr/local/modsecurity/bin/modsec-sdbm-util -s /tmp/ip.pag

If everything is okay, it should output a status report similar to:

Opening file: /tmp/ip.pag
Database ready to be used.
 [|] 10 records so far.
Total of 17 elements processed.
0 elements removed.
Expired elements: 7, inconsistent items: 0
Fragmentation rate: 41.18% of the database is/was dirty data.

Set up a cron job to run modsec-sdbm-util every half hour or so to remove expired elements from the collection file.

*/30 * * * *  /usr/local/modsecurity/bin/modsec-sdbm-util -k /tmp/ip.pag &> /dev/null

This should do it!  (Cross my fingers.)