Audit ModSecurity Log Quickly and Systematically with Reconity

Having recently installed ModSecurity as my web application firewall, I started to keep an eye on the audit logs generated by ModSecurity regularly.  The audit log records web access events which had set off any of the configured firewall rules.  For example, an event entry in the log may look this (IP addresses have been masked for privacy):

--d9g76d43-A--
[27/Jun/2016:08:18:10 +0000] V3GlK2-sf88lhesfakqlUgAAI X.X.X.X 57596 Y.Y.Y.Y 80
--d9g76d43-B--
GET /../../../../../../../mnt/mtd/OCxW HTTP/1.1
Host: Z.Z.Z.Z
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36
Accept-Encoding: gzip
Connection: close

--d9g76d43-F--
HTTP/1.1 400 Bad Request
X-Content-Type-Options: nosniff
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1

--d9g76d43-H--
Message: Warning. String match "Invalid URI in request" at WEBSERVER_ERROR_LOG. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "82"] [id "981227"] [rev "1"] [msg "Apache Error: Invalid URI in Request."] [data "GET /../../../../../../../mnt/mtd/OCxW HTTP/1.1"] [severity "WARNING"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"]
Apache-Error: [file "core.c"] [line 4306] [level 3] AH00126: Invalid URI in request %s
Stopwatch: 1467015490548268 657 (- - -)
Stopwatch2: 1467015490548268 657; combined=435, p1=213, p2=0, p3=1, p4=41, p5=131, sr=92, sw=49, l=0, gc=0
Producer: ModSecurity for Apache/2.8.0 (http://www.modsecurity.org/); OWASP_CRS/2.2.8.
Server: Apache
Engine-Mode: "ENABLED"

--d9g76d43-Z--

The above entry tells us that one of the installed rules caught and blocked an illegitimate attempt to access private and protected resource on the server.  According to the entry, the source of the intrusion was X.X.X.X, the offending request was "GET /../../../../../../../mnt/mtd/OCxW", the response was "400 Bad Request" and the rule in violation was 981227.

Often, my audit log also contains false alarms or events pertaining to the internal workings of ModSecurity rather than an external offense such as:

--9fj387gd-A--
[13/Jun/2016:15:48:05 +0000] V11sVtSgC7wd3k4LMd8eSAXAAAg 127.0.0.1 52076 127.0.0.1 80
--2ab87c12-B--
POST /foo HTTP/1.1
Host: localhost

--9fj387gd-F--
HTTP/1.1 200 OK
X-Content-Type-Options: nosniff
Content-Type: application/json; charset=utf-8

--9fj387gd-H--
Message: collections_remove_stale: Failed deleting collection (name "ip", key "X.X.X.X_c7ad53c03q60y9cdrf1e71t5e4k04bx21e589z6e"): Internal error
Apache-Handler: application/x-httpd-php
Stopwatch: 1465832885236582 714358 (- - -)
Stopwatch2: 1465832885236582 714358; combined=4576, p1=259, p2=0, p3=0, p4=0, p5=2162, sr=104, sw=2, l=0, gc=2153
Producer: ModSecurity for Apache/2.8.0 (http://www.modsecurity.org/); OWASP_CRS/2.2.8.
Server: Apache

--9fj387gd-Z--

This particular event is harmless and is caused by a bug in ModSecurity as discussed in a previous article.

A typical log of mine contains a garden variety of events like the ones above.  When the log is big, sifting through it becomes time consuming especially if I want to examine every single recorded event out of paranoia.  What I needed was a simple tool for exploratory analysis of the log.  Being a software engineer, I decided to spend some time building a tool to my specification.

And so born Reconity, an online interactive log auditing tool for ModSecurity.  The tool extracts key event information from part A, B, F and H of the log into an interactive table so I can inspect events quickly.  With a single mouse click, I can hide events like the aforementioned internal errors from view and switch my attention over to other more serious events.  The tool was built in mind to help audit log systematically by narrowing down important events quickly.

I wrote the tool mostly in JavaScript so that it can run in a browser.  There is no manual to learn or read as the web user interface is intuitive.  Since processing is done within the browser, there is no need to upload log file to a remote machine for processing or invoke commands at the command line.  This means sensitive log data remains private and local to the browser computer.  While JavaScript may not be the fastest programming language out there, I was able to audit 100,000 events (~200 MB log file) responsively with the tool.  With optimization, I should be able to extend the limit further if my log ever get bigger than 200 MB.

As I have been using the tool for weeks now, I find that it saves time for me from the laborious task of log auditing.  If auditing ModSecurity log is one of your regular routines, you are welcome to simplify the chore with Reconity!

Web Application Vulnerability Scanners

In a previous article, I discussed how to set up ModSecurity for Apache 2.4 on Amazon Linux AMI to protect web applications against exploits and abuses.  Having set up ModSecurity myself for my own web application recently, I was curious how my ModSecurity firewall would protect against publicly available hacking tools out there.  I was in for a learning curve as I learned about what's out there.

IMPORTANT:  The scanning tools introduced below may choke your web application with heavy traffic so proceed with caution and permission!

Kali Linux

So what's the easiest way for a layman to start?  I discovered the path with least resistance is to download Kali Linux.  Kali is a freely distributed Debian-based Linux system pre-loaded with many vulnerability scanners and hacking tools which are what I was after.  Since I already use VirtualBox, the quickest way to get it up and running is to download the Kali Linux virtual box image from:

https://www.offensive-security.com/kali-linux-vmware-virtualbox-image-download/

The image is about 2 GB so be patient.  If you prefer the Kali Linux ISO image instead, it's also available at:

https://www.kali.org/downloads/

Once you have downloaded the VirtualBox image, you can create a new virtual machine in VirtualBox by going to File > Import Appliance.  It took a few minutes to create the virtual machine, for example, on a Mac Pro.

Now, launch the new virtual machine and log in as root.  The default root password is "toor".  There are many tools to choose from.  Here's a screenshot of what's under the Application menu.

Nikto

One of the scanning tools under Application > Vulnerability Analysis is Nikto.  From the official website:

"Nikto is an Open Source (GPL) web server scanner which performs comprehensive tests against web servers for multiple items, including over 6700 potentially dangerous files/programs, checks for outdated versions of over 1250 servers, and version specific problems on over 270 servers. It also checks for server configuration items such as the presence of multiple index files, HTTP server options, and will attempt to identify installed web servers and software. Scan items and plugins are frequently updated and can be automatically updated."

For usage information, please refer to the Nikto official website at https://cirt.net/nikto2-docs/

The command to start a Nikto scan is nikto -h www.my_website123.com.  While scanning, Nikto reports potential vulnerabilities to the command console as they are found:

- Nikto v2.1.6
---------------------------------------------------------------------------
+ Target IP:          X.X.X.X
+ Target Hostname:    www.my_website123.com
+ Target Port:        80
+ Start Time:         2016-05-17 12:27:25 (GMT-4)
---------------------------------------------------------------------------
+ Server: Apache
+ Retrieved x-powered-by header: PHP/4.1.1
+ The anti-clickjacking X-Frame-Options header is not present.
+ The X-XSS-Protection header is not defined. This header can hint to the user agent to protect against some forms of XSS
+ The X-Content-Type-Options header is not set. This could allow the user agent to render the content of the site in a different fashion to the MIME type

Different scanners may discover new problems other scanners have not reported so you should try to run a few of them.  It's also very likely that some of the reported problems are false positives or not exploitable by hackers so don't panic.

OpenVAS

There are also other scanners not bundled with Kali Linux   One of them is OpenVAS.  To install OpenVAS on Kali Linux, please refer to:

https://www.kali.org/penetration-testing/openvas-vulnerability-scanning

If your scan finish quickly (<1 minute) and returns without any problems found, chances are your server isn't responding to ICMP requests instead of being absence of vulnerabilities.  You should set "Alive Test" to "Consider Alive" for your scan target.

On the other hand, if your scan finish quickly (<1 minute) and returns with an "Internal Error" and you are using OpenVAS Manager 6.0.5, it's a bug.  Find out which version you have at the terminal by running

openvasmd --version

Then, check if the error log at /var/log/openvas/openvasmd.log shows:

md  main:WARNING:2016-05-17 19h06.25 UTC:26326: sql_prepare_internal: sqlite3_prepare failed: no such table: current_credentials

To fix this, manually add the missing table to the sqlite3 tasks.db file:

CREATE TABLE IF NOT EXISTS current_credentials (id INTEGER PRIMARY KEY, uuid text UNIQUE NOT NULL);

Other Tools

In addition to OpenVAS, there are also other scanners readily available to download and install. Some of the ones I have found are:

Arachni  -  apt-get install arachni
Vega  -  apt-get install vega

At the time of writing, Vega is packaged with a graphical user interface but not Arachni.  If you're trying Arachni, note that a scan may take a long time to finish (as in days or longer) because of its comprehensive nature but it's possible to optimize for faster scans.

Hopefully, this will get you started on securing your web application.  Cheers!