ModSecurity for Amazon Linux AMI with Apache 2.4 and ELB

UPDATE:  Check out my online interactive tool for auditing ModSecurity log quickly.

I have been searching for a free open source solution to protect my web application against prying hackers, malicious screen scrapers, illegitimate crawlers, rampant bots and abusive API users.  Besides being free and open source, the minimum requirement is that the solution can identify rogue user IP addresses and blacklist them if necessary.  Preferably, the solution can also protect (somewhat) against denial-of-service (DOS) attack and implement API rate limiting.

My system requirements are:

• Amazon ELB
• EC2 Nodes
• Amazon Linux AMI
• Apache 2.4 (MPM Event)

Blacklist IP Addresses

Since my EC2 nodes are behind a load balancer, simple traditional solution like iptables which operates at the TCP layer will not work because it will pick up the IP address of the load balancer instead of the real user IP address which is being stored at the application layer in X-Forwarded-For of the HTTP header by the load balancer.

To identify and log rogue user IP addresses, you can modify Apache configuration file at /etc/conf/httpd.conf.  Edit the LogFormat lines to include "%{X-Forwarded-For}i".  For example, the edited lines may look like:

LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b" common

Then, restart Apache with sudo service httpd restart.

From now on, the user IP will be logged in the first field of each line of the access log at /var/log/httpd/access_log:

X.X.X.X, Z.Z.Z.Z 127.0.0.1 - - [16/May/2016:16:39:03 +0000] "POST / HTTP/1.1" 200 930 "-" "-"

With the user IP address available in the log, you can blacklist a user in different ways.  Without installation of additional modules or tools, you can blacklist manually by adding the IP address directly to the .htaccess file.  The following lines blacklist two IP addresses - X.X.X.X and Y.Y.Y.Y:

SetEnvIF X-Forwarded-For "(,| |^)X\.X\.X\.X(,| |$)" IP_BLACKLIST
SetEnvIF X-Forwarded-For "(,| |^)Y\.Y\.Y\.Y(,| |$)" IP_BLACKLIST
deny from env=IP_BLACKLIST

To automate blacklisting, you can install tools like Fail2Ban with sudo yum install fail2ban.  Fail2Ban will monitor the Apache log, extract the user IP and ban based on defined rules.  For more information, check out the numerous online materials available.

ModSecurity

There are a few free solutions available for DOS protection.  For example, there is the Apache module mod_evasive.  To the best of my knowledge, mod_evasive by itself does not work for a web server located behind a load balancer or proxy because it cannot access the user IP from the X-Forwarded-For header field.  The additional installation of mod_rpaf is required to bypass the limitation.  At the time of writing, the only way to install both modules on Amazon Linux for Apache 2.4 is to download and compile the source code.  Furthermore, mod_evasive is only compatible with Apache running in prefork mode so if your Apache is using MPM worker or event, you are out of luck.  To find out which mpm module Apache is using, check the configuration file at /etc/httpd/conf.modules.d/00-mpm.conf

The alternative solution I have been exploring is ModSecurity.  To install, run sudo yum install mod24_security.  ModSecurity is a web application firewall (WAF) designed to protect Apache, NGINX and IIS against common hacking exploits.  It works by examining web requests against a set of rules to identify malicious traffic pattern (e.g. HTTP header missing user-agent) and execute the corresponding actions (e.g. drop connection).  To make life easier, you can download a predefined set of generic attack detection rules called the OWASP ModSecurity Core Rule Set (CRS) via sudo yum install mod_security_crs.  You can take a look at what the rules look like at https://github.com/SpiderLabs/owasp-modsecurity-crs

The CRS rules are installed at /etc/httpd/modsecurity.d/activated_rules.  You may also add your own rules at /etc/httpd/modsecurity.d/local_rules.  Out of the box, the CRS rules will likely generate many false alarms for your particular website.  This means it will inadvertently shut your users off from your site if you are not careful.  For example, it may mistakenly identify a legitimate HTTP POST request with more than 255 parameters as an exploit even if your application expects it.

At the minimum, before you deploy ModSecurity to production use, find the following line from ModSecurity configuration file at /etc/httpd/conf.d/mod_security.conf

SecRuleEngine On

and change it to:

SecRuleEngine DetectionOnly

This sets ModSecurity to detection mode so it only reports potential exploits without enforcement.  Every time you make changes to the configuration or rules, you must restart Apache with sudo service httpd restart.  If everything goes well, your web application should function normally as before while ModSecurity checks every web requests and log potential problems to /var/log/httpd/modsec_audit.log.

You can control what information ModSecurity should log by editing the configuration file at /etc/httpd/conf.d/mod_security.conf

SecAuditLogParts ABHZ

For example, you can set it to log the request headers, request body, response headers, etc.  A log entry for a potentially malicious web request may look like this:

--61080530-B--
POST /index HTTP/1.1
host: my_website123.com
Accept: application/json, text/javascript, */*; q=0.01
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: https://my_website123/index
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
X-Requested-With: XMLHttpRequest
X-Forwarded-For: X.X.X.X
X-Forwarded-Port: 443
X-Forwarded-Proto: https
Content-Length: 12056
Connection: keep-alive

--61080530-H--
Message: Warning. Operator GT matched 255 at ARGS. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_23_request_limits.conf"] [line "31"] [id "960335"] [rev "2"] [msg "Too many arguments in request"] [severity "WARNING"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/POLICY/SIZE_LIMIT"]
Apache-Handler: application/x-httpd-php
Stopwatch: 1463421036197726 318695 (- - -)
Stopwatch2: 1463421036197726 318695; combined=64071, p1=199, p2=63680, p3=0, p4=0, p5=120, sr=67, sw=72, l=0, gc=0
Producer: ModSecurity for Apache/2.8.0 (http://www.modsecurity.org/); OWASP_CRS/2.2.8.
Server: Apache
Engine-Mode: "DETECTION_ONLY"

In this example, WebSecurity reports that a web request to https://my_website123/index originating from X.X.X.X violates one of the CRS rule for having more than 255 HTTP POST parameters.  The rule ID is 960335.

You can disable a particular rule by adding the following line to /etc/httpd/modsecurity.d/local_rules/modsecurity_localrules.conf

SecRuleRemoveById 960335

Removing a rule may weaken the firewall you are trying to build.  Instead, you can modify the rule to match your traffic pattern.  For the HTTP POST parameter limit violation, you can also increase the tx.max_num_args parameter value in /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf.

Denial of Service (DOS) Protection

Getting back to my original purpose of setting up a firewall, I can now use ModSecurity to blacklist IPs as well as protect (somewhat) against denial of service attack by using the anti-automation rule set modsecurity_crs_11_dos_protection.conf:

#
# Anti-Automation rule set for detecting Denial of Service Attacks. 
#
#
# Enforce an existing IP address block and log only 1-time/minute
# We don't want to get flooded by alerts during an attack or scan so
# we are only triggering an alert once/minute.  You can adjust how often
# you want to receive status alerts by changing the expirevar setting below.
#
SecRule IP:DOS_BLOCK "@eq 1" "chain,phase:1,id:'981044',drop,msg:'Denial of Service (DoS) Attack Identified from %{tx.real_ip} (%{tx.dos_block_counter} hits since last alert)',setvar:ip.dos_block_counter=+1"
SecRule &IP:DOS_BLOCK_FLAG "@eq 0" "setvar:ip.dos_block_flag=1,expirevar:ip.dos_block_flag=60,setvar:tx.dos_block_counter=%{ip.dos_block_counter},setvar:ip.dos_block_counter=0"

#
# Block and track # of requests but don't log
SecRule IP:DOS_BLOCK "@eq 1" "phase:1,id:'981045',t:none,drop,nolog,setvar:ip.dos_block_counter=+1"

#
# skipAfter Check
# There are different scenarios where we don't want to do checks -
# 1. If the current IP address has already been blocked due to high requests
# In this case, we skip doing the request counts.
#
SecRule IP:DOS_BLOCK "@eq 1" "phase:5,id:'981046',t:none,nolog,pass,skipAfter:END_DOS_PROTECTION_CHECKS"

#
# DOS Counter
# Count the number of requests to non-static resoures
# 
SecRule REQUEST_BASENAME "!\.(jpe?g|png|gif|js|css|ico)$" "phase:5,id:'981047',t:none,nolog,pass,setvar:ip.dos_counter=+1"

#
# Check DOS Counter
# If the request count is greater than or equal to user settings,
# we then set the burst counter
# 
SecRule IP:DOS_COUNTER "@gt %{tx.dos_counter_threshold}" "phase:5,id:'981048',t:none,nolog,pass,t:none,setvar:ip.dos_burst_counter=+1,expirevar:ip.dos_burst_counter=%{tx.dos_burst_time_slice},setvar:!ip.dos_counter"

#
# Check DOS Burst Counter and set Block
# Check the burst counter - if greater than or equal to 2, then we set the IP
# block variable for 5 mins and issue an alert.
#
SecRule IP:DOS_BURST_COUNTER "@ge 2" "phase:5,id:'981049',t:none,log,pass,msg:'Potential Denial of Service (DoS) Attack from %{tx.real_ip} - # of Request Bursts: %{ip.dos_burst_counter}',setvar:ip.dos_block=1,expirevar:ip.dos_block=%{tx.dos_block_timeout}"

SecMarker END_DOS_PROTECTION_CHECKS

You can tailor the burst detection pattern by editing the file at /etc/httpd/modsecurity.d/modsecurity_localrules.conf:

#
# -- [[ DoS Protection ]] ----------------------------------------------------------------
#
# If you are using the DoS Protection rule set, then uncomment the following
# lines and set the following variables:
# - Burst Time Slice Interval: time interval window to monitor for bursts
# - Request Threshold: request # threshold to trigger a burst
# - Block Period: temporary block timeout
#
SecAction \
  "id:'900015', \
  phase:1, \
  t:none, \
  setvar:'tx.dos_burst_time_slice=60', \
  setvar:'tx.dos_counter_threshold=100', \
  setvar:'tx.dos_block_timeout=600', \
  nolog, \
  pass"

ModSecurity is a powerful tool to protect web applications and as such it comes with a learning curve.   I have only touched on the basics in this blog entry.  Hopefully, I can devote some more blog time to it as I pick up the tool myself.

To develop your own tunnel vision quickly as I have been doing, I recommend taking a look at the official documentation at:

https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual