ModSec v3+OWASP CRS3 On Kong Discussion/Guide

jeremyjpj0916 · February 4, 2020, 10:30pm

Just started dabbling in running Kong with ModSecurity v3 with the Nginx connector and the OWASP CRS3. Figured I would throw together some snippets of code that may help folks out that want to play around with WAF embedded in their own Kong deployments.

First, the code. Note I am building via a docker file with base image Alpine, running Kong 1.4.3 right now.

Some dependencies you may need(not all relate to the WAF but 3-4 of these do).

RUN apk add --no-cache --virtual build-deps wget tar  ca-certificates \
        && apk add --update --no-cache readline-dev outils-md5 linux-headers bsd-compat-headers m4 \
	yaml-dev build-deps bash patch bash-completion automake bash-doc make gcc g++ libgcc zlib-dev curl curl-dev autoconf libtool perl pcre libxml2 libxml2-dev libmaxminddb-dev yajl yajl-dev pcre-dev unzip tzdata wrk luarocks git

Then the building of libs and some basic settings/files:

######------ ModSec NGINX WAF ------######

RUN git clone --depth 1 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity \
    && chmod -R 777 /ModSecurity \
    && cd ModSecurity \
    && git submodule init \
    && git submodule update \
    && ./build.sh \
    && ./configure \
    && make \
    && make install \
    && mv /ModSecurity/modsecurity.conf-recommended /ModSecurity/modsecurity.conf
    # ^^^ Rename the conf file to be proper ^^^

#Return to ROOT DIR
RUN cd /

#NGINX ModSec v3 connector dynamic module
RUN git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git \
    && chmod -R 777 /ModSecurity-nginx
    
#OWASP CORE RULE SET, PARANOIA LEVEL 2
RUN wget https://github.com/SpiderLabs/owasp-modsecurity-crs/archive/v3.2.0.tar.gz \
    && tar -xzvf v3.2.0.tar.gz \
    && mv owasp-modsecurity-crs-3.2.0 /usr/local \
    && mv /usr/local/owasp-modsecurity-crs-3.2.0/crs-setup.conf.example /usr/local/owasp-modsecurity-crs-3.2.0/crs-setup.conf \
    && echo 'SecAction "id:900000, phase:1, nolog, pass, t:none, setvar:tx.paranoia_level=2"' >> /usr/local/owasp-modsecurity-crs-3.2.0/crs-setup.conf \
    && chmod -R 777 /usr/local/owasp-modsecurity-crs-3.2.0
    
# Create combined config file
RUN touch /usr/local/modsec_includes.conf \
    && echo "include /ModSecurity/modsecurity.conf" >> /usr/local/modsec_includes.conf \
    && echo "include /usr/local/owasp-modsecurity-crs-3.2.0/crs-setup.conf" >> /usr/local/modsec_includes.conf \
    && echo "include /usr/local/owasp-modsecurity-crs-3.2.0/rules/*.conf" >> /usr/local/modsec_includes.conf \
    && chmod 777 /usr/local/modsec_includes.conf
    
# Make modifications to config
RUN sed -i "s/SecRuleEngine DetectionOnly/SecRuleEngine On/" /ModSecurity/modsecurity.conf \
    && sed -i "s/SecAuditLogType Serial/SecAuditLogType Concurrent/" /ModSecurity/modsecurity.conf \
    && sed -i "s|SecAuditLog /var/log/modsec_audit.log|SecAuditLog /usr/local/kong/logs/modsec_audit.log|" /ModSecurity/modsecurity.conf

######------ END OF ModSec NGINX WAF ------######

Then its worth noting, I am still using the kong openresty-build-tools deprecated lib for now, which allowed the adding of a 3rd party module with ease in its scripting, I am sure the newer repo offers that too . Note the --add-module /ModSecurity-nginx line, as my new additional arg.

RUN git clone https://github.com/kong/openresty-build-tools
RUN rm openresty-build-tools/kong-ngx-build
COPY kong-ngx-build openresty-build-tools/kong-ngx-build
RUN chmod -R 777 /openresty-build-tools
COPY patches/ngx/* /patches/
RUN cd openresty-build-tools \
    && mkdir work \
    && ./kong-ngx-build -p $KONG_PATH \
        --openresty $RESTY_VERSION \
        --openssl $RESTY_OPENSSL_VERSION \
        --luarocks $RESTY_LUAROCKS_VERSION \
	--add-module /ModSecurity-nginx \
        #--pcre $PCRE_VERSION \
	--force

Lastly in the nginx.conf produced by Kong you will want to ensure to include these within the server or location block of the conf so ModSec will run on your kong server inbound requests:

  server {
      server_name kong;
      modsecurity on;
      modsecurity_rules_file /usr/local/modsec_includes.conf;

Which enables ModSec and then a link to your file that contains rules/config references.

Then we thought up a number of threat vectors we wanted to validate security on. Some of the first that came to mind we check-listed against:

• Cross site scripting TESTED & VALIDATED IN P2 OWASP
• Jumbo Payload TESTED & VALIDATED IN P2 OWASP
• XML/JSON bomb (infinitely recursive data) TESTED & VALIDATED IN P2 OWASP
• Rate limiting (Kong does this)
• SQL Injection TESTED & VALIDATED IN P2 OWASP

So some examples of our tests to ensure ModSec was catching things was:

XML Bomb Payload:

<?xml version="1.0"?>
<!DOCTYPE lolz [
 <!ENTITY lol "lol">
 <!ELEMENT lolz (#PCDATA)>
 <!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
 <!ENTITY lol2 "&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
 <!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
 <!ENTITY lol4 "&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;">
 <!ENTITY lol5 "&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;">
 <!ENTITY lol6 "&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;">
 <!ENTITY lol7 "&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;">
 <!ENTITY lol8 "&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;">
 <!ENTITY lol9 "&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;">
]>
<lolz>&lol9;</lolz>

XSS

https://gateway.com/api/anonymous/user/test/v1?q=flowers+%3Cscript%3Eevil_script()%3C/script%3E

SQL INJECTION:

curl --request POST \
  --url https://gateway.com/ \
  --header 'content-type: application/x-www-form-urlencoded' \
  --cookie dtCookie=36%24946B53A8A942C3A95C28C4CA0FD72AD2 \
  --data 'item1=a'\'';DROP TABLE users; SELECT * FROM userinfo WHERE '\''t'\'' '

And well JUMBO payloads you get the point here, didn’t setup a remote XML doc that was massive though:

github.com

OWASP/CheatSheetSeries/blob/master/cheatsheets/XML_Security_Cheat_Sheet.md#jumbo-payloads

# Introduction

Specifications for XML and XML schemas include multiple security flaws. At the same time, these specifications provide the tools required to protect XML applications. Even though we use XML schemas to define the security of XML documents, they can be used to perform a variety of attacks: file retrieval, server side request forgery, port scanning, or brute forcing. This cheat sheet exposes how to exploit the different possibilities in libraries and software divided in two sections:

- **Malformed XML Documents**: vulnerabilities using not well formed documents.
- **Invalid XML Documents**: vulnerabilities using documents that do not have the expected structure.

# Malformed XML Documents

The W3C XML specification defines a set of principles that XML documents must follow to be considered well formed. When a document violates any of these principles, it must be considered a fatal error and the data it contains is considered malformed. Multiple tactics will cause a malformed document: removing an ending tag, rearranging the order of elements into a nonsensical structure, introducing forbidden characters, and so on. The XML parser should stop execution once detecting a fatal error. The document should not undergo any additional processing, and the application should display an error message.

The recommendation to avoid these vulnerabilities are to use an XML processor that follows W3C specifications and does not take significant additional time to process malformed documents. In addition, use only well-formed documents and validate the contents of each element and attribute to process only valid values within predefined boundaries.

## More Time Required

A malformed document may affect the consumption of Central Processing Unit (CPU) resources. In certain scenarios, the amount of time required to process malformed documents may be greater than that required for well-formed documents. When this happens, an attacker may exploit an asymmetric resource consumption attack to take advantage of the greater processing time to cause a Denial of Service (DoS).

To analyze the likelihood of this attack, analyze the time taken by a regular XML document vs the time taken by a malformed version of that same document. Then, consider how an attacker could use this vulnerability in conjunction with an XML flood attack using multiple documents to amplify the effect.

## Applications Processing Malformed Data

This file has been truncated. show original

And a simple example of caught attack:

curl http://localhost:8000?exec=/bin/bash

And what shows up in your logging too:

FINAL THOUGHTS:

Final thoughts after this initial play around come to my next steps, which involve how to harden and minimize the rule-set for checks specific to the API Gateway space and relevant attack vectors, as well as productionalizing this WAF and minimizing the FP(false positives) associated with legitimate API transactions.

I began to ask a few q’s in Slack channels around modsec and owasp to gain more insight, these were what some folks had to say:

How to minimize rules quickly:

coderpatros 1 day ago
The rules have different tags.
So if you don’t use php you can SecRuleRemoveByTag “language-php” to remove a bunch of rules
you don’t need. Someone here might be able to link to a definitive list. But doing a find all
for “platform-” “language-” and “attack-” in the core rule set should bring them all back.

On the discussion of minimizing the rule-set for API Gateway use case:

dune73 10:25 AM (https://twitter.com/ChrFolini - Author of the #ModSecurity Handbook 2ed. OWASP @CoreRuleSet project co-lead and program chair @SwissCyberStorm.)
Jeremy, what you are probably looking for is a cut down version of rules that cause even less
FPs than PL1. We are thinking about such a thing, maybe developing it together with some CDNs
and internet hosters. It might be a rule exclusion package, or (new idea) a paranoia level 0.
For the time being, I would run PL1, don’t edit the rule set yourself, but disable the parts
of the ruleset you do not like / need with SecRuleRemoveById 9xxxxx-9yyyyy.

FYI: I’m here at OWASP AppSecCali in Santa Monica. Yesterday I had a conversation with the new WAF teamlead from XYZ BIG CO and somebody from ABC BIG CO who works very closely with the WAF team. XYZ BIG CO runs now close to 100 Tbit/s over CRS. Not everything is blocking of course, and users need to pay to get access to the reports. But they run everything through CRS3. They are open to run experimental rules for us (to gauge the FP rate!) and they are also willing to share FPs with us. It sounds too good to be true. ABC BIG CO is more conservative, but they still use 10% of CRS3 on their traffic. Both companies agree that they combined control > 2/3 of the global internet traffic.
Cloudflare is just strong with the number of domains due to their free plan. But if this is about streaming the Olympics, it’s those guys. I’m a wee bit impressed.

My other thoughts would be how cool it would be someday to have OS Kong options to embed/enable the ModSec v3 with OWASP CRS3 or some trimmed down Kong approved rulset to run on Kong instances. This could all be done with Kong branded environment variables as is tradition and let Kong engineers do the heavy lifting with the build process in the background! Best OS API Gateway meets most adopted OS WAF+RuleSet in harmony ?

References:

https://coreruleset.org/faq/
https://owasp.slack.com/

Thanks for reading!

jeremyjpj0916 · February 4, 2020, 7:29am

Will note, we initially saw some really major performance hits with the OOTB configurations. Even after tuning you can see latency results here from some performance tests (also in that thread you can see many of my current tunings which may be of use to you):

My current opinion is that the latency/throughput hits are a downer, but not high enough necessarily to shake my interest in throwing some weight and continuing to leverage the application. I do hope subsequent releases may improve upon performance at the WAF layer or something more lightweight can be tuned further to support the API Gateway use case to block malicious traffic from REST/SOAP/Websocket/XYZ future data services.

jeremyjpj0916 · February 11, 2020, 8:53am

This will likely be my last post on findings. Wrapping up, we wanted a way to dynamically set properties of our WAF upon deploy time. Lucky for us, Kong has nice built in parsing of the nginx conf and a logic in place for ENV variables being able to override those conf static fields upon Kong’s deployment in a given environment.

In our case we took the Kong defaults lua file:

github.com

Kong/kong/blob/master/kong/templates/kong_defaults.lua

return [[
prefix = /usr/local/kong/
log_level = notice
proxy_access_log = logs/access.log
proxy_error_log = logs/error.log
admin_access_log = logs/admin_access.log
admin_error_log = logs/error.log
status_access_log = off
status_error_log = logs/status_error.log
plugins = bundled
go_pluginserver_exe = /usr/local/bin/go-pluginserver
go_plugins_dir = off
anonymous_reports = on

proxy_listen = 0.0.0.0:8000 reuseport backlog=16384, 0.0.0.0:8443 http2 ssl reuseport backlog=16384
stream_listen = off
admin_listen = 127.0.0.1:8001 reuseport backlog=16384, 127.0.0.1:8444 http2 ssl reuseport backlog=16384
status_listen = off
cluster_listen = 0.0.0.0:8005
cluster_control_plane = 127.0.0.1:8005

This file has been truncated. show original

Added in a few extra fields for some defaults (Otherwise the environment variables won’t take effect):

waf_mode = On
waf_request_file_size_limit = 25000000
waf_request_no_file_size_limit = 20000000
waf_pcre_match_limit = 10000
waf_pcre_match_limit_recursion = 10000
waf_debug_level = 0
waf_audit = Off
waf_paranoia_level = 1

Then what you can do is leverage environment variables via these fields:

KONG_WAF_MODE = On || Off || DetectionOnly
KONG_WAF_REQUEST_FILE_SIZE_LIMIT = 25000000 , for 25MB files
KONG_WAF_REQUEST_NO_FILE_SIZE_LIMIT = 20000000 , for 20 MB payloads(no file in payload)
KONG_WAF_PCRE_MATCH_LIMIT = 1000
KONG_WAF_PCRE_MATCH_LIMIT_RECURSION = 1000
KONG_WAF_DEBUG_LEVEL = 0 for off, 9 for on basically
KONG_WAF_AUDIT = RelevantOnly || On || Off
KONG_WAF_PARANOIA_LEVEL can be 1 || 2 || 3 || 4

And last but not least, you will need to add to the custom Kong template these references in the Mod Security module directive:

  server {
      server_name kong;
      modsecurity on;
      modsecurity_rules_file /usr/local/modsec_includes.conf;
      
      modsecurity_rules '
        SecRuleEngine ${{WAF_MODE}}
        SecRequestBodyLimit ${{WAF_REQUEST_FILE_SIZE_LIMIT}}
        SecRequestBodyNoFilesLimit ${{WAF_REQUEST_NO_FILE_SIZE_LIMIT}}
        SecPcreMatchLimit ${{WAF_PCRE_MATCH_LIMIT}}
        SecPcreMatchLimitRecursion ${{WAF_PCRE_MATCH_LIMIT_RECURSION}}
        SecDebugLogLevel ${{WAF_DEBUG_LEVEL}}
        SecAuditEngine ${{WAF_AUDIT}}
        SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=${{WAF_PARANOIA_LEVEL}}"
      ';

This enables us to quickly enable/disable/debug/tune the WAF without having to go through and complete a new build every deployment.

Hope this thread helps serve as a starter for those who want an Open Source WAF solution running on Kong!

mkopeyka · June 10, 2020, 3:52pm

Hi jeremyjpj0916, I use official docker image of Kong EE

registry.docker.private.linksynergy.com/kong-ee:v1.5.0.1-alpine

I’m trying to include ModSec v3 via Dokerfile but it I have this error message

nginx: [emerg] unknown directive “modsecurity” in nginx-kong.conf:66

Do you override some existing docker image of Kong or build Kong from scratch? What command do you use in your Docketfile to start Kong with ModSec?

jeremyjpj0916 · June 10, 2020, 3:57pm

@mkopeyka Yes I build Nginx/OpenResty from source to accomplish this. Specifically these lines are important, as its where adding ModSec v3 as an nginx module is required:

jeremyjpj0916:

RUN cd openresty-build-tools \
    && mkdir work \
    && ./kong-ngx-build -p $KONG_PATH \
        --openresty $RESTY_VERSION \
        --openssl $RESTY_OPENSSL_VERSION \
        --luarocks $RESTY_LUAROCKS_VERSION \
	--add-module /ModSecurity-nginx \
        #--pcre $PCRE_VERSION \
	--force

I am not sure how Kong Enterprise approaches customers that want to build nginx/openresty from source and then apply Kong EE to that, I would touch base with your Kong support team and can hopefully get a technical engineer to reference this forum post and may be able to help guide an implementation approach for EE clients.

mkopeyka · June 10, 2020, 9:52pm

I use this cmd to start kong

/usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c /usr/local/kong/nginx.conf

If I use “–force” option kong doesn’t start because of these errors:

Error loading shared library libopentracing.so.1: No such file or directory (needed by /usr/local/openresty/nginx/sbin/nginx)
Error relocating /usr/local/openresty/nginx/sbin/nginx: _ZN11opentracing2v26Tracer6GlobalEv: symbol not found
Error relocating /usr/local/openresty/nginx/sbin/nginx: _ZN11opentracing2v226propagation_error_categoryEv: symbol not found
Error relocating /usr/local/openresty/nginx/sbin/nginx: _ZN11opentracing2v26Tracer10InitGlobalESt10shared_ptrIS1_E: symbol not found
Error relocating /usr/local/openresty/nginx/sbin/nginx: _ZN11opentracing2v229DynamicallyLoadTracingLibraryEPKcRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE: symbol not found
Error relocating /usr/local/openresty/nginx/sbin/nginx: _ZN11opentracing2v227dynamic_load_error_categoryEv: symbol not found
Error relocating /usr/local/openresty/nginx/sbin/nginx: _ZN11opentracing2v229tracer_factory_error_categoryEv: symbol not found
Error relocating /usr/local/openresty/nginx/sbin/nginx: _ZN11opentracing2v23ext17sampling_priorityE: symbol not found

If I don’t use “–force” option kong doesn’t start because of this error:

nginx: [emerg] unknown directive “modsecurity” in /usr/local/kong/nginx-kong.conf:66

jeremyjpj0916 · November 23, 2020, 9:14pm

Sorry if I was a little late to responding here, hopefully you figured it out, or an alternative approach to your WAF needs, but this error indicates the module was never properly installed for NGINX to recognize it. Such as the installation step from source that does this:

RUN cd openresty-build-tools \
    && mkdir work \
    && ./kong-ngx-build -p $KONG_PATH \
        --openresty $RESTY_VERSION \
        --openssl $RESTY_OPENSSL_VERSION \
        --luarocks $RESTY_LUAROCKS_VERSION \
	--add-module /ModSecurity-nginx \
        #--pcre $PCRE_VERSION \
	--force

Which points at my /ModSecurity-nginx local dir containing my installed ModSec v3 stuff. See my original docker file in the post sample to see how the file structure is setup with what repo’s are leveraged.