ModSecurity Advanced Topic of the Week: Detecting Malware with Fuzzy Hashing

In the most recent release of ModSecurity v2.9.0-RC1, we introduced a new operator called @fuzzyHash which uses functionality from the ssdeep tool.  This blog post will demonstrate a powerful use-case with ModSecurity which is identifying malware in web attachment uploads.

What is ssdeep (Fuzzy Hashing)?

The ssdeep project gives the following information:

ssdeep is a program for computing context triggered piecewise hashes (CTPH). Also called fuzzy hashes, CTPH can match inputs that have homologies. Such inputs have sequences of identical bytes in the same order, although bytes in between these sequences may be different in both content and length.

A complete explanation of CTPH can be found in Identifying almost identical files using context triggered piecewise hashing from the journal Digital Investigation. There is a free version of this paper available through the Digital Forensic Research Workshop conference, free version of Identifying almost identical files using context triggered piecewise hashing.

There are some usage scenarios in the Quickstart guide and the Forensics Wiki entry on ssdeep.

The best example of practical fuzzy hashing usage is during forensic investigations when malware analysts want to compare unknown files with known malware families.  Even if the files are different, fuzzy hashing can let you know what malware groups it is similar too and to what degree (%).  Here is link to a previous blog post written by my SpiderLabs Research colleagues who analyze malware where they show ssdeep usage.

Detecting Web-based Malware

When dealing with web application defense, web-based malware such as webshells and backdoors are a huge problem.  Here is an example screen shot of a common webshell:

Here is a quick listing of the top categories of web-malware that we identified in a previous Global Security Report:

Attackers can use many different attack methods to get this malware uploaded to a target website including Remote File Inclusion (RFI) attacks but often times there are file uploading flaws in legitimate application software that can be abused.  This is where our use-case of using the new @fuzzyHash operator in ModSecurity comes into play.  

Fuzzy hashing is so useful in this scenario because standard cryptographic hash testing quickly fails as script kiddies are constantly modifying their code.  Granted, most of the code changes are cosmetic as they rip off each other's code and simply rename it or slap their hacker handle on it and give gr33tz to other kiddies.  These changes, however, render normal hash checks (like with md5 or sha1) almost useless.

Install ssdeep

Before turning to ModSecurity, the first step in this process is to install ssdeep on your web server host.  You can install from source or you can use one of the 3rd party OS repositories (example using - $ apt-get install ssdeep).  Once ssdeep is installed, you can then configure ModSecurity.

Create Fuzzy Hash Listing of Malware

We have captured thousands of webshells/backdoors as part of our research efforts in SpiderLabs.  With this archive, we are then able to use ssdeep to create a text file listing of fuzzy hashes.  Here is  a small example -

$ ssdeep -r -b * > webshells-ssdeep.txt ; head webshells-ssdeep.txt
ssdeep,1.1--blocksize:hash:hash,filename
24:OV9QJXQILgD9C7SwKjB8y4skRbsL4J7S52eYwsTdhd/4:QQN7SwAEvRbF7S02sTHp4,"529.php"
192:wUYrBEgQugt1Rv/2gptgSCF4iemhwZt4MWRHOE5wbyzvE1EbKUuHFHgXyc:wUYrBAeH5IU0owbWmEb0HFH3c,"AK-74 Security Team Web Shell Beta Version.php"
768:FpT8+vA/F+OxCDy1L8zWHdzD/xMm/f7+CiA:F3Y/EOxCDy140dzDpMif7+bA,"Antichat Shell. Modified by Go0o$E.php"
192:NQtjv0bj7tJ4q89i8V7y+Q7cmSUwNXMEKhsD2O0b6RWUE:Nej2jL4qinVW++cQwNXMLhsDv0w2,"Antichat Shell.php"
768:O0bU4SbqX+4jl7EBNRjal0OC5eVdVIhyTuIqYDvNTbervZWhVZ21wNX2u89VACrn:FbRSbqOMl7WtaK5ejVeI9zNvoWJD899R,"C99madShell v. 2.0 madnet edition.php"
192:NPgGy+ChwSFHkueyE+C/dw1U2Vk+EndG9X5Mf8x6p1PdFrrELtMIfiA7KcMYtv/d:NwhB8hmDDRMIfiA7KbYtv/d,"CasuS 1.5.php"
384:rIwHyY1gNv8yiVnggmS2TgFnGjmufsnoC0WgS+i3UgMGMqm7iF:rnHyegJHmggZ28Fc8oGBM7iF,"CrystalShell v.1.php"
384:FTOCnQnAJsznh9m2mwLTEtltCj2gQgrmUfycqfQNUvhdWQhv05Sh4Z6upKmM:HJwmUmiYXmM,"Cyber Shell (v 1.0).php"
384:GIr8nsnAJ0pncTyZhCmmg7ylEJKiIhyZzVxUdnA6abSczJMc84ZCrczRcmNbQ60x:EJ3yZhCE7y0bZP00Mj6G,"Cyber Shell.php"

Compile ModSecurity

During the configure step in building ModSecurity, it will try to identify the proper ssdeep source files installed on your system.

$ ./configure
checking for a BSD-compatible install... /opt/local/bin/ginstall -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /opt/local/bin/gmkdir -p
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
--CUT--
checking for libpcre config script... /usr/local/bin/pcre-config
configure: using pcre v8.32
checking for libapr config script... /usr/local/apr/bin/apr-1-config
configure: using apr v1.5.1
checking for libapu config script... /usr/local/apr/bin/apu-1-config
configure: using apu v1.5.4
checking for libxml2 config script... /usr/local/bin/xml2-config
checking if libxml2 is at least v2.6.29... yes, 2.7.8
configure: using libxml2 v2.7.8
checking for pkg-config... /usr/local/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for liblua config script... no
checking for lua install... /usr/local/lib /usr/local/include
configure: using lua v5.1
checking for libcurl config script... /usr/bin/curl-config
checking if libcurl is at least v... yes, 7.24.0
checking if libcurl is linked with gnutls... no
configure: using curl v7.24.0
checking for libyajl config script... /usr/local/bin/pkg-config
configure: using yajl v2.0.5
checking for ssdeep path... -lfuzzy -DWITH_SSDEEP -I/usr/local/
checking for library containing fuzzy_hash_buf... -lfuzzy
checking that generated files are newer than configure... done
configure: creating ./config.status

 After completing the compilation and install steps.  We can now configure the rules.

Adding Fuzzy Hashing Rules

Here is an example rule that can be added to a custom rules file:

SecRule FILES_TMP_CONTENT "@fuzzyHash webshells-ssdeep.txt 90" \
"id:87,phase:request,t:none,block,msg:'Malicious File Attachment Hash Match.'"

Looking at this rule, we see the following:

• We are inspecting a new variable that we added to ModSecurity v2.9.0-RC1 called FILES_TMP_CONTENT.  This variable will hold the data captured from multipart-form-data file attachment uploads.
• We are using the @fuzzyHash operator which takes two parameters:
  • First parameter is the path to the ssdeep hash text file we created earlier.  In this example, the file is located in the same directory as this rules config file.
  • Second parameter is the minimum threshold score of similarity between files.  This means that our operator will only match if the file inspect matches at least 90% to a file in our ssdeep fuzzy hash file.

Testing Fuzzy Hash Rule

With these configurations in place, let's run a quick test and simply upload one of the existing webshells we have called "ironshell.php".

$ curl -k -X POST -F "action=upload" -F "file=@./ironshell.php" http://localhost/

 We can then look at the ModSecurity audit log file:

As you can see from the highlighted ModSecurity message data, our fuzzy hashing rule matched with an ssdeep match score of 100.  That makes sense as we have not modified the ironshell.php file at all.  Let's make a few changes:

We can then upload this new webshell:

$ curl -k -X POST -F "action=upload" -F "file=@./ironshell-new.php" http://localhost/

Here is the new alert from the Apache error_log file:

As you can see, the file was still flagged however the match percentage went down to 97%.  Still, this means that we can catch modifications to this ironshell webshell code family!  This makes it much more powerful vs. other exact match hashing checks.

Conclusion

This is a new detection capability recently released in the v2.9.0RC1 release and we need help from the community for testing.  We will soon be adding this functionality to our Trustwave SpiderLabs commercial ModSecurity rules package.  

One note for usage and detection, ssdeep has a default minimum file size of 4096.  This means that really small files will not work correctly with @fuzzyHash.  If you want to test out reducing the minimum size threshold, you can edit the main.c file here:

// We print a warning for files smaller than this size
#define SSDEEP_MIN_FILE_SIZE 4096

Please let us know if you have any issues using this new feature.