Kamailio SIP fuzzing project setup

fuzzing Kamailio IT-Security

Details about my Kamailio SIP fuzzing project setup

Kamailio SIP fuzzing project

This is the second post of a series about my Kamailio SIP fuzzing project. I will describe here my current parallel fuzzing setup in more details.

In the first post I described my motivation and shared the presentation that I did at the Kamailio World conference 2018. If you haven't read yet and don't have any fuzzing background, I would suggest you start there.

Infrastructure setup

For effective fuzzing of a complex server like Kamailio you need to execute millions of test invocations. Therefore it helps a lot if you are able to use an appropriate setup with sufficient CPU and I/O power.

Old infrastructure

For my old fuzzing setup I used my home work station. This setup was of course convenient for the development, maintenance and ongoing monitoring tasks. But as I needed this machine also for other tasks this setup was somewhat limited. After my talk at Kamailio World 2018 I discussed with Fred Posner, who wanted to sponsor this project. He provided me a dedicated virtual server instance, which was really great.

New infrastructure

The new fuzzing setup is a medium sized virtual server hosted privately with Debian Linux. I setup a parallel afl fuzzing with one master instance and two to three slave instances. The master control the several slave instances. If one of the different instances finds a new execution path or a crash in the tested program, this information is periodically replicated to all the other instances. The following picture describe the setup graphically.

Parallel fuzzing setup

For each of the thousands execution paths during the tests at least one small file is created. If an actual crash is found, the crash will also stored in a file. This files are synchronized and also re-read from the different afl instances. Therefore the fuzzing puts a lot of stress on a file system. To avoid wear-out of disks/SSDs and also to improve the performance I placed the working directory in a standard RAM disk. This RAM disk is periodically copied for backup to a normal disk.

The afl suite provides some tools to monitor the state of the parallel fuzzing instances. I use tmux to monitor the progress during a test cycle and also some other parameter (disk space, CPU and I/O load). I check daily for new crashes and analyses them with Linux development tools.

I tried as well some more advanced tools from the afl suite and some related projects. Some of them did not work in my setup (e.g. test case minimization). I will investigate some of the more advanced concurrent control tools again after my setup grows. This is probably a good idea for a third blog post in the future.

Challenges in development the test driver

Standard afl needs to to drive the program under test from standard input. Kamailio is out of the box not able to process SIP messages from stdin. Therefore several adaptions are necessary to be able to test Kamailio successfully.

Old test driver

At first I choose the standard Linux text input functions getline()and getdelim() from the GNU C library (glibc). This functions works normally great, but where not up to the tasks to process thousands of invalid and malformed messages. As result I got after some tests thousands of SIGABRT crashes.

SIGABRT is commonly used by glibc and other libraries to abort the program in case of critical errors. I investigated several crashes, and glibc sends the SIGABRT in case of a detected double-free or other heap corruptions. Fortunately I could easily distinguish the SIGABRT crashes from valid SIGSEV crashes caused from Kamailio.

In the following picture you can see the state from one of the afl slave instances during a test run. From the thousands of crashes almost every one where caused from the glibc and not from Kamailio. This shows how amazingly stable the Kamailio core is, compared to other tools and even system libraries.

Screenshot of one fuzzer slave

After several iterations I started to get also SIGSEV crashes from the GNU C library. As my objective was not to fuzz glibc and the crashes started to fill up my working directory, I decided to create a new version of the test driver.

New test driver

The new test driver uses the standard glibc function fread() to read the messages as binary data. This makes of course much more sense, I should have used this function directly from the start. This driver has the additional benefit that it don't need an magic character as a delimiter of the SIP message anymore. It was great to get rid of this hack and getting closer to the original SIP message format. So far I did not saw any new crashes related to glibc functions.

Testing scope

I focus in my tests on standard SIP messages like INVITE, REGISTER or ACK against a default Kamailio configuration. I plan in the future to extend the scope to other SIP messages and also to test other Kamailio modules.

Future extensions

I plan to extend my fuzzing setup further in the future. The easiest option would be to add more machines to speed up the process. Another optimization is to further tailor the test data, to minimize the input to the fuzzer.

Statistics

The new setup is able to execute over 8 million test execution per day. My longest run so far found and successfully tested over 47.000 different program paths in Kamailio, in total over 250 million test execution have been done. My estimate for all different tests runs until today combined would be 500 million executions.

Update: After several weeks the execution count is now around 1 billion, still without any issues - amazing.

Results

So far I found in my research this two results:

Test your code, support this project

If you are running some specific or proprietary Kamailio modules in your VoIP infrastructure and would like to harden them against attacks, please contact me. These tests can be easily done on my infrastructure. I will tailor the test scope to your needs and document the results in a report.

If you would like to support me to improve Kamailio and make the server more secure against malicious attacks, you can sponsor me as well.

You can contact me here.

Information about updates

You can subscribe here to my newsletter to get regular updates.

Previous Post Next Post