A guide to fuzz testing

Table of Contents

• What is fuzz testing?
• Smart vs. dumb fuzzing
  • Dumb Fuzzer
  • Pros & Cons of dumb fuzzing
  • Smart Fuzzer
  • Pros & Cons of smart fuzzing
• Fuzzer types
  • Mutation-based fuzzers
  • Generation-based fuzzers
• Generating random data
• Code coverage in fuzz testing
• Why should you keep an eye on API Fuzzing?
• What are the taxonomies of API fuzzers?
  • Knowledge of Input
  • Knowledge of target application structure
  • Method of generating new input
• Fuzz testing APIs using Testfully
• Top API fuzzing tools
  • KITERUNNER
  • RESTRler
• Fuzz testing glossary

What is fuzz testing?

Fuzz testing is a type of automated software testing; a method of discovering bugs in software by providing random input to the software under the test and monitoring any crashes and failed assertions. Fuzzer is a computer program that offers random inputs to the software. Fuzz testing can be applied to virtually any software program, including HTTP APIs.

Smart vs. dumb fuzzing

As mentioned earlier, a fuzzer provides random data to the software under the test. The input can be entirely random without matching the shape of the expected input or generated to match a valid input.

Dumb Fuzzer

Dumb fuzzers produce completely random input that does not necessarily match the shape of the expected input. Lack of built-in intelligence about the software it’s fuzzing makes this type of fuzzer a dumb one. Due to their simplicity, dumb fuzzers can produce results with little work. On the other hand, dumb fuzzers may be able to fuzz only certain areas of the software.

For example, the presence of the newsletter boolean field in the request payload of a Rest API endpoint for creating a new user triggers the logic for newsletter subscription. But, as you may have already guessed, a dumb fuzzer won’t be able to begin the execution of the newsletter subscription logic and identify potential bugs in that area of the code due to the wholly randomized input that does not match the valid input.

Pros & Cons of dumb fuzzing

Smart Fuzzer

Smart fuzzers push the boundaries of fuzz testing by generating randomized data valid enough to pass program parser checks, get deep into the program logic, and potentially trigger edge cases and find bugs.

The more built-in intelligence you add into your smart fuzzer, the greater code coverage you will have.

Pros & Cons of smart fuzzing

Fuzzer types

Based on the way a fuzzer generates the randomized input data, we can divide fuzzers into mutation-based and generation-based fuzzers.

Mutation-based fuzzers

A mutation-based fuzzer takes valid inputs and generates a collection of inputs by changing (mutating) the valid inputs.

Generation-based fuzzers

A generation-based fuzzer analyses the provided valid input structure and generates entirely new data that matches the valid one from the structure perspective.

Generating random data

Mutation-based and generation-based fuzzers both generate random data, albeit taking different approaches. The experience has proven that including specific values can trigger edge cases and bring bugs to the surface. The table below goes through some values proven to bring bugs to the surface.

Code coverage in fuzz testing

Code coverage refers to the percentage of the executed code while running the test cases against the source code. Broadly speaking, the greater the coverage percentage, the better it is. Therefore, while doing fuzz testing, you should always keep an eye on the part of the code you fuzz. For example, a dumb fuzzer can potentially fuzz the parser code than your business logic. Moreover, a mutation-based or generation-based fuzzer will trigger the execution of the code in different branches of the source code.

The diagram below shows seven branches of code that a fuzzer can trigger via input. But, of course, not every fuzzer can start all branches.

Why should you keep an eye on API Fuzzing?

When it comes to keeping our apps secure, we perform various kinds of testing methods such as static code analysis, dynamic code analysis, penetration, and more to protect them from the eyes of attackers. But usually, APIs are in the blind spot of the application security testing methods.

Unfortunately, in many software companies, security leaders think their APIs and products are safe because of performing regular penetration testing, not knowing the hackers find the software vulnerabilities through fuzzing. So, here is the fun fact, if you want not to be a victim of hackers, think like a hacker and find your software loopholes.

Consider the fuzzing practice in your penetration tactics and techniques for more accurate results about API vulnerabilities.

What are the taxonomies of API fuzzers?

Different companies have various approaches to finding vulnerabilities through fuzzing, but Microsoft’s taxonomies are generally more common and accepted.

Knowledge of Input

Some called them “smart fuzzers,” as we discussed previously in this article; they know the expected input format of the application. So, for example, they know that the input file should be in text format.

Knowledge of target application structure

There are also different categories for fuzzers I want you to hear: black-box, white-box, and gray-box fuzzers.

In the white-box approach, the tester or the test tool has all the required information of the input format and structure of the target, so there is no need to guess them in the opposite of the black-box testing approach.

So in the world of fuzzers, the white-box fuzzer who knows all about the target application, and the gray-box fuzzer who has partial knowledge of the target application, are the smartest fuzzers.

Method of generating new input

As discussed earlier in this article, fuzzers can randomly generate new input from scratch or change the pre-existed input to test the target with them.

Fuzz testing APIs using Testfully

Testfully’s embeddable value data generators and data templates enable our customers to easily create intelligent fuzzers based on the expected request payload shape while controlling the generated data to test different branches of the code. This feature is available under all plans (including free plan).

The below short demo shows how easy it is to add random data to your requests.

Top API fuzzing tools

KITERUNNER

Website: github.com/assetnote/kiterunner | Price: Free and Open source

A content discovery tool that uses many open API’s specifications as a searching source to find unlinked files or folders. Also, it integrates with apps like Burp to replay the test with different permutations of the original request.

RESTRler

Website: github.com/microsoft/restler-fuzzer | Price: Free and Open source

It is the world’s first stateful API fuzzer, a Microsoft product that gets the open APIs specification and generates the tests automatically to find the vulnerability of an API.

Fuzz testing glossary