A guide to fuzz testing

A guide to fuzz testing

Table of Contents

Looking for an API testing tool?

Testfully offers multi-step tests, local & deployed API testing, team collaboration, historical test results and many more features. Import from Postman supported!

What is fuzz testing?

Fuzz testing is a type of automated software testing; a method of discovering bugs in software by providing random input to the software under the test and monitoring any crashes and failed assertions. Fuzzer is a computer program that provides random inputs to the software. Fuzz testing can be applied to virtually any kind of software program, including HTTP APIs.

Smart vs. dumb fuzzing

As mentioned earlier, a fuzzer provides random data to the software under the test. The input can be entirely random without matching the shape of the expected input or generated to match a valid input.

Dumb Fuzzer

Dumb fuzzers produce completely random input that does not necessarily match the shape of the expected input. Lack of built-in intelligence about the software it’s fuzzing makes this type of fuzzer a dumb one. Due to their simplicity, dumb fuzzers can produce results with little work. On the other hand, dumb fuzzers may be able to fuzz only certain areas of the software.

For example, the presence of the newsletter boolean field in the request payload of a Rest API endpoint for creating a new user triggers the logic for newsletter subscription. But, as you may have already guessed, a dumb fuzzer won’t be able to begin the execution of the newsletter subscription logic and identify potential bugs in that area of the code due to the wholly randomized input that does not match the valid input.

Pros & Cons of dumb fuzzing

Dumb fuzzing pros Dumb fuzzing cons
Straightforward to set up, run, and maintain Limited code coverage due to the fully randomized input
Requires minimum amount of work for the initial setup Sometimes, it tests a parser than your program

Smart Fuzzer

Smart fuzzers push the boundaries of fuzz testing by generating randomized data valid enough to pass program parser checks, get deep into the program logic, and potentially trigger edge cases and find bugs.

The more built-in intelligence you add into your smart fuzzer, the greater code coverage you will have.

Pros & Cons of smart fuzzing

Smart fuzzing pros Smart fuzzing cons
Greater code coverage in comparison with dumb fuzzers Requires more work to set up, run and maintain
Catches more bugs thanks to greater code coverage

Fuzzer types

Based on the way a fuzzer generates the randomized input data, we can divide fuzzers into mutation-based and generation-based fuzzers.

Mutation-based fuzzers

A mutation-based fuzzer takes valid inputs and generates a collection of inputs by changing (mutating) the valid inputs.

Generation-based fuzzers

A generation-based fuzzer analyses structure of the provided valid input and generates entirely new data that matches the valid one from the structure perspective.

Generating random data

Mutation-based and generation-based fuzzers both generate random data, albeit taking different approaches. The experience has proven that including specific values can trigger edge cases and bring bugs to the surface. The table below goes through some values that are proven to bring up bugs to the surface.

Value Description
Empty strings Sometimes, empty string by-pass missing value checks and trigger bugs
Long strings Bugs as a result of truncation come to the surface as a result of passing long strings to programs
Strings with variant length Short, medium, and long strings can trigger bugs as well
0 Similar to empty strings, value 0 can sometimes pass the missing value checks and trigger bugs
Negative numbers Triggers bugs related to assuming positive numbers but lacking validation for that
Decimals Triggers bugs related to assuming integers but lacking validation for that
Special characters Bring up bugs related to embedding values in URL or saving in database
Max / Min numbers Does the code cope well with a maximum allowed number? what about the minimum?

Code coverage in fuzz testing

Code coverage refers to the percentage of the executed code while running the test cases against the source code. Broadly speaking, the greater the coverage percentage, the better it is. Therefore, while doing fuzz testing, you should always keep an eye on the part of the code you fuzz. For example, a dumb fuzzer can potentially fuzz the parser code than your business logic. Moreover, a mutation-based or generation-based fuzzer will trigger the execution of the code in different branches of the source code.

The diagram below shows seven branches of code that a fuzzer can trigger via input. But, of course, not every fuzzer can start all branches.

Signup flow with 7 different branches

Fuzz testing APIs using Testfully

Testfully’s embeddable value data generators and data templates enable our customers to easily create intelligent fuzzers based on the expected request payload shape while controlling the generated data to test different branches of the code. This feature is available under all plans (including free plan).

The below short demo shows how easy it is to add random data to your requests.

Fuzz testing glossary

Term Definition
Fuzz The random input for fuzz testing
Fuzzer A program or programming code that generates random input
Dumb fuzzer A fuzzer that does not know the expected input structure
Smart fuzzer A fuzzer that knows input structure
Mutation-based fuzzer A fuzzer that generates input by changing the provided valid input
Generation-based fuzzer A fuzzer that generates input from scratch by analyzing the provided valid input
Code Coverage The percentage of the code that is executed by running the test cases
Source code branch A portion of the source code that will be executed under certain conditions

Looking for an API testing tool?

Testfully offers multi-step tests, local & deployed API testing, team collaboration, historical test results and many more features. Import from Postman supported!

comments powered by Disqus

Recent Articles

7 HTTP methods and how to use them

HTTP protocol works by clients sending requests to the servers and servers responding to the requests. We do CRUD operations (Create, Read, Update, Delete) by sending HTTP requests with different HTTP methods, sometimes called HTTP verbs.

Introduction to API Blueprint

API blueprint is a powerful high-level API design language for web APIs. In this article, we want to dive deeper into it and learn more about how it works, the differences between API blueprint and Swagger, and what makes it unique that leads to its extensive use. But before we dig into API Blueprint, we must ensure a solid base of information about the “API first approach” concepts.

False positive & false negative in software testing

Exports in automated software testing have borrowed false positive and false negative terms from the medical examination field. In the medical field, the purpose of a test is to determine whether the patient has a particular medical condition or not. As far as software testing is concerned, a false positive indicates a bug when there is none. Conversely, a false negative indicates no bug when there is one.