Performance testing is an automated software testing practice and an essential part of software development. It is the process of evaluating the execution time, throughput, and scalability of a software application and its components under various load levels.
Performance testing is a valuable tool for helping you identify where your website or application is slow, buggy, or otherwise not behaving as desired. To create a good performance testing strategy, you need to understand the core fundamentals. Learning about the different types of performance metrics will help you identify problems and develop effective solutions.
What is API performance testing?
API performance testing is the process of checking the robustness, responsiveness, and stability of an application programming interface (API). This type of testing helps to ensure that the API can handle a particular number of requests during a set period. Performance Testing is a broad term that covers a number of different test types. Each has its place and purpose in your overall API testing strategy. This article will review Load Testing, Stress Testing, Soak Testing, Spike Testing, and Breakpoint testing.
{{< cta-api-testing “top” >}}
What is Load testing
Load testing or load simulation is a method used to determine if a system or software will handle expected volumes of traffic; its performance is measured under a specific load. Load testing may include simulating client sessions, transactions, API calls, database operations, or other scenarios.
Load Testing focuses on evaluating software performance under the expected volume of traffic.
This kind of performance testing is not limited to APIs. For example, a PDF split software can be load tested to evaluate its performance while processing a PDF file with 200 pages.
What is Soak testing
Similar to Load Testing, Soak Testing is a method used to identify the system’s behavior under the expected traffic volume. Soak Testing simulates the expected traffic load for some time to determine if the system can handle the load for the test period.
While Load testing is done for a short period, Soak Testing is done using continuous load.
The continuous nature of Soak testing allows teams to identify software memory leaks.
What is Stress testing
Stress testing is a term that refers to the process of testing a website or application for its ability to withstand heavy amounts of traffic. Stress testing is primarily performed to ensure that web service will handle a hefty load and not crash or break down when a surprising number of users try to access it. A secondary purpose is to determine how long it will take for a website or application to return to normal functioning after experiencing a significant spike in traffic
Stress Testing focuses on evaluating software performance under the unexpected volume of traffic.
What is breakpoint testing
The goal of Breakpoint testing is to measure the system’s maximum capacity. It’s done by applying an incremental load to the system and observing its behavior until a crash is identified.
What is spike testing
Spike testing is the practice of increasing or decreasing the load on a system or software suddenly to analyze the performance and robustness of the system under test. In the context of software deployed in cloud providers like AWS, Google Clour, or Microsoft Azure, spike testing helps companies understand how the system scales up and down based on spikes.
Load testing vs. Soak testing
As discussed previously, both load testing and soak testing test a system or software under the expected load. Expected load is the traffic it is expected for the software to handle without any issue. For an API, it could be the number of requests to the API in one minute.
Test duration is one big difference between load testing and soak testing. Soak Testing gives the expected load for a more extended period than load testing to capture the metrics that otherwise would be impossible to capture. Memory consumption (to identify potential memory leaks) is one of the metrics that cannot be captured during a short load testing session.
Load & soak tests are very similar. They both examine performance under the expected load. However, soak testing examines the software for a more extended period than standard load testing.
Load testing vs. Performance testing
Performance testing is a way to find out whether your application is performing well according to your expectations. Loading testing, on the other hand, is a variation of performance testing. Load testing is used to analyze how a program behaves when it is loaded with the expected load.
Load testing is a subset of performance testing.
Load testing vs. Stress testing
While load testing measures the performance and robustness of the system under the expected load, stress testing tries to understand the upper limit of a system using extreme traffic. Thus, load testing answers the Does API handle the expected traffic? question. On the other hand, stress testing answers the Does API handle the unexpected traffic? question.
Load testing examines performance under the expected load. Stress testing, as the name suggests, looks at performance under unexpected traffic.
What is checked in a performance test?
Performance testing covers application responsiveness, robustness, and stability. To analyze the mentioned criteria, different metrics are captured during tests. This section will cover the most commonly collected metrics as part of the API performance testing.
Response time
Response time is the time it takes to respond to a user’s request. It is a performance indicator for applications and websites. Response time is critical in real-time or interactive applications.
During a performance testing session, the Response time of each request is captured to calculate the average response time. This key metric defines how fast the end-users will be able to take specific actions using the API. Remember that the lower your response time, the more customers you will get.
Error rate
The error rate is the percentage of errors an API generates during the performance test execution. It is similar to the response time but different in that it does not take into account the waiting time, but only the errors generated during the request. For instance, an operation that takes 100ms and returns 200 records with two errors results in an error rate of 2%.
Ideally, the error rate during a performance testing session should remain as low as possible. Error cases should be categorized based on their kind and investigated individually.
Memory leaks
A memory leak is a type of resource leak that occurs when a program incorrectly manages memory allocations, so that memory that is no longer needed is not released. This can result in gradual performance degradation as the program consumes more and more memory resources until it finally either crashes or starts swapping to disk.
Under heavy load, some API processes start consuming more and more memory without releasing it. Eventually, the API becomes completely unavailable and requires a restart. During a performance test, memory consumption of API processes is sampled periodically and put on a line chart. Ideally, the line should remain straight to prove that the process does not have a memory leak.
Auto-scaling capabilities
Auto-scaling is a process whereby a system automatically adjusts its resources depending on the current load of the system. A well-architected auto-scaling system can improve the flexibility and resiliency of a service or application, as well as increase the ability to handle unexpected loads.
To test the agility of a system to scale up and down based on demand, performance teams run performance testing sessions. During the session, they monitor the time it takes to allocate extra infrastructure resources to cope with the increasing demand. Metrics, including response time and error rate, defines whether the system scaled up on time or not.
Crash rate
A process crash occurs when a program encounters a situation it cannot handle. This can include running out of memory, encountering a divide by zero error, or not being able to create a file. These can be severe errors, but they are usually handled without user intervention.
Process crashes result in increasing error rates from the API. Moreover, it can introduce inconsistency to your system. Process crashes during performance testing offer the following benefits:
- New bugs are identified as part of a process crash
- Agility of the infrastructure to provision a new resource is examined
In our Fuzz testing article, we have explored how fuzz testing can help you identify edge cases that result in process crashes.
To measure the performance of your API or application, you need to have a good understanding of what is important.
Performance testing glossary
| Term | Description |
|---|---|
| Performance testing | process of evaluating the execution time, throughput, and scalability of a software application and its components under various load levels. |
| Load testing | a method used to determine if a system or software will handle expected volumes of traffic |
| Soak testing | a technique used to identify the system’s behavior under the expected traffic volume. Soak Testing simulates the expected traffic load for some time to determine if the system can handle the load for the test period. |
| Stress testing | process of testing a website or application for its ability to withstand heavy amounts of traffic. |
| Breakpoint testing | a technique to measure the system’s maximum capacity |
| Spike testing | the practice of increasing or decreasing the load on a system or software suddenly to analyze the performance and robustness of the system under test. |
Frequently Asked Questions
We got an answer for your questions
-
What is API performance testing?
API performance testing is the process of checking the robustness, responsiveness, and stability of an application programming interface (API). This type of testing helps to ensure that the API can handle a particular number of requests during a set period. Performance Testing is a broad term that covers a number of different test types. Each has its place and purpose in your overall API testing strategy. This article will review Load Testing, Stress Testing, Soak Testing, Spike Testing, and Breakpoint testing.
-
What is Load testing
Load testing or load simulation is a method used to determine if a system or software will handle expected volumes of traffic; its performance is measured under a specific load. Load testing may include simulating client sessions, transactions, API calls, database operations, or other scenarios.
-
What is Soak testing
Similar to Load Testing, Soak Testing is a method used to identify the system’s behavior under the expected traffic volume. Soak Testing simulates the expected traffic load for some time to determine if the system can handle the load for the test period.
-
What is Stress testing
Stress testing is a term that refers to the process of testing a website or application for its ability to withstand heavy amounts of traffic. Stress testing is primarily performed to ensure that web service will handle a hefty load and not crash or break down when a surprising number of users try to access it. A secondary purpose is to determine how long it will take for a website or application to return to normal functioning after experiencing a significant spike in traffic
-
What is breakpoint testing
The goal of Breakpoint testing is to measure the system’s maximum capacity. It’s done by applying an incremental load to the system and observing its behavior until a crash is identified.
-
What is spike testing
Spike testing is the practice of increasing or decreasing the load on a system or software suddenly to analyze the performance and robustness of the system under test. In the context of software deployed in cloud providers like AWS, Google Clour, or Microsoft Azure, spike testing helps companies understand how the system scales up and down based on spikes.
-
what is the difference between load testing and soak testing?
As discussed previously, both load testing and soak testing test a system or software under the expected load. Expected load is the traffic it is expected for the software to handle without any issue. For an API, it could be the number of requests to the API in one minute. Test duration is one big difference between load testing and soak testing. Soak Testing gives the expected load for a more extended period than load testing to capture the metrics that otherwise would be impossible to capture. Memory consumption (to identify potential memory leaks) is one of the metrics that cannot be captured during a short load testing session.
-
what is the difference between load testing and performance testing?
Performance testing is a way to find out whether your application is performing well according to your expectations. Loading testing, on the other hand, is a variation of performance testing. Load testing is used to analyze how a program behaves when it is loaded with the expected load.
-
What is the difference between load testing and stress testing?
While load testing measures the performance and robustness of the system under the expected load, stress testing tries to understand the upper limit of a system using extreme traffic. Thus, load testing answers the Does API handle the expected traffic? question. On the other hand, stress testing answers the Does API handle the unexpected traffic? question.