Fuzzing Forward: Turbocharging Quality with Restler’s Assurance Boost

Introduction:

In SDLC, software testing plays a major role in delivering high-quality software to the market by advancing the new capabilities of the application. This is done by validating the implementation of new features and their functionality, ensuring compliance with intended requirements and specifications, mitigating security risks, ensuring reliability, and addressing performance, usability, and scalability, etc. 

While manual and automation testing are essential, fuzz testing as a subset of automation testing further enhances the automation testing process. This comprehensive approach ensures thorough coverage. Manual testing delves into specific details and complexities, automation provides consistency and fuzz testing uncovers edge cases and vulnerabilities.

At Cashfree Payments we integrate Restler into our CI pipeline to bolster automated testing, underscoring the significance of fuzz testing within automation testing practices.

What is API Fuzzing:

API fuzz testing is an automated software testing technique that works by sending an intensive number of malformed inputs to test functional flows and the security of the application. The purpose of API fuzzing is to uncover the weaknesses in API implementation which cannot be done using traditional testing practices.

Fuzz testing swiftly simulates real-world scenarios at scale ensuring comprehensive test coverage, and delves deep into the software’s functionality to expose potential issues. The automated nature of fuzz testing guarantees consistent and reliable results, enabling developers to identify and rectify vulnerabilities efficiently.

Restler conducts fuzzing on input combinations for each parameter, considering data types. This process multiplies when additional parameters are included, estimating the total combinations.

To better understand how Restler conducts fuzzing on input combinations, let’s consider the login API as an example

Restler fuzzes the email parameter with different input combinations, the number of input combinations multiplies when the password parameter is included for fuzzing. Testing the login API through traditional practices typically involves 10-20 unique input combinations, while Restler fuzzing expands input combinations exponentially to around 180-190 combinations.

Restler’s advanced features, including smart test case creation and automatic specification discovery, make it our top choice for CI/CD pipelines. Its seamless integration and thorough testing coverage enhance confidence in API deployments, aligning perfectly with Cashfree Payments’ quality standards.

How Restler works:

Restler seamlessly incorporates OpenAPI specifications for precise fuzzing, generating test cases that align with expected API behavior.

Restler modes:

Restler runs in 4 main modes in the following order:

Step 1: Compilation mode

Restler carefully analyzes the Swagger/OpenAPI specification and creates a fuzzing grammar for effective testing. With this Restler grammar,  we can execute fuzz testing across various modes to ensure comprehensive evaluation and validation.

Step 2: Test mode

In Test mode, Restler tries to run every type of request mentioned in the Swagger specification at least once. The goal of this type of test is also known as a smoke test. Smoke test is used to quickly check if everything is working as expected without going into too much detail.

Step 3: Fuzz-lean mode

In Fuzz-lean mode, Restler swiftly tests each endpoint and method defined in a compiled grammar using default checkers. It aims to identify bugs promptly by executing once for every endpoint+method combination.

Step 4: Fuzz mode

In Fuzz mode, Restler engages in an extended fuzzing session on the tested service, aiming to uncover a wider range of bugs, resource leaks, performance glitches, and backend corruptions. However, it’s important to note that this mode can be aggressive and has the potential to disrupt the service, particularly if it’s not implemented well.

However, it’s important to note that this mode can be aggressive and has the potential to disrupt the service, particularly if it’s not implemented well.

Restler Output Results:

After the fuzzing process, Restler generates sub-directories named Test, FuzzLean, and Fuzz for respective modes, each containing the following files:

• speccov.json: Provides a summary of coverage for all requests.
• main.txt: Logs detailing the attempted execution of each request.
• request_rendering.txt: Report showcasing overall progress.
• network.testing.<threadID>.txt: Logs all HTTP(S) traffic generated by Restler, aiding in detailed debugging.
• network.gc.<threadID>.txt and garbage_collector.gc.<threadID>.txt: Logs related to Restler garbage collector, valuable for troubleshooting resource leaks.
• runSummary.json: A report summarizing all received HTTP response codes.
• errorBuckets.json: Samples of request and response pairs for HTTP error codes in the 4xx or 5xx ranges.
• Bug_buckets: When Restler finds a bug, it logs it in the bug_buckets directory. It tries to reproduce the bug and notes if it was successful. A summary of bugs is recorded in the bug_buckets.txt file, while detailed logs for each bug are stored in individual files within the bug_buckets directory.

Sample Bug Report:

Integrate Restler As part of Continuous integration:

Incorporating Restler into our CI/CD workflows at Cashfree enables developers to promptly identify and resolve issues, thereby ensuring quality and faster release time to production, allowing QA to allocate their time and resources to other critical tasks.

Let’s delve into the steps to integrate Restler effectively:

1. Docker image creation:

Begin by crafting a Docker image tailored to your project’s requirements. Incorporate all essential dependencies for API fuzz testing. Then, either upload the image to DockerHub or configure Docker to fetch the dependencies and seamlessly integrate Restler into the container.

2. Shell scripting for fuzz testing:

Create a detailed shell script for Restler to manage all tasks for each fuzz testing mode. Run Restler within this script, making sure to save the results as artifacts.

3. Integrate Swagger APIs within the script

Leveraging the springdoc-openapi-ui dependency in pom.xml in the development repository, we can streamline fetching updated OpenAPI specifications by using Swagger API in the script. Whenever we’re ready to test a new feature, we kick off the fuzzing process by running our script, which covers existing and new APIs. 

4. Bug Handling, Result Analysis, and Validation:

Integrate logic within your script to detect bugs and fail the pipeline if any bugs are found. After testing, if the pipeline fails, download the results and get the bugs. Fix issues quickly.

To smoothly connect with several services, use environment variables in your setup. Set the service name as an environment variable, making it easy to adjust and run Restler for each service separately. Ensure the script logic is written to fetch the Swagger file corresponding to the requested service based on the service name.

Impact :

Preliminary analysis of this integration to one service has led to the discovery of 5 critical bugs that Restler has identified. We are in the process of rolling it out to all our applications.

Next Steps :

We’re looking to introduce automatic bug creation within our pipeline to further streamline our processes and enhance tracking efficiency. By integrating JIRA APIs, we aim to generate bug tickets for each unique bug report file automatically. This integration not only fosters a more agile approach to bug tracking but also significantly reduces manual effort by eliminating the need to download and review each report individually.

In our pursuit of maximizing quality and efficiency, our next step involves seamlessly integrating Restler into the deployment pipeline of our development repository. This means that deployment will only proceed once all identified bugs are addressed. By implementing this approach, we can ensure a 100% guarantee on bug fixes, minimizing the risk of issues slipping through the cracks and optimizing our overall development lifecycle. At Cashfree Payments, our commitment to quality is strengthened by this synergy, guaranteeing robust and reliable software solutions.