Know the scanner's OWASP Benchmark results | Fluid Attacks Help

Know and reproduce the scanner’s OWASP Benchmark results

Fluid Attacks uses the OWASP Benchmark

Fluid Attacks is committed to delivering highly accurate security testing results. This means minimizing errors from the automated tools and security analysts involved in the Continuous Hacking solution. To measure accuracy, Fluid Attacks utilizes several benchmarks, including the OWASP Benchmark Project.

The Open Worldwide Application Security Project (OWASP) is a non-profit foundation dedicated to helping improve software security. It operates as an open, online community where anyone can contribute resources and expertise related to application security (AppSec). This collaborative environment allows developers and security professionals to learn from each other and stay up-to-date on the latest threats and best practices.

The OWASP Benchmark Project is a free Java test suite created in 2015 to assess the accuracy, speed and coverage of automated software vulnerability detection tools. It helps developers and security professionals identify the strengths and weaknesses of various AppSec testing solutions, allowing for objective comparisons between them.

The OWASP Benchmark allows for the evaluation of different types of security testing tools that apply any of these techniques:

  1. Static application security testing (SAST): Analyzes source code
  2. Dynamic application security testing (DAST): Tests running applications without access to source code
  3. Interactive application security testing (IAST): Analyzes code and application behavior through agents and sensors while the application is running

A tool's performance is measured by its ability to correctly identify vulnerabilities (true positives) and secure code (true negatives) while minimizing incorrect assessments (false positives and false negatives). This evaluation provides a benchmark for choosing the right security testing tool for your software development lifecycle (SDLC).

OWASP Benchmark measuring system

The OWASP Benchmark utilizes two primary metrics to evaluate tool performance:
  1. True Positive Rate (TPR) or Sensitivity: This indicates the percentage of actual vulnerabilities correctly identified by the tool. A higher TPR means the tool is more effective at finding real security risks.
  2. True Negative Rate (TNR) or Specificity: This indicates the percentage of safe code correctly identified as non-vulnerable. A higher TNR means the tool is less likely to generate false alarms.

As mentioned earlier, two other classifications of assessments are possible, however, they are not treated further in this page:

  1. True negatives: Correct reports of code, inputs or ports as being secure (these are desirable, as they allow you to address real security risks)
  2. False negatives: Incorrect reports of code, inputs or ports as being secure (these are highly undesirable, as they can lead to a false sense of security and the potential deployment of vulnerable versions of your system into production)
The following illustration may help you better grasp the result categories, where the bigger circle encloses what the tool reports as vulnerable:

Understand true positives, false positives, true negatives, false negatives

Fluid Attacks' results

To ensure high levels of accuracy, Fluid Attacks tested its automated vulnerability detection tool —a core component of the Continuous Hacking plans— against the OWASP Benchmark test suite.

The following scorecard shows how Fluid Attacks' scanner compares to other vulnerability detection tools measured against the OWASP Benchmark test suite:
OWASP Benchmark results for the Fluid Attacks scanner
Advice on comparing toolsYou can compare different tools by using the Youden's J statistic: J = Sensitivity + Specificity - 1

As demonstrated in the image above, Fluid Attacks' scanner consistently outperforms other vulnerability detection tools:

  1. 100% True Positive Rate: The scanner accurately identifies all actual vulnerabilities in the test suite.
  2. 0% False Positive Rate: The scanner does not flag secure code as vulnerable.

This exceptional performance translates to a perfect OWASP Benchmark Score of 100%, significantly exceeding industry averages:

  1. Almost 3x higher than the average score of commercial (paid) vulnerability detection tools
  2. More than 1.5x higher than the highest-scoring non-commercial (free) tool
What is most important, Fluid Attacks cares about what you care:
  • Finding all vulnerabilities before they impact your business
  • Maintaining your team's efficiency with zero false positives
Advice on CASA scansMoreover, Fluid Attacks' scanner is recommended by Google to validate Tier 2 requirements of the Cloud Application Security Assessment (CASA) framework.

Reproduce Fluid Attacks' OWASP Benchmark results

All Fluid Attacks products are open source.  You can download, inspect, and suggest to modify the source code behind them. Being open-source gives customers confidence in Fluid Attacks' transparency and security.

To verify the OWASP Benchmark results, follow these steps (or skip ahead to the one-step alternative):

  1. Meet the requirement: Install Docker to be able to run Fluid Attacks' scanner.

  2. Install the scanner: Download the Docker container and pull the image.

  3. Clone the OWASP Benchmark v1.2 repository:

    git clone https://github.com/OWASP-Benchmark/BenchmarkJava.git benchmark
    cd benchmark

  4. Create a configuration file: Use the following content:

    checks:
      - F004
      - F008
      - F021
      - F034
      - F042
      - F052
      - F063
      - F089
      - F107
      - F112
      - F130
    namespace: OWASP
    multifile: true
    output:
      file_path: results/Benchmark_1.2-Fluid-Attacks-v2024.csv
      format: CSV
    sast:
      include:
        - src/main/java/org/owasp/benchmark/testcode/
        - src/main/java/org/owasp/benchmark/helpers/DatabaseHelper.java
        - src/main/java/org/owasp/benchmark/helpers/SeparateClassRequest.java
        - src/main/java/org/owasp/benchmark/helpers/Thing1.java
        - src/main/java/org/owasp/benchmark/helpers/Thing2.java
        - src/main/java/org/owasp/benchmark/helpers/ThingFactory.java
        - src/main/resources/benchmark.properties

  5. Execute the scan:

    m gitlab:fluidattacks/universe@trunk /skims scan config.yaml

    Once the scan completes, the results are saved in a CSV file named Benchmark_1.2-Fluid-Attacks-v2024 located in the results/ folder of the cloned repository.

  6. Install the OWASP plugin: You need an OWASP plugin to create a scorecard from the results. Fluid Attacks currently uses a modified version of the plugin to ensure compatibility with the latest scanner version. (A pull request to add native support is open in the official OWASP Benchmark repository.)

    To install the plugin locally (make sure you have Maven installed):

    cd ..
    git clone https://github.com/alejolagosm/BenchmarkUtils.git benchmark_utils
    cd benchmark_utils
    mvn install
    cd ../benchmark

  7. Add YAML file to use the plugin: Inside the benchmark repository, add a YAML file named benchmark_config.yaml with the following contents:

    expectedresults: expectedresults-1.2.csv

  8. Run the plugin to generate the scorecards:

    mvn org.owasp:benchmarkutils-maven-plugin:create-scorecard -DconfigFile=benchmark_config.yaml

  9. Open the results in your browser: For example:

    On Firefox:

    firefox scorecard/Scorecard_Home.html

    On Google Chrome:

    google-chrome-stable scorecard/Scorecard_Home.html


Alternatively, you can simply run a script provided in the Fluid Attacks GitLab repository:

m gitlab:fluidattacks/universe@trunk /skims/benchmark/owasp

This script generates the scorecards (which you can open as described above in step 10) and a JSON file named benchmark.json summarizing the vulnerability scanner results.


Free trial message
Free trial
Search for vulnerabilities in your apps for free with Fluid Attacks' automated security testing! Start your 21-day free trial and discover the benefits of the Continuous Hacking Essential plan. If you prefer the Advanced plan, which includes the expertise of Fluid Attacks' hacking team, fill out this contact form.