Accuracy SLA

Description

F2 and F0.5 scores of at least 90% are achieved in reports of a client's software's risk exposure and vulnerabilities, respectively.

Since January 2025, this service-level agreement (SLA) indicator seeks to address management's pain (e.g., critical and overlooked issues) and developers' pain (e.g., waste of time and effort). The Accuracy SLA is breached when either the F2-score-measured accuracy reporting risk exposure (henceforth, "severity accuracy") or the F0.5-score-measured accuracy reporting vulnerabilities (henceforth, "quantity accuracy") is below 90%, given the criteria presented below.

Criteria

The following conditions must be met for this SLA indicator to apply:

Your group must have the Advanced plan.
Both the source code and its related environment must be accessible.
The environment must be paired with the code, i.e., the environment must correspond to the provided branch.
The environment must be stable, with no unsolved events (i.e., situations preventing testing) for at least 80% of business days.
All data required for application flows under continuous testing (e.g., credentials, input fields) must be complete and usable.
Remote access without human intervention must be enabled (e.g., no CAPTCHA, OTP).
A 100% Health Check must have been performed on a group potentially affected by a false negative.
An average of 400 weekly changes per author must have been made from the start of service to the potential false negative report.

Details

In addition to the general measurement aspects, the following is taken into account to measure this SLA indicator:

The risk exposure caused by vulnerabilities is calculated using the formula CVSSF = 4^(CVSS-4).
Accuracy is calculated based on false positives, false negatives, and the F-Score model.
In groups subscribed to black-box testing, vulnerabilities detectable only via source code review are not considered false negatives.

Indicator calculation

Fluid Attacks does not measure the Accuracy SLA through only false positive or false negative rates, as these two are complementary to the assessment of security testing accuracy. For instance, a good false positive rate may be achieved while no actual vulnerability is found (i.e., even if no alarms were raised, even legitimate ones). And a good false negative rate may be achieved when everything is flagged as a vulnerability (i.e., even if all possible false alarms were raised).

This SLA indicator involves two measures: Severity accuracy and quantity accuracy. Accordingly, the former is equal to the value of the F2 score, which places significant weight on false negatives. Using this model and involving the CVSSF metric allows Fluid Attacks to measure accuracy in the terms that matters most to your organization's management, i.e., that critical issues have not been overlooked. Quantity accuracy, in contrast, is equal to the value of the F0.5 score, in which false positives have more weight. This model, and using the number of vulnerabilities, allows Fluid Attacks to measure accuracy in a way that interests your development team the most, i.e., that reported issues to fix are indeed correct.

Severity accuracy is calculated as follows:

Compute the CVSSF for each individual vulnerability using the formula CVSSF = 4^(CVSS-4).
Calculate the total CVSSF for True Positives, False Positives and False Negatives.
Compute the following intermediate indicators:

Precision: True Positives / (True Positives + False Positives)
Recall: True Positives / (True Positives + False Negatives)

Compute the F2 score with the formula 5 * {[Precision * Recall] / [(4 * Precision) + Recall]}

Quantity accuracy is calculated as follows:

Calculate the total number of vulnerabilities differentiating by True Positives, False Positives and False Negatives.
Compute the following intermediate indicators:

Precision: True Positives / (True Positives + False Positives)
Recall: True Positives / (True Positives + False Negatives)

Compute the F0.5 score with the formula 1.25 * {[Precision * Recall] / [(0.25 * Precision) + Recall]}

Free trial