Description
F2 and F0.5 scores of at least 90% are achieved in reports of a client's software's risk exposure and vulnerabilities, respectively.
Since January 2025, this service-level agreement (SLA) indicator seeks to address management's pain (e.g., critical and overlooked issues) and developers' pain (e.g., waste of time and effort). The Accuracy SLA is breached when both the F2-score-measured accuracy reporting risk exposure (henceforth, "severity accuracy") and the F0.5-score-measured accuracy reporting vulnerabilities (henceforth, "quantity accuracy") are below 90%, given the criteria presented below.
Criteria
For this SLA indicator to apply, your group must be subscribed to the Advanced plan. Additionally, false negative and false positive reports must meet the defined conditions to be considered as true. (See the articles False negatives and False positives for details.)
Details
In addition to the general measurement aspects, the following is taken into account to measure this SLA indicator:
- The risk exposure caused by vulnerabilities is calculated using the formula CVSSF = 4^(CVSS-4).
- Accuracy is calculated based on false positives, false negatives, and the F-Score model.
- In groups subscribed to black-box testing, vulnerabilities detectable only via source code review are not considered false negatives.
Indicator calculation
Fluid Attacks does not measure the Accuracy SLA through only false positive or false negative rates, as these two are complementary to the assessment of security testing accuracy. For instance, a good false positive rate may be achieved while no actual vulnerability is found (i.e., even if no alarms were raised, even legitimate ones). And a good false negative rate may be achieved when everything is flagged as a vulnerability (i.e., even if all possible false alarms were raised).
This SLA indicator involves two measures: Severity accuracy and quantity accuracy. Accordingly, the former is equal to the value of the F2 score, which places significant weight on false negatives. Using this model and involving the CVSSF metric allows Fluid Attacks to measure accuracy in the terms that matters most to your organization's management, i.e., that critical issues have not been overlooked. Quantity accuracy, in contrast, is equal to the value of the F0.5 score, in which false positives have more weight. This model, and using the number of vulnerabilities, allows Fluid Attacks to measure accuracy in a way that interests your development team the most, i.e., that reported issues to fix are indeed correct.
Severity accuracy is calculated as follows:
-
Compute the CVSSF for each individual vulnerability using the formula CVSSF = 4^(CVSS-4).
-
Calculate the total CVSSF for True Positives, False Positives and False Negatives.
-
Compute the following intermediate indicators:
-
Compute the F2 score with the formula 5 * {[Precision * Recall] / [(4 * Precision) + Recall]}
Quantity accuracy is calculated as follows:
-
Calculate the total number of vulnerabilities differentiating by True Positives, False Positives and False Negatives.
-
Compute the following intermediate indicators:
-
Compute the F0.5 score with the formula 1.25 * {[Precision * Recall] / [(0.25 * Precision) + Recall]}