Testing

Welcome to our section describing how to test Flags.

The tests in Flags are organized in the flags/test/lib_cspm directory and follow these main components:

Test Groups: Tests are organized into groups based on finding ranges (e.g., 001-099, 100-199, etc.).
Mock Data: Test data for each cloud provider (AWS, Azure, GCP) is stored in flags/test/lib_cspm/data/.
Test Configurations: Template configurations are stored in flags/test/lib_cspm/test_configs/.
Test Results: Expected results are stored in flags/test/lib_cspm/results/.

Running Tests

To run tests locally, use the following command:

m . /flags/test group_name

Where group_name is one of the available test groups, for example:

cspm_findings_001_099
cspm_findings_100_199
etc.

Note

Tests use the test_clean_cspm_cache fixture to disable caching during test execution, ensuring each test runs with a clean state.

Writing Tests

When writing new tests for Flags, follow these steps:

Add Mock Data: Create mock responses for cloud provider APIs in the appropriate directory:

AWS: Uses moto library for service simulation and mocked responses in data/aws/
Azure: data/azure/
GCP: data/gcp/

Define Expected Results:

Add expected finding results in results/ directory
Use the format FXXX.csv where XXX is the finding number

Add Test Case:

Add your findings to the appropriate test group in test_findings_cspm.py
Use the existing test infrastructure, which handles:

Configuration generation
Mock data injection
Result verification

Cloud Provider Mocking

AWS Mocking with Moto

Flags uses the moto library to simulate AWS services in tests. Moto provides a mock AWS environment that allows testing AWS interactions without making real API calls.

Example of using moto in tests:

from test.lib_cspm.data.aws.moto_patch import mock_aio_aws

# Using moto context manager for AWS service simulation

with mock_aio_aws():

    run_finding("F101")  # The finding will use moto's mock AWS environment

Other Cloud Providers

For other cloud providers, we use custom mock responses:

Azure: Uses custom mock responses defined in data/azure/
GCP: Uses custom mock responses defined in data/gcp/

Example Test

Here’s a simplified example of how findings are tested:

@pytest.mark.usefixtures("test_clean_cspm_cache")

@pytest.mark.flags_test_group("cspm_findings_100_199")

def test_cspm_findings_100_199() -> None:

    findings = [

        "F101",

        "F148",

        "F157",

        # Add your new finding here

    run_multiple_findings_test(findings)

The test infrastructure will:

Create a temporary configuration
Set up mock environments (including moto for AWS)
Run the finding check
Verify the results against the expected output

Tip

Have an idea to simplify our architecture or noticed docs that could use some love? Don't hesitate to open an issue or submit improvements.