Testing - Unit tests

Philosophy

Unit tests focus on verifying the functionality of individual units or components of our software. A unit would be the smallest testable part of a software, such as a function, method, or class. Unit tests in Fluid Attacks must be:

Repeatable: Regardless of where they are executed, the result must be the same.
Fast: Unit tests should take little time to execute because, being the first level of testing, where you have isolated functions/methods and classes, the answers should be immediate. A unit test should take at most two (2) seconds.
Independent: The functions or classes to be tested should be isolated, no secondary effect behaviors should be validated, and, if possible, we should avoid calls to external resources such as databases; for this, we use mocks.
Descriptive: For any developer, it should be evident what is being tested in the unit test, what the result should be, and in case of an error, what the source of the error is.

Architecture

Location: To be discovered by the testing framework, test files must be located next to the file to be tested, with the _test suffix, and every test method must start with the _test prefix. Take a look at the add group tests for reference.
Utilities: Some utilities are added to simplify tasks like populating the database, mocking comfortably, and including test files. It allows developers to focus on actually testing the code.
Coverage: Coverage is a module-scoped integer between 0 and 100. Current coverage for a given module can be found at <module-path>/coverage. For example, api/coverage.

Writing tests

See the following examples to understand how to write tests:

1: Populate DB

from integrates.dataloaders import Dataloaders, get_new_context
from integrates.db_model.organizations.types import Organization
from integrates.organizations.utils import get_organization
from integrates.testing.aws import IntegratesAws, IntegratesDynamodb
from integrates.testing.fakers import OrganizationFaker
from integrates.testing.mocks import mocks

@mocks(
    aws=IntegratesAws(
        dynamodb=IntegratesDynamodb(
            organizations=[
                OrganizationFaker(id="test-org-1"),
            ],
        ),
    ),
)
async def test_get_organization() -> None:
    # Arrange
    organization_id = "test-org-1"

    # Act
    loaders: Dataloaders = get_new_context()
    organization: Organization = await get_organization(
        loaders,
        organization_id,
    )

    # Assert
    assert organization is not None

2: Populate S3

from integrates.s3.operations import list_files
import integrates.s3.operations
from integrates.testing.aws import IntegratesAws, IntegratesS3
from integrates.testing.mocks import Mock, mocks

@mocks(
    aws=IntegratesAws(
        s3=IntegratesS3(autoload=True),
    ),
    others=[
        Mock(integrates.s3.operations, "FI_AWS_S3_PATH_PREFIX", "function", "")
    ],
)
async def test_list_files() -> None: 
    # Arrange
    bucket = "integrates.dev"
    expected_output = ["file1.txt", "file2.txt"]

    # Act
    files = await list_files(bucket=bucket, name="")

    assert len(files) == len(expected_output)
    assert all(file in files for file in expected_output)

3: Parametrize

from integrates.dataloaders import Dataloaders, get_new_context
from integrates.db_model.organizations.types import Organization
from integrates.organizations.utils import get_organization
from integrates.testing.utils import parametrize
from integrates.testing.aws import IntegratesAws, IntegratesDynamodb
from integrates.testing.fakers import OrganizationFaker
from integrates.testing.mocks import mocks

@parametrize(
    args=["organization_id"],
    cases=[
        ["test-org-1"],
        ["test-org-2"],
    ],
)
@mocks(
    aws=IntegratesAws(
        dynamodb=IntegratesDynamodb(
            organizations=[
                OrganizationFaker(id="test-org-1"),
                OrganizationFaker(id="test-org-2"),
            ],
        ),
    ),
)
async def test_get_organization(organization_id: str) -> None:
    # Act
    loaders: Dataloaders = get_new_context()
    organization: Organization = await get_organization(
        loaders,
        organization_id,
    )

    # Assert
    assert organization is not None

4: Raises

from integrates.custom_exceptions import OrganizationNotFound
from integrates.dataloaders import Dataloaders, get_new_context
from integrates.organizations.utils import get_organization
from integrates.testing.utils import raises
from integrates.testing.mocks import mocks

@mocks()
async def test_get_organization_fail(): -> None:
    # Arrange
    organization_id = "test-org-3"

    # Act
    loaders: Dataloaders = get_new_context()
    with raises(OrganizationNotFound):
        await get_organization(loaders, organization_id)

@mocks decorator allows us to populate the database with test data using AWS parameters. A clean database will be created and populated for each parameter provided via @utils.parametrize. We got deeper into these decorators and helper methods in the next sections.

Note

Write tests for the specific module you’re working on, as coverage is only calculated for files within that specific module.

Tip

Make sure you use existing tests as a reference for creating your own.

DynamoDB

Integrates database is populated using IntegratesAws.dynamodb in the @mocks decorator. This parameter is an instance of IntegratesDynamodb, a helper class to populate the main tables with valid data:

@mocks(
    aws=IntegratesAws(
        dynamodb=IntegratesDynamodb(
            organizations=[OrganizationFaker(id=ORG_ID)],
            stakeholders=[
                StakeholderFaker(email=ORGANIZATION_MANAGER_EMAIL),
                StakeholderFaker(email=ADMIN_EMAIL),
            ],
            organization_access=[
                OrganizationAccessFaker(
                    organization_id=ORG_ID,
                    email=ORGANIZATION_MANAGER_EMAIL,
                    state=OrganizationAccessStateFaker(has_access= True, role="organization_manager"),                
                ),
                OrganizationAccessFaker(
                    organization_id=ORG_ID,
                    email=ADMIN_EMAIL,
                    state=OrganizationAccessStateFaker(has_access=True, role="admin"),                
                ),
            ],
        ),
    ),
    others=[
        Mock(logs_utils, "cloudwatch_log", "sync", None),    
    ],
)

In the example above, we are populating the database with one organization, two stakeholders, and giving access to both stakeholders to the Organization with different roles.

Every faker is a fake data generator for one element. Parameters are optional to modify your data for your tests (e.g., assigned role in OrganizationAccessStateFaker or the email in StakeholderFaker). A fake name gives a hint about where it should be used in the IntegratesDynamodb parameters.

Note

If any faker is missing, feel free to create a new one in the testing.fakers module and implement a new parameter for the faker on testing.aws.dynamodb module.

The others parameter is a way to list all the startup mocks that you require in your test. In the example above, we are mocking the cloudwatch_log function from logs_utils module to avoid calling CloudWatch directly and always return a None value. Mock is a helper class that creates a mock based on module, function, or variable name, mode (sync or async), and a return value.

Caution

Mock should be used only when it is necessary to avoid external calls or non-testable components.

Mock is not for internal functions or classes. It hides possible bugs that tests can prevent.

This declarative approach ensures isolation. Each test will have its own data and will not conflict with other tests.

S3

Integrates buckets are created for testing at the same time when @mocks is called, and no more actions are required. You can use the buckets in your tests and also load files to buckets automatically before every test run.

To load files to the buckets automatically, you must use:

@mocks(aws=IntegratesAws(s3=IntegratesS3(autoload=True)))

Use the following file structure as a reference:

main.py (logic here)

main_test.py (tests here)

▶ 📁 test_data/

▶ 📁 test_name_1/

📄 file_1.txt (It won't be loaded)

📄 file_2.txt (It won't be loaded)

▶ 📁 test_name_2/

▶ 📁 integrates.dev/

📄 README.md (Loaded to integrates.dev bucket)

▶ 📁 test_name_3/

▶ 📁 integrates/

📄 README.md (Loaded to integrates bucket)

<test_name> directory is searched to load files into the corresponding buckets. For example, a README.md file will be loaded into integrates.dev for test_name_2, and a different README.md file will be loaded into Integrates for test_name_3. This approach ensures both isolation and simplicity in the tests.

Utils

For easy testing, some utilities and decorators are provided.

Use @parametrize to include several test cases:

from integrates.testing.utils import parametrize

@parametrize(
    args=["arg", "expected"],    
    cases=[
        ["a", "A"],        
        ["b", "B"],        
        ["c", "C"],    
    ],
)
def test_capitalize(arg: str, expected: str) -> None:    
    ...

Use raises to handle errors during tests:

from integrates.testing.utils import raises

def test_fail() -> None:   
    with raises(ValueError):
        ...

Use get_file_abs_path to get the file’s absolute path in the test_data/<test_name> directory:

from integrates.testing.utils import get_file_abs_path

def test_name_1() -> None:   
    abs_path = get_file_abs_path("file_1.txt"):

    assert "/test_data/test_name_1/file_1.txt" in abs_path    # True

Use @freeze_time when you want to set the execution time (time-based features).

from integrates.testing.utils import freeze_time

@freeze_time("2024-01-01")   
def test_accepted_until() -> None
    ...

Running tests

You can run tests for specific modules with the following command:

integrates-back-test <module> [test-1] [test-2] [test-n]...

where:

<module> is required and can be any Integrates module.
[test-n] is optional and can be any test within that module.

If no specific tests are provided, this command will:

Run all tests for the given module.

Fail if any of the tests fail.

Generate a coverage report.

Fail if the new coverage is below the current one for the given module (Developer must add tests to at least keep the same coverage).
Fail if the new coverage is above the current one for the given module (Developer must add new coverage to their commit).
Pass if the new and current coverage are the same.

If specific tests are provided, this command will:

Only run the given tests.

Fail if any of the provided tests fail.

Skip the coverage report generation and evaluation.

Tip

Try running tests for mailmap with:

integrates-back-test mailmap

Debugging tests

You can also run Integrates tests with the debugger from any VSCode-based IDE. To do so, all you need to do is press F5 (while the cursor is within a file from integrates workspace) and select Debug integrates tests (specific module) from the dropdown. Before launching, the IDE will ask you to select the module you want to test. After this, a debugging console will be prepared with the flakes development environment, which may take a while and might even imply a failure in the first attempt. With a successful launch, you will be able to set breakpoints and inspect the code as you would do with any other debugging session.

Note

Within a debugging session, the code coverage is not calculated, since the instrumentation from pytest-cov seems to affect the proper functioning of the debugger and its breakpoints.

Read recommendations for testing actions not caught by unit tests.

More about the Integrates backend