Integrates | Platform & API Architecture | Fluid Attacks Helps

Introduction

Integrates is the product responsible for the platform and its API.

Public Oath

  1. The platform is accessible at app.fluidattacks.com.
  2. Significant changes to the user interface of the platform will be announced via the appropriate communication mechanism.
  3. The API is accessible at app.fluidattacks.com/api.
  4. A six-month notice period will be given for backward-incompatible changes in the API. This includes but is not limited to: deprecating attributes and entities, making optional arguments mandatory, changes in the authentication or authorization system, and so on.

Architecture

integrates architecture

  1. Integrates is a standard client-server application that serves Fluid Attacks’ main API.
  2. It declares its own infrastructure using Terraform.
  3. Sensitive secrets like Cloudflare authentication tokens are stored in encrypted YAML files using Mozilla SOPS.
  4. It is written in typed, functional Python.
  5. It uses Starlette as its main framework.
  6. It uses Hypercorn as its web server.
  7. It serves a GraphQL API.
  8. It has three environments:
    1. Production: The production environment used by end users.
    2. Ephemerals: A testing environment for each developer accessible via the Internet.
    3. Local: A testing environment that developers can run on their machines. Instructions for this can be found here.
  9. There is a Tasks application that performs out-of-band processing for cloning client repositories.
  10. DNS records, cache, custom headers, redirections, and firewall for both production and ephemeral environments are managed by Cloudflare.
  11. A Kubernetes cluster serves:
    1. Production environment.
    2. Ephemerals environments.
    3. Tasks application.
  12. CloudWatch is used for storing production logs.
  13. CloudWatch alerts are used to check the queue size of Tasks. If the queue size goes beyond a given limit, email alerts are sent to developers.
  14. There is one Application Load Balancer (ALB) for Production and one for each Ephemeral environment.
  15. DynamoDB is the main database. It has two tables:
    1. Main for storing all current information.
    2. Historic for storing historical states of entities.
  16. OpenSearch is a secondary search database that mirrors DynamoDB. When changes occur in DynamoDB, a DynamoDB stream triggers a Lambda. Such lambda transforms the DynamoDB data to a compatible OpenSearch format and then stores it there.
  17. For storage, several S3 buckets are used:
    1. client-repositories stores source code repositories from clients.
    2. storage stores blobs uploaded by users (evidence, example files, etc.)
    3. machine-executions stores the results of Skims executions and provides configuration files.
  18. The DynamoDB database is backed up using Backup Vaults by Amazon Web Services (AWS), as promised in 1 and 2.
  19. Out-of-band processing Jobs , like ZTNA repository cloning and machine executions are performed by AWS Batch.
  20. It uses Twilio to send SMS OTPs.
  21. It uses Sendgrid to send email notifications to end users.
  22. Webhooks are supported so end users can get machine-readable notifications to their endpoints.

Contributing

Please read the contributing page first.

Development Environment

Configure your Development Environment.
cd integrates && direnv allow  to source the environment into your shell.

Local Environment

Two approaches for deploying a local environment of Integrates are described below. Either of them will launch a replica of app.fluidattacks.com and app.fluidattacks.com/api on localhost:8001.

All in one

You can use mprocs for handling all components in a single terminal:
  • Run integrates-local.
  • Jobs can be restarted using r.
  • Jobs can be stopped using x.

    • Individual components

    Run each of the following commands within the universe repository in different terminals:
    integrates-back-cli dev
    DAEMON=false integrates-db
    integrates-monitor
    cd views && direnv allow && views
    integrates-storage
    Each terminal will serve a key component of Integrates.

    Accessing the local environment

    1. Go to https://localhost:3000 and accept the self-signed certificates offered by the server.

      2. This will allow the back-end to fetch the files to render the UI.

    2. Go to https://localhost:8001 and, again, accept the self-signed certificates offered by the server.

      3. Now you should see the login portal of the application.

    Ephemeral Environment

    Once you upload your local changes to your remote branch in GitLab, a pipeline will begin and run some verifications on your branch.

    Some of those verifications require a complete working environment to test against. This environment can be found at https://<branch_name>.app.fluidattacks.com, and it will be available once the pipeline stage deploy-app finishes.

    In order to log in to your ephemeral environment, SSO needs to be set up for it. You can write to help@fluidattacks.com with the URL of your environment so it can be configured.

    Notes
    Note
    In case you want to deploy to your ephemeral environment back-end manually, you can run the following command:
    integrates-back-deploy dev 

    Enable SSO on Ephemeral Environments

    Google

    This requires you to have access to the Fluid Attacks organization on Google Cloud.
    1. Access the Google Cloud Console.
    2. Select the Integrates project.
    3. In the left sidebar, go to APIs & Services > Credentials.
    4. On the Credentials dashboard, under OAuth 2.0 Client IDs, choose the client ID not created by Google Services.
    5. Finally, under Authorized redirect URIs, add the URI of the ephemeral environment you want to enable SSO on, you want to enable SSO on:  https://<branch_name>.app.fluidattacks.com/authz_google.

    Azure

    This requires you to have access to the Fluid Attacks organization on Microsoft Azure.
    1. Access the Azure Portal.
    2. Navigate to Azure Active Directory (or Microsoft Entra ID in newer versions).
    3. In the left sidebar, choose App registrations.
    4. Select the appropriate app registration for Integrates.
    5. In the app registration overview, click on Authentication in the left menu.
    6. Under Redirect URIs, click Add URI.
    7. Add the URI of the ephemeral environment you want to enable SSO on: https://<branch_name>.app.fluidattacks.com/authz_azure.
    8. Click Save to apply the changes.

    BitBucket

    This requires you to have access to the Fluid Attacks workspace on Atlassian/BitBucket.
    1. Access your BitBucket workspace settings.
    2. Navigate to OAuth consumers under the Apps and features section.
    3. Select the OAuth consumer configured for Integrates.
    4. In the consumer settings, locate the existing Callback URL field.
    5. Set the callback URL to the ephemeral environment: https://<branch_name>.app.fluidattacks.com/authz_bitbucket.
    6. Click Save to update the OAuth consumer configuration.
    Notes
    Note
    Unlike other login providers, BitBucket only supports one callback URL at a time per OAuth consumer.
    Idea
    Tip
    Have an idea to simplify our architecture or noticed docs that could use some love? Don't hesitate to open an issue or submit improvements.