Database Design | NoSQL Schema & DynamoDB | Fluid Attacks Help

Database Design

Database schema definition

Following the NoSQL design principles, we have facets that represent different entities in the database. Each facet is a separate cue file that contains the schema for a specific entity, even though these items are in the same table. Some tools like NoSQL Workbench for DynamoDB can help you visualize the schema in a more friendly way.

The database schema definition itself can be found in integrates/schemas/database_schema/database_schema.cue. This contains the most basic constraints for the schema.

Tables

Integrates requires two DynamoDB tables: integrates_vms and integrates_vms_historic. You may find its definitions in integrates/schemas/tables and facets contents in integrates/schemas/tables/<table name>/facets.

As best practice, please keep a separate cue package for every facet.

Format your changes with cue fmt

When applying changes to cue files, please apply the formatter:
integrates-schemas-fmt

Export changes to database design

For updating changes made to the schema or table contents to integrates/schemas/database-design.json simply execute:
integrates-schemas-export
This job will fail if the produced output is not up to date with the previous state of the database design, and will be successful if there are no changes. This behaviour is intentional so that CI can catch differences between the current state of the cue schemas and the final content.

Types generation

Since the cue is now our source of truth for the database schema, we can generate types for the database schema to avoid double maintenance. By now, the produced types are Python TypedDict , which should model how the DynamoDB item looks right before being inserted or right after being retrieved. Once generated, types as available in the fluidattacks-core external package.

For producing the types, run:
integrates-schemas-export
This job will fail if types are not up to date. The output will be placed according to the table name in integrates/utils/pypi/fluidattacks-core/fluidattacks_core/types/dynamo/items.py.

Run all

For convenience, you can run all the previous commands fmt, export and gen_types in one go:
integrates-schemas

Validate the current schema without changing the database design

If you want to run schema validations against your running database instance, run:
integrates-schemas-val <stage> <table-name>
Valid stages are dev and prod. For example:
integrates-schemas-val dev integrates_vms
This will help you find items that are non-compliant with the desired schema.

Utilities

NoSQL Workbench

NoSQL Workbench for Amazon DynamoDB is a cross-platform client-side GUI application that you can use for modern database development and operations.

For FluidAttacks users, it can be installed via Self Service on your work machine.

To connect to the database, you must follow the steps below:
  1. Run the DB using:
    2. integrates-local
    or
    DAEMON=false integrates-db
  2. Open NoSQL Workbench
  3. Follow these steps in the GUI:
    1. Operation Builder (in the left pane)
    2. Add connection
    3. DynamoDB local
    4. Type in Hostname: localhost and in Port: 8022 
    5. Connect
    6. Click the Open button in the new connection.
Now you will be able to visualize the tables and information.
Learn more about the cue language at https://cuelang.org/.
If VS Code is your IDE, this extension is recommended: https://marketplace.visualstudio.com/items?itemName=asdine.cue.
A Vim plugin for syntax highlighting is also available at: https://github.com/jjo/vim-cue.

More about the database

  1. Database migrations
  2. Patterns
  3. Streams
  4. Data model

Idea
Tip
Have an idea to simplify our architecture or noticed docs that could use some love? Don't hesitate to open an issue or submit improvements.