This section describes everything we do in our development cycle to reach a high integrity level.
We have a Git repository for all our applications. By taking this approach instead of dividing applications into smaller repositories, we achieve the following:
As mentioned in the "Everything as Code" subsection, we try to keep as much material as possible versioned in our Git repository. Application integrity becomes a matter of keeping a healthy source code. This is after making the source code the only variable that affects an application.
Please read this.
All our infrastructure is versioned in our Git repository written as code. Such code can be deployed anywhere and has all the properties of any other source code, such as auditability, history, revert capabilities, etc.
By having our infrastructure written as code, we can recreate it on a daily basis. Regenerating our infrastructure every day brings the following advantages:
The infrastructure code can be audited, and changes can only be made by changing such code. This provides full transparency on what was changed, when, and who did it. Also, no administrative protocols like ssh or administrative accounts are needed.
We run an Application Build Process for every change a developer wants to introduce to the application's source code via a merge request. The Application Build Process includes steps like the following:
By always building and testing everything, we can guarantee that every change is compliant with the application's quality standards.
We recognize that not all the steps of a building process can be automated, especially some tests. That is why developers also need to ask a peer to review their code changes before their merge requests can go to production. Reviewers usually evaluate code quality, commit message coherence, and other semantic properties of the change. (At least one review is required. Anyone other than the author of the Merge Request can approve.)
Peer reviewing also becomes an activity where product teams discuss philosophies, standards, and future plans for the application. This is an ideal space for senior developers to guide juniors on the right path.
In addition to running an automated building process for every change, we also run an automatic deployment process. Once a merge request is accepted, an additional Continuous Deployment (CD) pipeline is triggered, automatically deploying a new production version based on the new source code.
Instead of having long-term development environments like staging, we use testing environments that are created during a Continuous Integration (CI) pipeline. We call them ephemeral environments, as they only exist in pipeline time. These environments are created on demand when a developer triggers a CI pipeline. They are also written as code, regenerable and immutable, allowing us to certify that a new version of an application is stable and secure before it reaches production environments. Once a change reaches production, its ephemeral environment is destroyed forever. Ephemeral environments only contain mocked data; they do not share any data with production.
We use Trunk Based Development to keep only one long-term trunk branch. That branch is the source of truth regarding what code is running in the production environments.
Merge requests made by developers cannot be bigger than 150 deltas of code. A delta consists of either a removed or an added line of code. The following are some advantages of working with micro-changes:
Developers can only have one short-term branch with their names (talentatfluid) for every application. Once they develop a portion of code (150 deltas maximum), they run the CI phase, create a merge request, and ask for peer review. If everything goes well, their branch is merged to the trunk branch, their changes are deployed to production, and their short-term branch is deleted.
Some of our dependencies do not require OS libraries like libc. Instead, they are completely built from scratch, thus guaranteeing total reproducibility.
Additionally, these dependencies do not require any administrative privileges like sudo. They are entirely built on user space, considerably reducing the possibility of compromising OS core files.
All external dependencies are pinned to a specific version (this is highly related to the immutability property), meaning that to update a dependency, a developer must do the following:
Change the version in the source code.
Run all CI tests on the generated ephemeral environment with the new dependency version.
Get the change approved by a colleague after running a peer review.
If all tests and the peer review are passed, a new production version with the updated dependency will be automatically deployed.