Rationale

Kubernetes is the system we use for hosting, deploying and managing our applications. It comprises infrastructure solutions like RBAC Authorization, distributed persistent storage, managing resource quotas, managing DNS records, managing load balancers, autoscaling, blue-Green deployments, rollbacks among many others. It allows us to serve and scale our applications in an easy, secure and automated way.

The main reasons why we chose it over other alternatives are:

1. It is capable of deploying complex applications, including related Servers, DNS records, and load balancers in an automated way, allowing us to focus more on the application development and less on the infrastructure supporting it.
2. It can be fully managed using Terraform.
3. It supports Blue-Green deployments, allowing us to deploy applications many times a day without service interruptions.
4. It supports Rollbacks, allowing us to revert applications to previous versions in case the need arise.
5. It supports Horizontal autoscaling, allowing us to easily adapt our applications to the loads they're getting.
6. It supports Service accounts, RBAC Authorization, and IRSA, allowing to give applications permissions to external resources using a least privilege approach.
7. It supports resource quotas, allowing to easily distribute containers among physical machines using a granular cpu/memory per container approach.
8. It has its own package manager, which makes deploying services very easy.
9. It has its own local reproducibility tool for simulating clusters in local environments.
10. It is Open source.
11. It is not platform-bounded.
12. Azure AKS, AWS EKS, GCP GKE, support it.
13. It can be IaaS when implemented under a cloud provider.
14. Migrating it from one cloud provider to another is, although not a simple task, at least possible.
15. It is widely used by the community.
16. It has many open source extensions.

Alternatives

The following alternatives were considered but not chosen for the following reasons:

1. AWS ECS: It is a serverless service for running containers. It is expensive as only one container exists within an entire physical machine. It does not support extensions. It is platform-bounded. It is not Open source.
2. AWS Fargate: It is a serverless service for running containers without administering the infrastructure they run upon. It is expensive as only one container exists within an entire physical machine. It does not support extensions. It is platform-bounded. It is not Open source.
3. AWS EC2: It is a service for cloud computing. AWS EKS actually uses it for setting up cluster workers. It does not support extensions. It is platform-bounded. It is not Open source.
4. HashiCorp Nomad: Currently, no cloud provider supports it, which means that having to manage both managers and workers is required. It takes a simpler approach to orchestrating applications, with the downside of losing flexibility.
5. Docker Swarm: Currently, no cloud provider supports it, which means that having to manage both managers and workers is required. It takes a simpler approach to orchestrating applications, with the downside of losing flexibility.

Usage

We use Kubernetes for:

1. Hosting our platform
2. Automatically deploying ephemeral environments on CI/CD workflows
3. Automatically deploying DNS records for applications
4. Automatically deploying load balancers for applications
5. Automatically scaling worker nodes based on application load
6. Running application performance monitoring using New Relic

We do not use Kubernetes for:

1. Rollbacks: We should version production artifacts in order to be able to automatically return to a previous working version of our applications.
2. GitLab Runner: It was slow, unreliable and added too much overhead to workers. We decided to go back to Autoscaling Runner.
3. Chaos Engineering: In order to harden ourselves against errors, we should create a little chaos in our infrastructure.