Cluster | Kubernetes Infrastructure Platform | Fluid Attacks Help

Cluster

luster is in charge of providing a company-wide Kubernetes Cluster.

Public Oath

Fluid Attacks will constantly look for new stack that simplifies serving applications to the Internet in a secure and automated manner.

Architecture

cluster arch
  1. The module is managed as code using Terraform.
  2. We have one Kubernetes cluster called common-k8s that is shared by all the components.
  3. The cluster is hosted on AWS.
  4. The cluster is divided into namespaces, which keep resources in isolation from other namespaces.
    1. The default namespace is unused, we try to put things into a namespace appropriate to the product.
    2. The dev namespace currently holds the ephemeral environments of Integrates.
    3. The prod-integrates namespace holds the production deployment of Integrates, and a Celery jobs server.
    4. The kube-system namespace holds cluster-wide deployments for, load balancer, DNS, node termination handler, cloudflared, observability and autoscaler.
    5. Other kube-* namespaces exist, but they are not used for anything at the moment.
  5. Every namespace runs in a specific worker group whose physical machine instances run on EC2.
  6. The cluster spawns machines on all availability zones within us-east-1 for maximum spot availability.
  7. The cluster supports autoscaling based on several metrics like cpu consumption, memory consumption and queue size.
  8. It provides observability tools for debugging.
  9. It creates AWS application load balancers and Cloudflare DNS records for ingress resources.
  10. It supports a Cloudflare ZTNA tunnel that allows developers to access the AWS VPC via Cloudflare WARP.
  11. Developers can access the cluster via Okta for debugging.


Contributing

Please read the contributing page first.

General

  1. Any changes to the cluster infrastructure and configuration must be done via Merge Requests.
  2. Any changes related to the Platform (deployments, autoscaling, ingress…) for both development and production must be done via Merge Requests.

Components

Our cluster implements:
  1. AWS EKS Terraform module for declaring the cluster as code using Terraform.
  2. AWS Load Balancer Controller for automatically initializing AWS load balancers when declaring ingress resources.
  3. AWS Kubernetes Autoscaler for automatically scaling the cluster size based on resource assignation.
  4. ExternalDNS for automatically setting DNS records when declaring ingress resources.
  5. Kubernetes Metrics Server for automatically scaling deployments like production Platform based on application load (CPU, Memory, custom metrics).
  6. New Relic for monitoring both production Platform and general infrastructure.

Debugging

Connect to cluster
  1. In order to connect to the Kubernetes Cluster, you must:
  2. Login as an Integrates developer using this guide.
  3. Install kubectl and aws-cli with nix-env -i awscli kubectl.
  4. Select cluster by running aws eks update-kubeconfig --name common-k8s --region us-east-1.
  5. Run kubectl get node.
Your input should be similar to this:

kubectl get node
NAME                            STATUS   ROLES    AGE   VERSION
ip-192-168-5-112.ec2.internal   Ready    <none>   58d   v1.17.9-eks-4c6976
ip-192-168-5-144.ec2.internal   Ready    <none>   39d   v1.17.11-eks-cfdc40
ip-192-168-5-170.ec2.internal   Ready    <none>   20d   v1.17.11-eks-cfdc40
ip-192-168-5-35.ec2.internal    Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-5-51.ec2.internal    Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-6-109.ec2.internal   Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-6-127.ec2.internal   Ready    <none>   18d   v1.17.11-eks-cfdc40
ip-192-168-6-135.ec2.internal   Ready    <none>   31d   v1.17.11-eks-cfdc40
ip-192-168-6-151.ec2.internal   Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-6-221.ec2.internal   Ready    <none>   13d   v1.17.11-eks-cfdc40
ip-192-168-7-151.ec2.internal   Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-7-161.ec2.internal   Ready    <none>   33d   v1.17.11-eks-cfdc40
ip-192-168-7-214.ec2.internal   Ready    <none>   61d   v1.17.9-eks-4c6976
ip-192-168-7-48.ec2.internal    Ready    <none>   30d   v1.17.11-eks-cfdc40
ip-192-168-7-54.ec2.internal    Ready    <none>   39d   v1.17.11-eks-cfdc40


Common commands

Most commands have the following syntax: kubectl <action> <resource> -n <namespace>.
  1. Common actions are: get, describe, logs, exec and edit.
  2. Common resources are: pod, node, deployment, ingress, hpa.
  3. Common namespaces are: development, production and kube-system. Additionally, the -A flag executes <action> for all namespaces.
Some basic examples are:
Command Example Description
kubectl get pod -A​ N/A​ Get all running pods
kubectl get node -A N/A Get all cluster nodes
kubectl get deployment -A N/A Get all cluster deployments
kubectl get hpa -A N/A Get all autoscaling policies
kubectl get namespace N/A Get all cluster namespaces
Some more complex examples are:
Command Example Description
kubectl describe pod -n <namespace> <pod>​ kubectl describe pod -n development app-dsalazaratfluid-7c485cf565-w9gwg Describe pod configurations
kubectl logs -n <namespace> <pod> -c <container> kubectl logs -n development app-dsalazaratfluid-7c485cf565-w9gwg -c app Get container logs from a pod
kubectl exec -it -n <namespace> <pod> -c <container> -- <command> kubectl exec -it -n development app-dsalazaratfluid-7c485cf565-w9gwg -c app -- bash Access a container within pod
kubectl edit deployment -n <namespace> <deployment> kubectl edit deployment -n development integrates-dsalazaratfluid Edit a specific deployment
Idea
Tip
Have an idea to simplify our architecture or noticed docs that could use some love? Don't hesitate to open an issue or submit improvements.