How Borneo Handles Production Deployments for our customers for the modern cloud

Praveen Thuwat — 4/5/2023 — 7 Min Read

And how it will benefit our customers in the long run

It is well-known that distributed systems are complex, making deployment, testing, and development challenging. It would be best to have a high level of automation and modern cloud-native deployment toolsets to handle complex distributed system deployments.

All Borneo applications are distributed, composed of many AWS resources and micro-services allocated across multiple availability zones. On top of that, as the Borneo software handles the customer data directly, most deployments are on-premise, which further complicates the deployment situation.

If the Borneo deployment software doesn't address these challenges pragmatically, they result in one or more of the following undesirable outcomes:

Error-prone configuration sharing: The Borneo deployment team has to share many artefacts and steps before deployment, which can overwhelm a new user, and potentially error-prone.
Inflexible pre-requirements: Various tools like terraform, helm CLI, kubectl, terragrunt etc. have to be of compatible versions with the Borneo deployment codebase.
Time-wasting manual pre-steps: Because of these, any successful deployment takes more than one joint session, often resulting in the loss of many hours on both sides.
Time-wasting post-deployment activities: Similarly, activities like software upgrades and software life cycle management and requires constant attention and assistance from the Deployment team.
Delivery delays: Frequent deployment upgrade delays and inability to promptly deliver new features and bug fixes.
Unreliable deployments: Due to the lack of version control mechanism for managing cluster or customer-specific remote deployment configurations.

All these can potentially lead to unpleasant deployment experiences and fewer customer conversions.

How does Borneo Handle production deployments?

The Borneo deployment model is based on a layered architectural model where each inner layer exists independently. This helps us quickly change the internal workings without affecting the outer layers and transparently reduce the complexity of deployments by bringing invaluable abstractions.

(Abstractions through layered Architecture)

At the core of the deployment, we use reusable Terraform modules and code to create the required AWS infrastructure and deploy the micro-services to Kubernetes(EKS) cluster using helm charts. All the Helm Charts and related Terraform codes are hosted on a GitHub repository accessible only to customers through a unique GitHub deploy key. We create individual GitHub deploy keys per customer for easy repo access and management, and this also helps us quickly revoke customer access without affecting other users if needed.

The terraform backend state-file management, terraform per stack configuration, and installation of Terraform binary is managed by a Command Line Interface(CLI) tool called borneo, developed and managed by the Borneo deployment team. This CLI tool abstracts away a lot of complexity for the user and simplifies the overall deployment experience using various automation and integrations, and some notable ones are:

Automatic installation of Borneo deployment Terraform binary version: the user only needs to worry about installing a specific version of the borneo CLI tool and nothing else. The tool will install other required deployment tools and versions if necessary.
Automatically create Terraform S3 bucket: for remote Terraform state file storage and deploy configuration storage.
Generation of initial deploy configuration: generated from user input and backup the generated configuration for future use(S3 backup store keeps multiple configurations).
Automatically download deploy configuration from backup (S3): for if the local configuration goes missing or the user switches the workstation.
Automatically download the necessary deployment code: downloaded from GitHub repo with the version provided.

Another important reason for the need for a wrapper tool is the inability of Terraform to independently deploy and upgrade each micro-service without changing the base terraform configuration root directory (can't deploy code from subdirectories) --- An essential goal if you want to manage the deployment lifecycle of each micro-services independently without creating multi-level gigantic terraform modules.

The below diagram shows how the CLI tool works at a high level.

The borneo CLI tool is written in Go Programming language and is reasonably fast, and compiles down to a simple binary without any other run time dependency. This makes go binaries easy to ship and distribute. Also, go has enabled us to use some of the Hashicorp curated terraform libraries directly. As of this time of writing, these libraries are only available if you use the Go Programming language.

Deployment in action

Users need to download the borneo CLI binary based on their workstation CPU architecture and GitHub deploy key before starting the deployment (need Borneo deployment teams to help here, including allowing access to Borneo ECR repo). Borneo deployment team uses GitHub actions and goreleaser open-source tool to automate the build and distribution of borneo binary for different platforms. Once the necessary files are downloaded, the user needs to gain the AWS API access privileges to their AWS account to deploy the new Borneo application.

Once the above steps are completed, the user can start with first-time initialisation. If the CLI is not in the user's default PATH, the user can add the CLI binary location as an additional step. This enables the user to use the deploy tool as any standard terminal command.

Step 1: Initialisation

$ borneo init

During the first run, the tool will install the terraform binary, download the deploy code from the Borneo GitHub repo, and ask a handful of questions from the user like cluster name, AWS region, user email ID, etc. and creates a local copy of initial deploy configuration. The tool also saves a copy in the S3 bucket for retrieval if the local copy goes missing or gets corrupted. If needed, the user can rerun the init command multiple times without any side effects, as this action is idempotent. The init step usually takes less than a minute to complete. After the initialisation, a user can proceed with the deployment.

Step 2: Deploy the Application

$ borneo deploy

If there is no option provided with the deploy command, by default, it will start deploying the whole Borneo Application stack, including the underlying AWS infrastructure. The below flowchart explains the deployment steps in detail.

A typical deploy process usually takes up to ~30 to 35 min to complete, enabling our users to enjoy and assess the value of Borneo applications quickly.

Benefits for the customer

The main highlights of the Borneo deployment model are following:

A one-stop solution for deploying Borneo distributed application and making it genuinely self-service --- A CLI similar to apt or yum.
Easy deployment bootstrapping --- No exchange of huge deployment code and steps before the actual deployments
Share only the deploy CLI binary( i.e., based on the customer workstation CPU Arch) and the required GitHub deploy key. The deploy tool will automatically download the required deployment code and install any means necessary for the deployment.
Reruns are safe, and all actions are idempotent, i.e., a trait of any modern deployment tool.
Deployments are reliable and predictable --- Transparent logging and zero race conditions between dependent component deployments.
Quick deployment process by grouping independent components and deploying them in parallel.
Hassle-free uninstallation of deployed software --- This is very important if a customer wants to quickly assess the value of Borneo applications in a Sandbox environment and uninstall cleanly to avoid any AWS recurring costs.
Easy upgrade and zero downtime deployments --- Kubernetes rolling update with readiness probe to avoid any potential downtime
Easy to integrate with modern Continuous Deployment tools like Spinnaker or AWS Code Deploy. Internally Borneo Deploy team uses the same deploy code and CLI and uses Spinnaker to manage various deployments.

Conclusion

Using optimal deploy tool abstractions, we could abstract away most of the complexity of deploying distributed Borneo applications for our users. Also, underneath, without overwhelming the user, this model enabled us to utilise many modern cloud-native open-source deployment toolsets that most customers are familiar with --- helm charts for application packaging, Terraform for infrastructure as code management and Kubernetes based application deployment to avoid any vendor lock-in.