Buffer's Engineering Team Has a New Superpower 🦸

Context

Buffer has a lot of interdependent backend micro-services and workers. Some of this can be attributed to real need while others attributed to tech debt.
Often times, a new feature or a code refactor/change in architecture could span multiple micro-services and to truly test a change; all of them need to be deployed in staging first.

Here's how it usually goes for a change spanning services X, Y, and Z

• Alice deploys service X
• Alice deploys service Y
• Bob deploys service X
• Alice deploys service Z

At this point, Alice is ready to test her change, only to discover that service X has already been overridden by Bob's deployment.

Even if engineers coordinated among themselves, they may still be overridden by a merge to the main branch since we consistently keep staging up to date with what's on the main branch.

This considerably slowed down our team and made the local dev environment the only reasonable way to move with a little bit of speed. However, if the changes affected the mobile app then dealing with staging shenanigans was an unavoidable pain (especially for engineers who don’t usually work on the mobile app)!

Introducing: Buffer Isolated Backend Environments (BIBEs) 🥁

Here's how we describe BIBEs to our team

BIBEs (Buffer Isolated Backend Environments) are isolated, fully functional backend environments that mirror either our production or staging deployments — but just for you.

Engineers can create a BIBE whenever they need one! Each BIBE includes

• One pod of each service
• All services communicate with each other just like in prod/staging
• Your changes, your environment — without affecting anyone else

There're two types of BIBEs

• Prod BIBE: Uses production data.
• Staging BIBE: Uses staging data.

Key points:

• BIBEs spin up in ~90 seconds.
• By default, each BIBE stays in sync with their parent env (prod or staging). If a service in their parent env gets updated, it is also updated inside the BIBE!
• Once an engineer deploys their own version of a service inside a BIBE, that service stops syncing with the parent env — while the rest of the services continue to mirror it.
• Each BIBE is a complete redeployment of the Buffer product with all its prod/staging quirks. Engineers can test any area of the product and/or perform multiple checks to confirm everything is good. For example:
  • check dashboards
  • check error tracking software
  • run E2E tests against their BIBE
  • hook their mobile app to a BIBE

The only thing BIBEs doesn't test is production load (but we have another tool for that 😉)!

How much extra maintenance do BIBEs need?

Almost none!

BIBEs fully rely on gluing together off-the-shelf software we already use and maintain anyway with as little glue code as possible.

• When engineers declare a new service, our infra automations automatically include it in all BIBEs. We use terraform and ArgoCD (with a very small config management plugin + kustomize) for this!
• When engineers cleanup an old service, our infra automations automatically remove it from all BIBEs. (Also uses terraform and ArgoCD)
• DNS, SSL Certificates, etc is all taken care of automatically (We use terraform, OctoDNS and some clever structuring to avoid writing extra code that we'd have to maintain later 😉).
• When an engineer references one service from another service, our infra automations automatically understand if they're inside a BIBE and would direct that traffic to the pod inside their BIBE.
  • Note: We use SSM Parameters for configuration but made the decision 3 years ago to invest in our own tooling for parameter resolution rather than use an off-the-shelf solution. Multiple reasons are behind this decision but it means we could manipulate configuration in very powerful ways using very simple means (so no, we don't need a complicated service mesh for this 😂).

What's Next For BIBEs

Currently, BIBEs are still in their initial version. Yet, in the past month, our team has created ~75 BIBEs with pace growing as adoption increases.

BIBEs were made possible thanks to our investment in infrastructure automation, robust parameterization/secrets management, as well as solid processes for provisioning services and the infrastructure for them.

We plan to continue improving BIBEs with better tooling for observability, regression detection, debug-ability, and integration with our other internal tooling.

How does your team deal with staging environments? Let me know :)