Table of Contents
Overview
At Cashfree Payments, we wanted a system where teams could seamlessly develop new features, test, and go live. This includes being able to work on multiple services, test the feature independently without interrupting or conflicting with other developers, have visibility and control over deployments, and hassle-free rollbacks.
What were we trying to achieve?
- Using an effective Branching Strategy
- Increasing visibility and control over the deployments
- Standardizing infrastructure components across environments
- Automation of new environment creation
Why did we need a better branching strategy?
We have over 100 microservices with close to 100 prod deployments every day. Often multiple developers work in the same service on different features. As our release timelines are shorter, we needed faster and more frequent deployments.
We used to have dedicated branches for non-production environments. One of the major issues with this approach was a scenario where one developer merged the changes in the non-prod branch (say beta), and after testing some bugs were detected. There are 2 options here:
- Keep the code in beta and fix the bugs, which means all other devs have to wait till all the bugs are fixed, and no one else can use the beta branch till then.
- Revert the code in beta, this is another pain point as in case of conflicts, code revert is not straightforward. Sometimes we have to revert by cherry-picking, and we all know how much of a headache it can be.
Hence, using dedicated branches per env was not a viable option. We needed to use a strategy that could resolve the above issues.
Let’s deep dive into how we did that and improved our Software Development Life Cycle at Cashfree Payments. Read on for Part 1.
Using an effective Branching Strategy
Branching Strategy helps developers to propagate the changes in their independent branches through a set of environments (Test, Stage, Sandbox, Production).
Every software development team needs a strategy when they use a version control system for managing the codebase. As the name suggests, the strategy focuses on how branches are used in the development process.
What criteria did we assess in formulating our branching strategy?
An effective strategy is dependent on several factors such as:
- Organisation’s handling of the codebase
- At Cashfree Payments, we have 100+ microservices that have clear ownership across teams.
- Number of developers & frequency of deployments
- Every team has multiple developers often working in the same service on separate features simultaneously which means a high number of deployments per day, so extreme collaboration is needed while releasing individual features.
- Environments used
- We have requirements for separate environments for Automation, Manual, and Stress testing.
- Quality of Code
- Most features are developed independently, code changes are never merged without code reviews, and all the builds must go through quality checks.
Types of Strategies
The following are 4 most popular industry standards strategies:
GitFlow
GitFlow is a robust branching strategy that works well for large projects with long release cycles, but it can be complicated to set up and maintain. It also requires strict discipline around feature branch creation and merging the PRs.
This strategy has dedicated branches per environment which doesn’t fulfil our use case.
GitLab Flow
GitLab Flow is preferred when we want to maintain multiple environments, have a separate branch, and thus separate builds for each environment. The code changes propagate through the environments synchronously and whenever the main branch is ready to be deployed, we can merge back into the production branch and release it.
This strategy again has dedicated branches so we face the same limitation as GitFlow.
Github Flow
In this flow, we start with the master branch then developers create feature branches from it to isolate their work which are then merged back into the main. This strategy treats the steps of Deployment and Code Merge as an atomic operation that enables quick and complete rollback.
This strategy can be used but there are no steps to ensure that only reviewed and approved code is deployed.
Trunk-based development
This strategy is based on the concept of developers working on a shared trunk by creating feature branches similar to the GitHub Flow, but the major difference is that the code changes are merged before production deployment.
This strategy has the same limitations as Github Flow.
How did we decide our Branching Strategy?
As seen above, we knew one thing for certain we couldn’t use a strategy that uses dedicated branches for each environment. It doesn’t just create bottlenecks in parallel development but increases maintenance too.
No strategy fulfills all of our requirements but Github flow and Trunk-Based are the most suitable ones. So, we decided to follow the trunk-based strategy with a few enhancements of our own.
Trunk-Based Strategy (Enforcing code reviews)
This strategy allows developers to deploy their builds in any non-production environment from separate feature branches. Let’s see how it works.
- For a new task, a feature branch is created from the master branch.
- Developers create a dev branch from the feature branch where they can directly push all their changes.
- After completing dev testing, they raise a Pull Request (PR) against the feature branch.
- Once the PR is approved and merged, a new PR is raised from the feature branch to the master.
- This triggers a pipeline that takes care of creating a new build and config which gets deployed to the QA, Stage, and Sandbox environments.
- Once the sanity, feature, and regression testing are completed, the master branch is locked and the feature branch is merged into the master branch.
- The build is then deployed to Prod-Integration and Production environments, and the master branch is released. This locking feature was added to avoid any conflicts between parallel pipelines running on separate feature branches.
Pros:
- Code review checks – Every single change has to be merged in the feature branches through a pull request which must be reviewed and approved by the peers.
- Build Stability across Environments – The build is created only once from the feature branch and then propagated through the environments.
- Non-blocking testing – If a feature branch is deployed in a test environment but can’t be moved ahead due to a blocker, another feature can be deployed and tested independently.
- Quick hotfixes – Urgent prod issues can be moved quickly via hotfix branches with automated tests.
Cons:
- As we have completely automated our strategy via the CI/CD pipeline, incremental changes have to go through the same process and thus take the same amount of release time as the larger features unless a hotfix branch is used.
For hotfixes, we follow a similar strategy but we skip the deployments to QA/Stage environments.
Summary
We have multiple developers working on multiple features parallelly, often in the same repos. Bugs can be detected during testing and can take time to get fixed which can block others. To handle all these pain points, we finalized the above as our branching strategy.
We even created a pipeline that implicitly follows this strategy and greatly reduces manual effort. Developers can simply raise PRs, sit back and the pipeline takes care of everything from code quality checks, image building, and deployments as well as merging the PRs to the master branch. We’ll be talking about this more in Part 2 of this blog (Increasing visibility and control over the deployments).
This strategy has helped us increase automation and accelerate our development lifecycle.
Many other parameters can decide the efficiency of the SDLC such as the ease of creating new environments, visibility and control over each deployment, and creating templates for infrastructure resources to avoid repeated or duplicate efforts.
Now, you must be wondering how we tackled those other parameters.
Stay tuned for Part 2 of this blog post. Coming soon!
Did this post intrigue you? If so, get in touch with us.
We have some exciting opportunities in store for excellent engineers just like you!
