When, why, and how GitHub and GitLab use feature flags
Features are 🍕 Feature flags are the 🍕 📦
Welcome to Product for Engineers, a free newsletter created by PostHog for engineers and founders who want to build successful startups.
Feature flags seem simple… they just toggle features on and off, right?
That’s true, but the bigger the company, the greater the benefits of even this simple functionality, and they don’t come much bigger than GitHub and GitLab.
We recently spent a bunch of time researching how they use feature flags. Here’s what we learned.
This week’s theme is: Using feature flags like a pro
1. Feature flags = less stress AND faster shipping
If features are pizza, feature flags are the pizza box. Imagine if you just fed pizzas right out of the oven into someone’s face… it would be messy.
Both GitHub and GitLab value the safety feature flags provide. By being an easy way to rollback features, they lower the stakes when releasing new features.
This means:
Reduced stress on developers
Greater psychological safety
Fewer failed deployments
Higher rate of shipping
2. Not using them wastes loads of time
Implementing flags isn’t free, but GitLab calculated the cost of fixing an issue without flags is equivalent to developing a whole new feature!
Why? Because, while implementing flags adds a small degree of upfront effort, resolving an outage without them requires:
Reverting the release.
Performing “any cleanups that might be necessary.”
Reverting the commit while “ensuring the main branch remains stable.”
Bringing the main branch back to stability while “ensuring we don't block any future releases until the problem is resolved.”
“As history has shown, these steps are time-consuming, complex, often involve many developers, and, the worst of all: our users will have a bad experience using GitLab.com until the problem is resolved.”
3. Flags > branches for collaboration
Despite their names, neither GitHub nor GitLab uses Git in all possible situations.
Developers could use long-living feature branches to collaborate, but they prefer feature flags because:
They keep code changes small.
Reviews are easier and more frequent.
They limit merge conflicts.
Because of these benefits, feature flags make collaboration between developers easier and increase the throughput of changes.
4. Targeting beyond users
It is common to evaluate feature flags based on the user, but this isn’t the only way. Both GitHub and GitLab also target by “actors.”
Actors are a more expansive target. Examples include:
Organizations
Teams
Enterprises
Repositories
Projects
Apps
Using them instead of users helps create consistent experiences for related groups and situations.
For example, GitHub wants repositories to be the same for everyone using that repository. Therefore, you don’t want new repository features, such as discussions or projects, to be available to some users and not others.
5. When to not use flags
Although feature flags are great, there are times when you shouldn’t use them, including:
New, inaccessible features, such as new API routes.
Low-traffic areas, such as technical doc pages or non-critical settings.
Low-risk areas, such as secondary frontend functionality.
GitHub and GitLab use feature flags in high-risk, high-traffic areas – e.g. significant UI, database, or storage changes.
They also use them when introducing new restrictions or externalities, such as permissions or third-party dependencies.
This protects risky changes while limiting the cost of new changes overall.
6. Rollouts need a process
Feature flags are only one part of a process for successfully rolling out new features. Without the rest of the rollout (and rollback) process, there can still be issues, such as:
Accidentally rolling out the wrong variant.
Implementing the flag incorrectly.
Not rolling back the code when there is an issue.
Removing the flag too early (GitHub provides an example of this)
GitLab’s feature rollout process starts as a rollout plan issue containing owners, stakeholders, expectations, rollout steps for different environments, and rollback steps.
Only after completing this, and refining it with engineers and product managers, does any code get written.
7. Manage flags consistently with scripts
Developers regularly work to turn processes into code, and this is no different for handling feature flags. Both GitLab and GitHub built scripts to make flag-related processes consistent. For example, they wrote scripts to help them enable flags in different environments, disable flags, and clean up unused flags from code (which become technical debt if left too long).
Specifically to GitLab, they use their “ChatOps” service, which runs scripts via Slack commands. This provides easy access to these commands as well as transparency into what commands are being run. They fill their rollout process with ChatOps commands to manage flag values through dev, staging, and production.
📖 More good reads on feature flags
How we ship code faster and safer with feature flags – Alberto Gimeno
How and why GitHub uses feature flags, from reducing deployment risk to testing features to managing the costs of feature flags.
Feature flag best practices and tips (with examples) – Ian Vanagas
Learn how to best use feature flags like grouping changes behind a flag, falling back to working code, using local evaluation, naming your flags well, and more.
Learn more about Feature Flags: The next step in Progressive Delivery – Chrissie Buchanan
What does GitLab mean by “progressive delivery” and what do feature flags have to do with it? Find out more about their guiding philosophy in this post.
Working with feature branches and feature flags – Manuel Augustin
Manuel compares how Yoast SEO plugins use feature branches and flag workflows to ship changes and the advantages and disadvantages of both.
💸 Apply to PostHog for Startups
Get $50,000 of PostHog credits for your startup. That’s:
30 million events. Analyze events, create funnels, build dashboards and more. All in one platform.
50k session replays. Watch user sessions, with console logs and network performance tracking. No extra code needed.
Over 4 billion API calls. Run multivariate experiments and complete safe, staggered rollouts with feature flags.
Words by Ian Vanagas, who is a little too proud of that pizza box metaphor.