While microservices architecture offers significant benefits for operating complex large-scale systems, it also introduces specific challenges. Monzo, which runs over 2,800 microservices, has greatly benefited from this architecture but faced complex problems during migrations from obsolete to new libraries. In this article, we’ll explore the strategy and core technologies Monzo used to address these migration issues.
Monzo’s Centralized Migration Strategy
Monzo has adopted a centralized migration approach, preferring to have a single team lead migrations rather than leaving the responsibility to individual service owners. This minimizes the overhead of coordination and the risk of project disruption during the migration process, enabling faster and more efficient migrations.
1. Monorepo and Consistent Technology Choice
Monzo manages all service code using a monorepo, making large-scale refactoring much easier and playing a crucial role in maintaining a consistent technology environment. All services are written in **Go**, and automation tools are easier to use because they all use the same library version. The monorepo provides an ideal environment for consistent refactoring since all services needing changes can be handled at once.
2. The Need for Centralized Migration
In large-scale migrations, many services are changed simultaneously, which can result in project disruption and inconsistency. To prevent this, Monzo prefers a centralized migration approach. This ensures consistency by having a single team lead the migration, preventing individual service owners from handling it themselves.
Monzo’s team relies heavily on automation tools during the migration process. These tools maximize efficiency by automating tasks like large-scale service deployment or rollback checks. Maintaining technical consistency is critical, ensuring that all services are migrated in the same way.
Step-by-Step Migration Procedure
1. Migration Planning and Alignment
Migration requires careful planning. Monzo runs its [migration planning process transparently], allowing all engineers to contribute. Migrations are planned through proposals and architecture reviews. The proposal is shared on a Slack channel where everyone in the company can give feedback, and critical changes are discussed in synchronous architecture review meetings.
2. Wrapping Legacy Libraries
Before installing a new library, Monzo wraps the legacy one to ensure compatibility and make redeployment easier. This allows a gradual migration to the new library through the existing one, avoiding the need to update every call site at once. The wrapped library is instrumented with Monzo’s telemetry library.
3. Updating Call Sites
Commonly referenced functions and types are refactored via [automation tools], primarily using `gopls` and `gorename`. Functions with less common patterns are manually migrated, and in some cases, switched to the existing API. This minimizes confusion during the migration process and helps maintain code consistency.
Massive Service Deployment and Automation
Mass deployment of services is a key element in Monzo’s migration strategy. Once a service is verified to support the new library, changes are applied across all services using automated asynchronous batch jobs. Automated rollback checks allow quick recovery in case of an error. For large-scale changes, Monzo uses progressive roll-forwards and fast rollbacks to quickly address issues.
Fast Rollout Through Configuration Control
Instead of using traditional deployment methods, Monzo rolls out new features via its configuration system. Configuration-based rollouts are very fast, with all services refreshing their configurations every 60 seconds. This allows for immediate rollbacks if necessary, and supports the ability to activate new features for only a subset of users.
Conclusion: The Importance of Consistency and Automation
Monzo’s success in migrating over 2,800 microservices was due to its consistent technology choices and large-scale automation. The centralized migration, led by a single team, maintained system consistency and enabled efficient migrations. Monzo also heavily relied on automation tools to manage large-scale deployments and rollbacks.
This approach allowed Monzo to perform quick and safe migrations even in a large-scale system, and it’s a strategy that other companies could also benefit from adopting.
References: monzo, “How we run migrations across 2,800 microservices”