Lessons for planning and executing sweeping architecture improvements
LiveRamp has embarked on a quest to move the platform powering our core products into the future. We envision all of the core functionality of our backend system easily available to developers building applications on top of the capabilities we expose. This will enable an explosion of new development and products sharing our common platform. It is an essential step to unlock future scaling for our engineering team and our product. It will improve engineer’s lives, enable critical new product features, simplify customer experience, and unlock strategic growth opportunities for our business.
This sounds amazing, but what does it actually mean and how do we go about doing it? It’s easy to talk about the ideal of a perfect platform, but it’s much more difficult to make it a reality. How do we figure out the next evolution to the LiveRamp platform, and how do we actually get there? How do we rally the organization to move in a common direction? How can we ensure the project doesn’t get out of hand while still achieving our ambitious goals?
This blog post begins to answer these questions by examining the approach we took to define the project, how we got buy-in from business stakeholders, and our strategy to execute the work. The most significant factor allowing us to make progress is joining the technical vision with the product vision into a compelling shared story. By weaving technical and product vision into one, we’re able to commit to a year long investment in our architecture without sacrificing product enhancements essential to the business.
How we got where we are today.
LiveRamp originally developed a system to support the use case of Data Onboarding. We built a data pipeline, domain model, and UI that performs Data Onboarding.
Over the years, we’ve expanded to support many new use cases. DataMarketplace, TV, Business to Business, LiveRamp SafeHaven, and Measurement products are some examples. Each of these applications makes use of the systems originally built for Data Onboarding. In this way, our systems are already operating as a platform. With each new use case we’ve made incremental changes to our backend to make it more generic and reusable.
Despite these improvements, the systems remain intensely coupled with the Data Onboarding use case they were built to support. This means all the other use cases aren’t supported as well as they should be. It also makes it challenging and laborious to introduce each new use.
We’ve become convinced that we need to take a more radical approach to modernize our architecture. We need to decouple and generalize major parts of our system, and develop a more flexible domain model that can be interacted with via clean RESTful APIs. This is the problem we are solving.
Getting buy-in for a year long project.
We were fortunate to have alignment across all levels about the value of developing a more extensible, API driven platform. It’s a major part of our product and engineering strategy. The challenge, though, was individual teams struggled to prioritize taking steps on the project over their own tasks. The scope of this project was simply too large for teams to justify prioritizing in isolation.
Achieving our long term goals will require a fundamental shift in systems spanning our entire engineering team. It will require a significant, coordinated effort that will involve at least 20 of our engineering Scrum teams, and take up to a year. We need a sound strategy to prioritize, plan, and execute such a large and ambiguous project.
To empower teams to prioritize work, we had to make the project more real for them and their stakeholders. We needed to understand the overall technical vision, the product features delivered, and the roadmap between teams. With this information a team can recognize the full value their work will have within their team, but also for downstream teams in the project. This allows teams to point to their role in the broader roadmap and more easily prioritize their part of it.
How we made the project more real.
The first step in design was looking at the history and limitations of our system and the most important features we couldn’t support cleanly. We leveraged this input to define concrete deliverables for the project. By staying realistic and focused, we developed an achievable design and roadmap that delivers the most critical product value.
The obvious challenge here is that the problem spans a majority of our system. Any holistic solution requires design and development from most of our teams. The project is well outside of any individual team’s scope, so we began with a group of several senior architects and technical leaders. Their goal wasn’t to perform the design themselves, but rather to be accountable for finding a solution together with teams. Their role was to coordinate, direct, and provide perspective, experience, and vision.
We knew that a project involving so many teams would only be successful if we could meaningfully engage all of them with the problem. We decided to convene technical leaders from each major area of the system, and charge them with creating the actual design and roadmap.
To focus their efforts, we carefully scoped the problem space, and provided a guiding philosophy. The exact scoping is relevant only for our particular effort, but the philosophy behind it is generally applicable.
- Prioritize decoupling systems and reducing tech debt before additional features.
- Building hacks on top of an overloaded system takes us further from our architectural goal. Instead of forcing features where they don’t make sense, we want to enable them along the path to a better state.
- Determine the right abstractions and interfaces before worrying about implementation roadblocks.
- This is a simple idea. We first define the correct abstractions and interfaces without holding ourselves back by current tech debt. Once we know where we are going, we plan how to get there.
- Begin with something realistic over idealistic. (Good, not perfect).
- It doesn’t matter how elegant the design is if it doesn’t solve real problems for our product and technology, or if there’s no achievable path towards it. The goal is to marry business priority with technical improvement to move everything forward together.
- Instead of trying to build the perfect system, we’re trying to build a system that’s easy to change as we learn new information.
None of these principles are radical or new. The difficulty lies in holding ourselves to them in the face of substantial complexity and scrutiny. If we fail to do so we will inevitably stumble and fall.
With these guidelines in mind, the technical leaders came together and developed the domain model and key interfaces that are realistic and achieve our goals. We first established several high level abstractions that make up our core functionality. We wrote the interfaces between these abstractions so we could separate our efforts to design the internals of each independently.
With abstractions and interfaces agreed upon, we next worked together to untangle dependencies and work out a plan to migrate our system. We broke the project up into a list of major milestones. We then ordered the milestones based on dependencies and what order drives the most incremental product value. This led to a roadmap for the execution of this project.
With a clear roadmap articulated, we could now show stakeholders a clear path forward with significant value. This allowed us to prioritize the work on individual teams, and shift our focus towards actually getting it done.
Best practices for executing a project this large.
We’ve designed, planned, and prioritized the work, and we’re moving on to executing our plan. Execution presents a monumental challenge. We will need to juggle the completion of ~20 major milestones from many teams with significant interdependencies. We expect the process to take multiple quarters, and if any piece falls behind we jeopardize the entire project. Finishing design and planning was a significant accomplishment, but the success or failure will come down to execution. We are following a number of best practices as we move forward to minimize the inherent risk, and to make continual progress towards our goals.
Here are a number of the practices we’re holding ourselves to:
- Set appropriate expectations with stakeholders.
- It is extremely important to keep the right mindset and expectations. This is a large challenge, but it’s also achievable. There will be ups and downs, so it’s critical we maintain focus.
- This is a long and difficult project, and we have to approach it with it’s massive scope in mind. We’ve broken it down into reasonable milestones that will require continued effort. We will realize value all along the way, but must keep our focus on the long term goals we’re after.
- Isolate risk by breaking the project into smaller milestones.
- This allows us to unlock incremental value and makes it less of an all or nothing gamble.
- We’re particularly focused on isolating dependencies. Frontloading decoupling work frees up teams to move independently.
- Where there are dependencies, we’re rushing a focus of resources – engineers, architects, and technical project managers.
- Maintain disciplined and consistent prioritization.
- By articulating how this project improves our product and development, we maintain motivation to achieve our goals.
- By being disciplined and realistic when we prioritize work against other projects, we can ensure the necessary amount of focus remains to accomplish our goals.
- When we see resourcing mismatches with the global priority of this project, we can move resources to the teams that need it.
- Providing teams external resources so no team becomes a blocker.
- We have dedicated coordination throughout execution by architects, team leads, TPMs, and product managers.
- We’ve established an API team dedicated to building shared tooling and empowering the efforts of each team involved. They will also be embedding in teams to lend extra resources.
Following all of these guidelines, we still have a tough challenge ahead of ourselves. There are always surprises, good and bad, in a project this large. We’ll need to improvise, iterate, and adjust our strategy throughout all phases of the execution.
There are always risks when aspiring to achieve something significant, but we’re ready to hit the ground running. We’ve established the necessary commitment, focus, and direction to get this done. We have a unique opportunity to achieve something remarkable for our product and technology, and we’ve put ourselves on a path to success. I’m excited for us to get started and confident we’re set up for success.