Scaling Data Platform Teams: A Step-by-Step Guide (Scaling Your Data Team, Transition #3)
A 5 steps guide to successfully scale your Data Platform team, ensuring it is equipped to handle the demands and complexities of growing organizations.
In previous articles of the series (transformations #1 and #2), we focused on how to organize Data Analysts, Scientists, and, to a lesser extent, Data Engineers. In this article I finally focus on Data Engineering work by explaining how to scale Data Platform teams.
Having re-organized Data Platform teams of different sizes multiple times in my career in different contexts and organizations, I share here my learnings and a standardized step-by-step process you can follow if you are facing similar challenges.
Getting a good start
How you start a Data Platform team will vary from company to company. I recommend hiring one or two Data Engineers to form a Data Platform team at the same time you hire your first Data Analyst(s).
These Data Engineers will be responsible for the data infrastructure of your company (either bought or built). They will strike a balance between the speed at which decision makers need data and tools to extract value out of it and the rigor one should treat data and its lifecycle. For example, if you let anyone use the data infrastructure at their own will, it will quickly get clogged with dirt and underused tables and documents.
In some cases, Data Engineers may also be in charge of moving the initial data from operational systems to the analytical database and creating the first “datasets as products”. These will be used to visualize trends and extract relevant insights for the rest of the company.
Because of that, it is useful to distinguish between platform, “datasets as product” effort or even data visualization work within the team early. Enablement and governance activities are additional areas where Data Engineers might become overburdened, so keep an eye on these as well.
As the company grows and its data needs expand, the workload of the Data Platform team increases. The company might now need tools to manage the lifecycle of Machine Learning models, tools to perform A/B testing, introduce a data observability tool, aggregate data from new domains, etc. With these new responsibilities, the platform team will need to grow.
Steps to scale a Data Platform team
When a Data Platform team grows from 2 to 5–10 engineers (or even 25), roles and duties get hazy, cognitive load increases and work becomes more specialized. The team will frequently break organically into sub-teams also. People will naturally focus on different areas and a new communication system within the team and with stakeholders will emerge.
What occurs is that the organization attempts to treat everyone as if they are on the same team, but they are not. This typically causes the team to lose focus and can also convert them into a bottleneck in the view of stakeholders.
These issues feel very aligned with the reasons for applying a grow-and-split pattern for forming new smaller and more focused teams. Growing and splitting the Data Platform team allows for the distribution of tasks and responsibilities, preventing overload and ensuring that each tool has the right people maintaining it.
But how to do it? How can you effectively organize a Data Platform team of 7 to up to 25 engineers? Let’s detail five steps to split a Data Platform team:
- Analyze through the Team Topologies lens;
- Define capabilities;
- Group capabilities by affinity;
- Recommend a new structure and its trade offs;
- Implement and iterate.
#1 Analyze through the Team Topologies lens
Based on Team Topologies research there are three basic types of teams (and an extra type not relevant for this article):
- Stream Aligned Teams are the primary delivery team which has end-to-end accountability for delivery of a software product or service. Analytics Engineers focused on creating datasets as products fall into this group.
- Platform Teams provide the tools, utilities, and technical services that make it easier for Stream Aligned Teams to do their job, typically delivering “X-as-a-Service” capabilities. Data Engineers focused on maintaining the data infrastructure fall into this group.
- Enabling Teams act as “consultants” that help stream aligned teams to overcome obstacles, typically interacting collaboratively, providing guidance, training,… Whatever roles are working on defining and managing processes, data governance, golden paths,… fall into this group.
In this first step, you should analyze the activities the team is doing through the lens of Team Topologies, for example:
If aside from the data infrastructure, a Data Platform team also owns data assets as products (datasets, dashboards,…) and enablement or governance activities it means that the same group of people covers the work of three (!) different topologies. This results in difficult prioritization, increased cognitive load, lack of focus, individuals being a bottleneck, etc.
Some relevant follow-up questions to analyze the source of the problem might be:
- How big are the efforts on each topology?
- How many people are dedicated to each topology?
- Do we have people that are doing a little bit of everything?
- How many people would you need to successfully maintain both activities?
The outcomes of this step yields a basic idea of what cognitively diverse activities your team is now handling and how this affects their productivity, deliverables, motivation, and so forth.
If you want to explore more about team topologies applied to Data Engineering teams you can read this article by Pedro Gomes Mota: Team Topologies for Data Engineering Teams.
#2 Define capabilities
At this stage, it is obvious that the team needs some division of tasks and the ability for members to focus on a subset of these activities. To nail this divide, examine the capabilities that your Data Platform team currently supports. We understand a capability as:
- A service, product or dataset that allows users to do something.
- Denotes the “what”.
- Encapsulates the underlying functionality.
A “data platform” is already a capability per se (from the perspective of the company), but for the sake of this exercise you should go, at least, one level deeper. Ideally, you should specify capabilities as finely as necessary for the split. The outcome of this step is a list of capabilities your Data Platform team currently supports.
An example list of capabilities for a Data Platform team of 12 to 15 people would be as follows:
- Data storage infrastructure
- Core datasets as products
- Datasets as products developer experience
- Data models/schema management
- Data retention service
- Data deletion service
- Data access policies
- Data observability tooling
- Big data distributed processing
- Batch ingestion service
- Real-time ingestion service
- Notebook tooling for analysis
- Data visualization tooling
- Data infrastructure developer enablement
- Web analytics tool
- CI/CD for Machine Learning products
- Machine Learning model lifecycle management
- Feature store infrastructure and tooling
- Machine Learning platform enablement
- Real-time event tracking data as a product
- A/B test management and result analytics
- A/B test enablement
As an example of the level of detail: depending on the maturity of your platform it might make sense to just list a single capability named “Machine Learning Platform” for Machine Learning related topics. In some other organizations, you would like to separate “CI/CD for Machine Learning products” from “Machine Learning platform enablement” and assess future ownership of these capabilities separately.
#3 Group capabilities by affinity
The prior set of capabilities should be grouped by affinity in this step, first taking into account Team Topologies and then any potential limitations or restrictions your team and organization might have.
Restrictions may be related to:
- Limitations on capacity (the number of people who are available)
- The expertise and interest of those individuals
- The need for explicit optimizations that may result from a bigger data strategy
- In general, any form of trade-off
In the case of the example above there could be several ways of splitting the Data Platform team all with their pros and cons. Three alternatives are presented in the following figures:
Option 1 optimizes for having fewer teams (compared to Option 2 and 3) and separates enablement from platform and stream-aligned work. The “Data Infrastructure” team is quite heavily loaded though.
Option 2 keeps enablement in a separated team and groups 3rd party tools management under a “Data Solutions” team to split the load of the “Data Infrastructure” and “Business Intelligence” teams.
Option 3 clearly separates different specializations (ML vs. A/B testing for example) creating specific teams to focus on those domains. It also optimizes for keeping the enablement work as part of the domain, against team topologies recommendations.
#4 Recommend a new structure and its trade offs
The final step is to present the alternatives to Senior Management and recommend one to be implemented.
In our example case, we believe “Option 3” has the best chances to succeed long term and allow for further growth and scalability of the team. The pros and cons for this choice are:
- We explicitly separate work by domain (A/B testing vs. Machine Learning vs. Data Infrastructure vs. Datasets as Products) to allow individuals to focus and grow in a specific area.
- Even though it means more teams with less number of people each, the capacity we have can support this fragmentation.
- Because enablement work for each domain is tiny and particular, it makes no sense to bundle all enablement work under the same team. We optimize for keeping enablement work in the individual teams responsible for the technology because we believe it will feed back them better.
- We feel it makes sense for the “Business Intelligence” team to be responsible for supplying the core data as a product to third-party analytics tooling as a trade-off between “Data Infrastructure” and “Business Intelligence” teams.
#5 Implement and iterate
After deciding on the best organization, you must communicate and implement the change. Two considerations must be made during this step:
You should be on the lookout for issues of contention and feedback from the new organization. You may wish to revisit the operating model in 3 or 6 months and make minor changes, and be receptive to it.
Because individuals will no longer be participating in the same ceremonies or establishing objectives together, new teams may lose day-to-day interaction with old colleagues. Make sure that the overarching Data Platform team continues to meet on a regular basis for the activities that make sense, such as demos, weekly lunches, socials, and so on.
Final words
In this article we’ve summarized the steps to scale a Data Platform team from the learnings of having done this type of reorganization several times before. For other transitions your Data team can go through you can check the rest of the articles of the Scaling Your Data team series:
