Introduction
There seems to be a belief among information technology (IT) teams that a data warehouse team can’t be Agile. However, you really can build data warehouses using Agile methodologies. Stop and think about what it means to be Agile. Agile is nothing more than a way of working. There’s nothing special to Agile other than people working collaboratively to solve business problems. That’s it. That’s Agile in a nutshell. When someone says a data warehouse team can’t be Agile, what they are saying is, “A data warehouse team can’t collaborate to solve business problems.” In this article I will lay out where this belief comes from and what organizations must do to eliminate this belief from data warehouse teams.
Before I dive into the larger picture I want to define what a data warehouse team does and the value such a team brings to an organization. A data warehouse team is a group of engineers who build large data repositories typically referred to as a data warehouse. A data warehouse is a specific kind of repository that contains cleaned and structured data, and a data warehouse is often built to answer various business questions. Any data engineer will likely argue that this definition is an oversimplification, but given that this article is not about data warehouses, but about working in an Agile way, this simplified definition works for the context of this article.
The problem with data warehouses is also one of their biggest strengths: it’s structure. Given that data warehouses are intended to help users answer business questions, data engineers want to know those questions up front. The questions users have informs how the data warehouse gets built. Data engineers then construct the data warehouse to store and quickly retrieve relevant data to answer those business questions. However, if users have new questions they want answers to, questions that were never considered during the design of the data warehouse, then the data warehouse, or parts of it, must be redesigned. With the amount of data contained in the warehouse, and the way data is referenced and organized within the warehouse, changing even a part of the data warehouse is labor intensive (i.e. it takes a while). Even getting new data into a traditional data warehouse can take a long time.
What’s a long time? Often months. So it takes a long time to build a data warehouse. You know what can also take a long time to build? Software applications. And yet, we build software applications using Agile methodologies.
The sources of this belief.
The first source of the belief is management. If you’re a manager, then one of the fundamental expectations of you is to improve your part of the business and grow its potential. If you’re a manager of a data warehouse group, then part of your job is to ensure your organization has a scalable and flexible data solution to meet the organization’s demand of data. If you’ve sat an your laurels and did nothing to ensure your data environment is scalable and flexible, then you didn’t do your job. Now, not every manager falls into this category. Some managers have tried, but they are constrained by budgets—I’m not talking to these managers. There are plenty of managers who were given the opportunity to adapt their data infrastructure, but chose to do nothing.
Employees, the data engineers and architects themselves, are the second source of this belief. Far too many data engineers are blinded by what they’ve always done. They were taught (many back in the early 1990s) that there is only one way to build a data warehouse, and so they always build data warehouses that one way. These types of employees will fight anyone who challenges their thinking. These people are data priests who worship at the altar of data structure. They even have a mantra: “The key, the whole key, and nothing but the key, so help me Codd.”
The third source of this belief comes from the tools a team has at its disposal. Most organizations have ancient data warehouses and ancient tools to maintain them. There are code-less platforms that allow data warehouses to be restructured in a matter of minutes. However, when you combine the cost of such platforms with managers who are unwilling to invest in a flexible data environment, then you have big problems—especially if you want to leverage data to create a competitive advantage.
So how do you manage this belief and start turning things around?
Change your approach.
Stop thinking that a data warehouse is only useable when it’s built. For far too long people have thought that a data warehouse is like a house, you can’t live in a house until after it’s built. This applies to people, but it doesn’t have to apply to data. You should view a data warehouse as always being built. A warehouse is never finished and its always changing.
Focus on business value not the totality of the data warehouse. When adding a new data source to the warehouse there is often a reason why the business wants to do so. That reason is because they have specific questions they need the data to answer. It’s not uncommon for data sources (i.e. software platform) to have thousands of tables with each table having dozens of columns. Your users don’t need ALL of the data to answer their questions; most users only need a fraction of the total data. With a little digging, you’ll probably find that most users need less than 100 data elements to help them answer most of their questions. So just give them the data they need and stop trying to pull in ALL of the data from the new platform into the warehouse.
Train your analysts to be effective analysts. This topic warrants an article post all its own. Long story short, most organizations simply do not have analysts who are trained to think and act like analysts. So most analysts struggle in their job, and many don’t know what they don’t know. They have no clue how to effectively analyze a system and define how data moves through the environment. If your analysts are not trained to do their job, then they will create ambiguous user stories for your teams. Your developers will then waste time doing analysis work that should’ve been done by the analysts. And since it is difficult to estimate effort for analysis work (think spike), your ambiguous development stories take months to complete.
Train your managers how to be effective leaders. Most managers receive little to no training on how to be effective leaders in Agile organizations, let alone what is expected of them as a manager. If your managers don’t care about being agile, and if your managers don’t care about creating competitive advantages with data, then your managers will slowly kill your organization from within.
Leverage data lakes and operational data stores (ODS). If you need to dump all of your data into a single space, then use a data lake. You’ll be able to quickly land your data somewhere. Then build small operational data stores to answer those specific questions your business users keep bugging you about. Operational data stores are much smaller than data warehouses and serve very specific business needs. This may not be the best approach for your old data priests, but do you want to do what’s best for your priests or for your organization? Building operational data stores doesn’t have to mean building sloppy data repositories. You can build an ODS simply and with minimal overhead. This doesn’t mean you should do away with your data warehouse altogether.
Upgrade your toolsets. Invest in code-less tools that allow your engineers to rapidly build and modify your data repositories. If you believe your data is valuable (some would say priceless), then the expense will be well worth it. The agility such tools can bring to your teams is immense. And if money is locked away in data insights, then your data will pay for the toolsets in short order.
The question you have to ask yourself is: Are you in business to service customers, or are you in business to build complicated data warehouses nobody uses?