BDQ: It is frequently said that data scientists and analysts spend about 80% of their time cleaning and prepping data and only 20% actually analyzing it. Does DataOps address that problem?
CB: Looking at the 80% problem, the issue is that they are doing things other than what they really want to do. Whether you call them data scientists or data analysts, they want to create and invent, and sit between the data and the customer to figure out if it really helps with what their VP of marketing or their VP of ecommerce needs to know. And that's a really interesting, creative job, but they're spending a lot of time not doing things that they think are part of that role, and a piece of that is preparing the data. They're also just going to too many meetings and they're dealing with things going wrong. And then they're talking to people and having more meetings. When there's an issue with the data or the charts that they've produced, or with the model, it takes days, and sometimes the smartest people, to figure out where it is.
BDQ: What are the other challenges?
CB: The other part is just when they want to do something new. In a lot of organizations, it takes months for them to move from the time that they have an idea to actually getting it in their customers' hands because of all the meetings and coordination. Data and analytics is not individual acts of heroism; it's a team sport.
BDQ: You are saying the tools are not the problem.
CB: Most people have the tools that they need to do their data work. In their tool chest, they have a tool that does data transformation or ETL, they have a database, a tool to do charts and graphs. They need those and they need the data that they can act upon. And of course they have a customer who they're trying to understand but, in addition to that, they need to start thinking about the tools to help them with the process. And that's automated testing, monitoring, and deployment, and managing the technical environments they actually do development and production in. Those are the sort of facilities that the DataOps market is providing.
BDQ: How is this progressing?
CB: There have been a bunch of companies that in the last year have entered the market and gotten funding to do it around testing, deployment, or models. And the core capabilities of DataOps, as a data platform are becoming standard and people are building companies around that idea.
BDQ: Are there specific companies or market sectors where DataOps is being deployed most heavily?
CB: Yes, there are companies that are doing it in Silicon Valley that just have a lot of data and they've always worked in a very iterative customer focused way. And then second there are companies now that want to be like that and feel threatened by the Amazons of the world, and they want to be more data-driven. Those are typically companies in financial services, healthcare, and manufacturing that have a lot of data that they can't get value from, so they've hired a chief data officer and they're trying to be data-driven. Typically, they are people in marketing and sales functions because they have a lot of need, or people who have hired a group of data scientists and data engineers and have people doing self-service analysis. Five years ago, there were barely any of them. And now there are chief data officers in every organization, and that is a good indication that people are making the investment to not see data as a piece of furniture, but actually as a source of competitive advantage.
BDQ: Are there any roadblocks to DataOps that you see right now that are preventing people from putting their data to work?
CB: For people who are experienced in data and analytics, we're saying to them: You can make changes to your systems fast, and you can do that with very minimal problems. And that is, for some people, almost heresy because they just got [their system] working and they don't want to change it. They are thinking; it's running, don't tell me I can change it every week, or change it every day.
People have spent a lot of time building very brittle systems that require a lot of care and feeding and they've done the best they could, but this change in focus to how you do it—as opposed to what you do—is my motivation. I suffered for years not working this way. And it is a challenge for people to see that this different mindset can actually help them and is not a threat to them; it's actually a way to leave alleviate their suffering.
BDQ: Why is now the time to jump in to DataOps?
CB: In my experience running analytics teams, I just got phone calls when things went wrong or were going too slow. I'm seeing that same pattern happen again and people are going through the same experience I had. The time is now because organizations have invested in data and analytics systems, and they want more change and more insight, but that insight just doesn't come from hiring a bunch of data scientists; it comes from building a team and a process that continuously creates value. I think most people want to build Toyota Corollas and not AMC Pacers.
This interview was conducted, edited, and condensed by Joyce Wells.