BDQ: There is also a strong emphasis on blending what were previously siloed processes. How are methodologies such as DataOps being used now?
O’Brien: A significant aspect of DataOps around continuous integration and continuous deployment is that the team not only has to do the engineering work and the testing, but now they are responsible for embedding governance. We have actually leaned more on DataOps to embed data governance inside of every data pipeline with monitoring and alerting notifications.
We created an organizational strategy of enabling teams with the goal of making these agile delivery teams better at deploying their own code. If, organizationally, you have this team in an enablement, supportive role, they will make sure that those agile delivery teams have everything they need to do a good job, which is embed governance—embed data quality, auditing, and proper security. The challenge that we have heard from some clients in the field further down this journey is that the data engineer who likes to write and integrate code now becomes a full-stack engineer which means that person needs to understand security and all these other parts.
BDQ: Has it been effective?
O’Brien: Some teams have struggled a little bit because it is a shift from a data engineer to becoming a full-stack engineer. In some cases, these teams are deploying things into containers and deploying Kubernetes into the fabric, and they are saying that it is just too much technology and too many layers. In the past, there was a security layer, a data layer, and a governance layer, and you could be in just one space. Now that we're slicing vertically, data engineers have to sign up to do all of that.
BDQ: How quickly is DataOps being embraced?
O’Brien: I do believe the key is having what we call IT enablement organizations that are focused on how agile delivery teams are doing. These teams need best practices, good processes, and a methodology more than they need the technologies. The old way of thinking is, "I'll buy a technology; it'll solve my problem." It doesn't work that way. The number-one challenge in these companies is: "How do I change culture? How do I change to democratize the data workforce and enable self-service?” That is what they need to focus on and that's where the challenge lies.
BDQ: Are there any other approaches on the horizon that you think are going to be helpful?
O’Brien: What I think will be interesting in the future is related more to machine learning. We have been talking about operationalizing analytics for over 5 years. It has finally come together to become MLOps in order to scale. It is an inverted paradigm shift from BI. In BI, we build a dashboard and we're done. In ML, you put it into production and your work is beginning because you have to start monitoring all the time.
MLOps will be one of the big trends that I think will continue to evolve. One of my favorite new kinds of technologies in the space is the automated feature engineering. We get a lot of questions about whether all of the data from 2020 is really garbage for a training dataset because it was so unrealistic. Training datasets are a big challenge. There is new technology that allows you to take AI and give it a 1,000-column dataset and it will churn through that until it gets down to the dozen or so that are the most relevant for you. And the AutoML piece allows you to look at the high predictors. I think the most significant advancements in the ML space are going to be MLOps, AutoML, and automated feature engineering that really enable ML to become more scalable and a little bit less of a bottleneck. I think the self-service world is actually going to help with that because the self-service people, the businesspeople, can build training datasets faster.
BDQ: Is there anything else?
O’Brien: The other major trend for 2021 is going to be robotic process automation. We expect RPA to come in and make a big hit on the self-service data analytics world. Those are probably the two main categories that will dominate over the next year or two.
BDQ: What’s next?
O’Brien: We are at a turning point right now as companies are going from surviving to thriving. That's how we view it. There is a lot of pent-up demand to figure out how to do it right, which is nice. In years past, everybody wanted to just go buy something.
Interview conducted, edited, and condensed by Joyce Wells.