Video produced by Steve Nathans-Kelly
Radiant Advisors' John O'Brien explained modern data architecture at Data Summit Connect Fall 2020, presented by DBTA and Big Data Quarterly.
There are a lot of components to a modern data architecture, O'Brien explained. "If you're looking at a hybrid cloud or multi-cloud architecture, you have new components in the data architecture side. For example, a data lake might be new for you. Data hubs could be the breakdown of your data warehouses and data marts. Data labs are those self-service analytic work benches, and even a data marketplace is another term for data catalogs, right? So right there, there are four pieces of a new modern data architecture. They all have their strengths. It's all built on polyglot persistence and other methodologies. You set up your architecture and, either way, you have components, you're going to have integration components in the cloud,from data pipelines to streaming data hubs, to data replication; orchestration is a big challenge once we move to the cloud."
According to O'Brien, if you have an on-prem BI team, when moving to the cloud, you have to consider what are the SaaS offerings, what are the PaaS offerings? How do you take advantage of elastic, scalability, hybrid multi-cloud situations, or even serverless functions? And, then, O'Brien said, throw those considerations on top of a mix of database technologies, data managements—because DataOps is continuing to gain traction—and we're really strengthening data governance and quality programs. In addition, you have to consider what it means to be in a data science platform—that is a strategy in itself.
From an architecture perspective, there are a variety of strategies at a high level that are going to be brought to your attention as you adopt and migrate to your cloud. "You want to move to the cloud quickly—that's accepted. My point is that there's a lot to do. We have to break this down. We can't run off for a year and say, we're going to go build this and we'll come back in a year. We have to continue to deliver value to the business via our projects and inject architecture as part of that," said O'Brien.
Architecture is always going to be evolving, he emphasized. "That's the new mindset. Speed—deliver something to the business. The agile way is the most critical point, but then iteratively, evolve that, improve that, refactor that."
When approaching a project there are certain givens that everyone must work with, said O'Brien. In a new progject, half ot he compoenents may be ones with which you have no experience, said O'Brien. "Nothing can replace that hands-on experience." In addition, we have to allow that, in the architecture, cost is going to be fixed. "You have so many teams, you have so many resources, you have only so much budget. You can only absorb so much technology in that way, but quality doesn't change. Anytime we deliver any small solution, it is going to be a quality production-worthy solution. It's just that we need to shift, maintain our speed but shift our scope."
Videos of full presentations from Data Summit Connect Fall 2020, a 3-day series of data management and analytics webinars presented by DBTA and Big Data Quarterly, are also now available for on-demand viewing on the DBTA YouTube channel.