The year just ending has been an interesting one for data managers. Artificial intelligence (AI) and machine learning took center stage, which also meant an increasingly glaring spotlight on data sourcing, management, and viability. The continued rise of the Internet of Things (IoT) also meant no letting up on demands for data environments to deliver requirements fast and furiously. The year ahead will bring more of the same—as well as a continuation of the transformation of information management.
Here are some of the changes and challenges on the horizon for 2019, as seen by leading industry participants and observers.
MASSIVELY DISTRIBUTED DATA GETS EVEN MORE DISTRIBUTED
Things keep moving away from the center. As Ken Tsai, global VP and head of cloud platform and data management for SAP, put it, customers’ data now sits in an average of six to eight clouds, as well as their own data centers. “This is contributing to the rapid evolution of data processing technologies to become massively distributed,” he said. He added that “data integration technologies are shifting from extract-transform-load to a process- and pipeline-driven approach, with data management and governance capabilities to support both centralized and federated models.”
At the same time, Tsai cautioned, this high level of distribution will “increase cost and complexity in managing mixed data landscapes and locations and it is difficult to know where the data resides, what data is available, as well as how to govern and trust the data source and accuracy, and monitor its usage and lineage across this distributed environment.”
This will change the way information is managed, Tsai added. “Information won’t be tethered to just one system and will flow freely and be connected—no matter how the business needs evolve or where the data consumer is located, or the device they want to use for access.” Enterprises will approach information orchestration from raw feeds with intelligence and real-time analysis on vast quantities and varieties of data types, he said. This will “make it much easier to enrich the mission-critical information with the new sources. Applications and data platform technologies are, and will continuously get infused with, intelligent technologies so business scenarios can be automated by AI and heuristic usage data, creating a new class of solutions.”
DATA GETS EVEN MORE STRATEGIC
In the year ahead, data managers will continue seeing their mandates extend well beyond their original course of action for managing and securing day-to-day transactions. It’s now about “leveraging information to make strategic, operational, and tactical decisions that result in increased revenues, improve operational efficiencies, and enhance customer experience,” said Satya Sachdeva, VP of insights and data for Sogeti USA. Systems are evolving with this growing mandate. “Database technologies used for information management have been rapidly evolving from traditional relational database management systems and OLAP technologies to MPP-based appliances,” he said. “Data lakes and Hadoop-based data environments for ingestion, wrangling, and analytics are gaining footholds across many organizations.”
DATA GOVERNANCE GAINS
Data governance will continue to become even more critical in the year ahead. The EU’s General Data Protection Regulation (GDPR) was a major driver for data governance programs in 2018, observed Emily Washington, senior vice president of product management at Infogix. “Suddenly, every organization conducting business in the European Union needed a well-functioning data governance program.”
However, Washington continued, “this new focus on data governance didn’t automatically translate to a better understanding of enterprise data and new analytical insights. Although many businesses are now collaborating across departments, there is still a disconnect between the analytics team and those that are focused on governance efforts, resulting in various metadata definitions across teams.”
Introducing governance to analytical models will help businesses “aggregate the metadata around their models to ensure all teams have a complete understanding of their data and can leverage it in analytical models,” she stated. “More businesses are now successfully cultivating open communication between various departments because of their data governance programs. This suggests that many companies are ready to expand their data governance program to a more strategic focus beyond just governing data.”
MORE CLOUDS ON THE HORIZON
For some time, the idea of relying on public cloud services to handle critical or sensitive data assets was a non-starter. Attitudes have changed, and in the year ahead, public cloud may be the go-to solution at many enterprise data sites. “Public cloud solutions have made resource consumption cheaper, simpler, and dynamic,” said Gaurav Yadav, founding engineer and product manager for Hedvig. There’s another advantage beyond this, as public cloud services offer a degree of standardization that is needed across global enterprises or partner networks. “Businesses are struggling to keep up with this extreme pace of data generation and traditional data analytics tools are not capable of handling such globally distributed data.”