Data, the potent fuel powering the AI-driven enterprise, is always on the move, as are the tools, technologies, and techniques being employed to manage and deliver it. With the intense emphasis on AI and machine learning these days, it’s urgent to ensure that data is available, timely, accurate, and relevant. That means no letup with efforts to ensure data pipelines are flowing unimpeded. With so much at stake, we canvassed industry leaders and experts on what challenges lie ahead and how to address them. Here are some leading issues:
THE CHALLENGE: DATA COMPLEXITY
Data complexity is growing and comes out of rising data volumes, data variety, and data velocity. “The amount of data being generated and stored is growing exponentially,” said Kunal Shah, senior product marketing manager at SAS. “This is due to the increasing number of connected devices and sensors, as well as the proliferation of online applications and services.”
The trend: Data complexity is exacerbated by data coming into AI applications and systems “from a wider range of sources than ever before, including structured and unstructured data, created and processed at ever-increasing speeds,” Shah said. “This is making it increasingly difficult for organizations to manage, store, analyze, and secure their data. Data management solutions can be complex and expensive; developing a data culture takes time; and adopting new technologies can be challenging.”
The solution: Companies need to invest in data management solutions “that can help to break down data silos, ensuring users can access, integrate, cleanse, and transform data, no matter where it resides,” said Shah. Plus, it’s important to “develop a data culture that prioritizes data quality, security, and governance by educating employees about the importance of data and training them on how to use data responsibly. Adopting new technologies, such as AI and machine learning, will help to manage and analyze complex datasets more effectively.”
THE CHALLENGE: UNSTRUCTURED DATA
The explosive growth of unstructured data has become a force to be reckoned with. “What has evolved is the burgeoning value of unstructured data, especially in the realm of generative AI that is able to make sense of diverse unstructured documents,” said Aron Brand, CTO of CTERA. “This exponential growth isn’t merely about the volume, but rather the imperative to turn this data into actionable intelligence. In a data-centric world, the winning enterprise is not necessarily the one flooded with data but the one that skillfully organizes, manages, and harnesses it.”
The trend: The situation with unstructured data “is intensifying,” said Brand. “As technology continually evolves and as enterprises become increasingly reliant on data-driven decisions, the magnitude of unstructured data grows concurrently.”
The solution: “As AI permeates every facet of business, from mundane operations to pivotal decisions, its hunger for data grows—especially for unstructured data like text, images, and videos,” said Brand. “Only recently have generative AI solutions been able to process this data efficiently. This highlights the urgency of adopting a data-centric mindset. Yet, AI’s vast potential also sheds light on the shortcomings of traditional storage techniques. The horizon calls for platforms adept at managing copious amounts of unstructured data produced and processed in varied locations.”
THE CHALLENGE: INFRASTRUCTURE FOR THE EDGE
Pay attention to the edge, because a great deal of data and processing will be happening there. “The current state of edge computing is growing exponentially as a reflection of the increased need to process, store, and analyze data captured at the edge,” said Bruce Kornfeld, chief marketing and product officer of StorMagic. Enterprises “should consider transitioning from a cloud-first to an edge-first strategy. The need to process and analyze data where it is created will improve customer experience, as well as efficiency and corporate profits.”
Technologies created for more centralized cloud “aren’t always applicable to solve edge data problems,” he added.
The trend: “The challenges for all organizations with small/edge locations are generally the same: How can we process and store all of the data being created here?” said Kornfeld. “How do we protect and secure it? And how can we use this data at all of these sites? Traditional methods of moving all data from edge to cloud are no longer practical for many environments, as cloud computing can be very expensive, unreliable, and may not meet application performance needs due to the latency of sending/receiving data to/from the cloud. The speed of light can often get in the way.”
“While there is some relief today with low-cost, small-form-factor hyperconverged systems being able to process and store data, the concept of protecting it all and making it easy to use for business improvement is not there yet,” said Kornfeld. “Consequently, it has the potential to get worse and is a huge area for upcoming innovation and improvement in the industry.”