APIs, Borth continued, “have emerged as having the most positive impact on an enterprise’s ability to compete on data—providing an accessibility layer to enable access to data at both the
source and though API hubs. APIs bring order to chaos and help prepare organizations with a semantically structured landscape to be faster and more cost-effective.”
Many enterprises that are building distributed applications using APIs are doing so with service mesh approaches, which Borth defines as “a dedicated infrastructure layer, to provide service-to-service communication, load-balancing for unexpected outages, and security from a central location.”
The need to connect diverse environments is taking on urgency as organizations recognize the limitations of relying on a single approach. “Many companies believed Hadoop and a data lake would solve all their problems and unlock untold golden nuggets hidden in their data,” said Andrew Stevenson, co-founder and CTO at Lenses.io. “However, projects failed to make it to production and many organizations have been left with a data swamp with no context around their data, no discoverability, and the ability to access this data limited to an elite few software engineers.”
A better approach, Stevenson continued, “is to adopt best-of-breed technologies that allow for a data mesh architecture to be adopted.” A data mesh would consist of “a streaming data layer to decouple producers and consumers and facilitate real-time analytics such as Apache Kafka, a scalable compute layer to orchestrate data application landscape such as Kubernetes, and the correct data store, such as Apache Druid for real-time analytics—or Elastic or Azure Synapse.” Data mesh platforms need the accessibility and flexibility that commoditized technologies such as Apache Kafka and Kubernetes provide, Stevenson emphasized.
Janet Liao, product marketing manager for Talend, sees this highly connected architecture as a “data fabric,” or platform that is “capable of orchestrating disparate data sources intelligently and securely in a self-service and automated manner.” This fabric, she added, is “becoming smarter and more autonomous.” The fabric can become dynamic for different organizational contexts—such as business, technical, or social. Plus, she continued, within a data fabric, “the use of AI and machine learning algorithms can create a feedback loop that enables the fabric to self-tune its performance.”
Liao noted that a data fabric is also a complex emerging technology that continues to evolve. It is not a suite of targeted tools that are customized but rather a unified platform of capabilities that can be applied consistently to the data pipelines which the organization creates, she explained.
EDGE COMPUTING
Edge computing and the Internet of Things (IoT) are also reshaping data enterprises. Until recently, data has been siloed, or—as Atish Gude, chief strategy officer at NetApp, put it—“trapped on islands.” Edge computing is opening up this data to new possibilities, he said. Cloud-based services “can now be delivered where the data is, rather than migrating the data to where services are available,” Gude said. “This has led to the development of cloud computing services stitched together with edge infrastructure to address growing edge computing demands.”
Not only are IoT edge devices transforming how businesses use data, but they are also leveling the playing field between large enterprises and small companies. “The increasing compute power and storage capacity in edge devices enable companies to connect with their customers, manage factories remotely with digital twinning, and improve employee safety,” said Stephen Manley, chief technologist of Druva.