AI caught everyone’s attention in 2023 but with more eyes on data and analytics than ever before, it’s become clear that it’s not the only solution on the market that can assist companies in gaining value from their data.
"Expect 2024 to be a year of transformation and change driven by adoption of AI and a reshaping of the data, analytics, and AI leadership role within leading companies," said Randy Bean, industry thought leader who currently serves as innovation fellow, data strategy, for global consultancy Wavestone. "With 33% of midsize to large organizations having appointed or in search of a chief AI officer, and with 83.2% of leading companies having a chief data and analytics officer in place today, it is inevitable that we will witness consolidation of roles, restructuring of responsibilities, elimination of some positions, and some critical rethinking of data and AI leadership expectations during the course of 2024."
Bean, along with Thomas H. Davenport, the president's distinguished professor of information technology and management at Babson College and a fellow of the MIT initiative on the digital economy, surveyed more than 500 executives for their report, "Five Key Trends in AI and Data Science for 2024."
According to the report, companies are investing in platforms, processes and methodologies, feature stores, machine learning operations (MLOps) systems, and other tools to increase productivity and deployment rates. Automation is helping to increase productivity and enable broader data science participation.
Additionally, respondents noted there are two versions of data products that will dominate. Eighty percent of data and technology leaders in one survey said that their organizations were using or considering the use of data products and product management.
But they mean two different things by "data products." Just under half (48%) of respondents said that they include analytics and AI capabilities in the concept of data products. Some 30% view analytics and AI as separate from data products and presumably reserve that term for reusable data assets alone. What matters is that an organization is consistent in how it defines and discusses data products.
Here, data professionals share their data and analytics predictions for 2024:
What has truly changed about cybersecurity in the last 20 years? Less than you’d think: Since the beginning of the cyber world, we’ve had cyber-criminals. If we look back at the earliest days of the internet at the turn of the century, the ILoveYou Virus infected 50 million systems and cost more than $15B to repair. Threat actors started with fairly simple spam emails and graduated to sophisticated email, voice, and SMS phishing campaigns – all with elements of social engineering at the heart. And even the generative AI wave is not as new as you’d think. Cybersecurity practitioners have wielded sophisticated AI tools long before other industries. Many foundational elements—like fear—are the same for every innovation cycle. So, what’s truly changed this past year? For one, threat actor motivations have shifted. There was a time when cyber-attacks were mostly motivated by clout, as a way of showing off hacking skill. Today’s threat actors are largely financially driven, looking to monetize activity. This has in part been driven by the prevalence and ease of access to anonymous, digital currencies. Another mindset shift we’ve seen in 2023 is treating organizations at the receiving end of attacks as victims, rather than putting the blame on the targeted company. This is the right way to look at it, as cyber-attacks become virtually inevitable for organizations. —Mike Nichols, VP of product management, Elastic
Standardization across new applications: I suspect a lot of organizations are running into a knowledge gap. For example, between older applications and employing people who do not understand how they work. That is why I’m excited about innovations like OpenTelemetry. It enables organizations to monitor and instrument their applications much easier and make them more repeatable. This type of standardization allows you to look at monitoring in a general way. It also adds value because anything that connects to these systems now has a general way to be able to speak. That is not to say OpenTelemetry is going to change old applications unless they move into the cloud, but at least standardization might trigger that conversation. If you want to lift and shift to the cloud, you are going to ask, “How do I monitor this?” and OpenTelemetry is a really good way to start. There is the added benefit that it’s future-proofing your investment because it gives you the flexibility to make changes as you need them. –Adam Luciano, director of AI and incubation, OpenText
Practice makes perfect: In 2024, we’re going to see development best practices, in terms of code, make their way into data. The concepts of data uptime and data downtime, which are related to data observability and part of data operations, will come in. Users have different ways to ensure the quality, and that if something goes wrong, you can pinpoint where it went wrong, saving you precious time. The increased complexity and costs are taking engineers away from the more important things and what they want to do, which is much higher value projects than shoveling data all day long. –Read Maloney, Dremio CMO
In 2024, BI will take a quantum leap driven by generative AI: 2024 will present a momentous change in the way companies are able to intuitively interact with their business data. With the powerful combination of gen AI and BI, you’ll be able to “Google your own data” and intelligently communicate with your enterprise data, unveiling valuable insights and patterns that would otherwise remain hidden. What was previously used to access traditional reports and dashboards, will transform into a highly evolved view of your business that is akin to having a conversation with a highly skilled colleague. Data silos will be shattered, and all people in an organization will be able to have access to business insights and make data driven decisions by simply talking to their data. Gone are the days of waiting for your data scientist or analyst to share reports because in 2024 and beyond, BI will be completely turned on its head.” - Danny Lange, VP of engineering, data analytics, Google Cloud
Organizations will prioritize getting to the root of database issues. One-third of IT professionals surveyed manage upwards of 300 databases. In 2024, teams will embrace AI and automation to ensure the health, stability, and scalability of their services. In addition to the ability to identify issues at their root cause, IT teams will benefit from real-time issue remediation, understanding database implications as new code is deployed, and even avoiding costly outages. –Jeff Stewart, vice president of global solutions engineering at SolarWinds
Optical networking will get closer to mainstream: The increased focus on network efficiency, reliability, sustainability and being future-ready will bring optical networking to the fore in 2024. Recent trials prove the potential – achieving 1.2Tbps transmission rates. Across all industries, over 90% of senior executives are looking to modernize their networks, ensuring they are set up for the existing challenges they face, as well as those yet to come. There is also a wider concerted effort to overcome the limitations of existing infrastructure with optical technologies. More than 100 organizations are now partnering to drive the IOWN (Innovative Optical and Wireless Network) forward, ensuring the world is ready for future technologies and bringing optical networking closer to mainstream. —Shahid Ahmed, group EVP, new ventures and innovation, NTT Ltd
Database on Kubernetes takes center stage as the #1 workload: As more organizations recognize the inherent difficulty of managing and operating databases to support their digital businesses, they have increasingly turned to Platform as a Service (PaaS) or Database as a Service (DBaaS) solutions. And at the heart of both? Kubernetes (K8s). The growth of K8 has already been the talk of the town, with industry leaders dubbing it the emerging cloud operating system. However powerful, generic K8s structures must be taught how to manage specific database platforms and workloads. In 2024, databases will surpass other K8s workloads as providers better enable K8s to deeply understand database internals. This will offer uniquely attractive solutions for lowering operational costs, and the ability to scale databases reliably - even autonomously. —Pieter Humphrey, Director of Developer Advocacy at MariaDB plc
In 2024, we’ll see Kubernetes become virtually invisible: Kubernetes is reaching a mature state where companies can now deploy Kubernetes at scale using as-a-service models with little to no expertise. In the last five years, we’ve seen a lot of innovation and stability from Kubernetes. For example, take the rise of platform engineering and how the maturation of DevOps has been formalized within the organization and consolidated responsibility to ensure Kubernetes is deployed centrally across the business. When technology becomes this ubiquitous, it’s almost like it’s invisible. –Murli Thirumale, Portworx co-founder and general manager, cloud native business unit, Pure Storage
Modern data integration hubs: Data integration hubs and DIH technology providers will try to move up and cover a bigger chunk of the data value chain by providing other capabilities like analytical/AI/ML processing and persistence. They will gravitate towards becoming unified data processing platforms. –Lalit Ahuja, chief product and customer officer at GridGain
2024: The year of the data product: Data mesh represents a transformative shift in modern data architecture, moving from centralized models to decentralized approaches. This paradigm shift empowers domain-specific teams with the autonomy to manage and govern their data independently, enhancing agility and understanding. Alongside this shift is the rise of 'data products,' akin to physical products, which redefine the focus of data and analytics. Instead of being project-based or one-time deliverables, these data products embody a continuous lifecycle, paralleling tangible goods. This will be a rising trend in 2024, with more organizations adopting data products to deliver value and drive significant business outcomes. –Diby Malakar, VP of product management at Alation