AI AND DATA FLOWS
Successful data management delivers AI, while AI tools and platforms can deliver successful data management. AI-based models can be used to “identify exceptions in data, classify unstructured data fields, and identify patterns in errors and omissions to help improve successive generations of a dataset,” said Sonnenblick. “In the next few quarters, we’ll begin to see generative AI approaches that automate API generation and usage, dramatically simplifying the process of connecting disparate systems and merging data streams.”
AI can also assist with “data semantification and governance,” said Sellers. “Both activities heavily depend on metadata, which is conventionally created and maintained by data producers in a manually intensive process that is often skipped, as creators don’t always see it as valuable. Generative AI has already demonstrated that it can help augment data engineers by proposing metadata that describes data properties and schemas.”
Generative AI “can further associate tags and annotate data with provenance characterizations,” Sellers continued. “Human developers can quickly verify machine-generated suggestions far more efficiently than creating this content themselves, freeing them from significant rote work. Data becomes more discoverable, contextualized, trustworthy, and reusable.”
For the database itself, “AI and ML work in real time to identify and address headaches before they happen,” Lanehart illustrated. “Take anomalies or data drift, for example. If undetected or detected too late, they require hours of tedious investigation to identify the root of the problem and then to retrain the model. Instead, when having AI monitor data systems in real time, these issues can be caught early on or prevented entirely.”
AI introduces a range of unpredictable issues, “and we will need AI-fluent humans to train and code AI to improve flows and infrastructure for AI to better ingest,” said De Cremer. “Humans are out of their depth already, and the only big ROI we are providing to AI is when we are integrated with AI.”
NEW TECHNOLOGY, NEW ROLES
For the rise of AI to be supported by data management, as well as supporting data management, means a rethinking of data-associated roles. This suggests new opportunities for those involved in data management and associated development.
Data administrators are “transitioning to roles centered on the ethical operation of AI systems, focusing on privacy and compliance standards,” said Lango. “Data managers are evolving into strategic roles, using AI to derive insights that align with business objectives and drive innovation.” Looking at data engineers, these roles will increasingly be “tasked with creating AI-supportive infrastructures, prioritizing flexibility, scalability, and ethical considerations. Empathy, emotional intelligence, and adaptability are coming for all these roles as they work to integrate AI in a way that enhances human experiences within business processes.”
AI also can help alleviate the more burdensome tasks associated with data management, “such as pattern identification and code generation for data scientists, developers, and analytics practitioners,” said Soceanu. Their role will be to “adopt comprehensive data governance and data management practices to ensure data is accurately sourced, validated, and fit for analytical purposes.”
AI also opens the way for developers to become more intimately involved in data management processes. “Going forward, harnessing the power of AI and generative AI approaches to data processing and analysis will be an essential developer skill,” said Sonnenblick. “Developers will move beyond static user interfaces and need to employ effective prompt engineering to productively use these tools for data management, aggregation, and insight generation.”
The new mission of data and AI managers will be “to remove these silos and install a collaborative work climate between different groups,” said De Cremer. “This requires that business leaders need to be AI-savvy enough to use a narrative that all these groups can understand to create a common language that shows that the use of data and AI should result in business value across the board and [that] all these groups have a role to play in this process.”
Companies also need to prepare organizationally to be receptive to the innovations data-driven AI promises. AI copilots—or virtual assistants—will increasingly guide the technical development side, said Sellers. “The best data administrators, managers, and engineers will be those who most effectively utilize these copilots to enhance their work. Every technologist must learn how to craft prompts, validate generated content, rethink quality controls, and understand the limitations of generated content. Productivity will increase, but human creativity and critical thinking will remain as key differentiators even while new AI technologies democratize the translation of ideas to mostly serviceable computer code.