There is an unresolved distinction between data architecture and agile development that, based on the history of modeling vs. the history of agile, has not been fully resolved in the real world, according to Henry Olson, director of product management, Embarcadero Technologies.
Olson tackled the cultural, technical and methodology gaps between the worlds of data architecture and agile development the challenge of retaining the benefit of data architecture and data modeling in the agile world in a lively presentation at DBTA’s Data Summit in NYC that drew numerous follow-on questions from attendees in the room.
“There was a time when agile was something for the new kids, web apps, Silicon Valley," said Olson. "But, I think today increasingly we are seeing agile as a key discipline in large enterprises and that more than anything may account for the rise in prominence of this particular issue to the point where our customers are reaching out to us and asking for help.”
While developers have long railed against the mountains of documentation and process that went along with large-scale enterprise development and there is “a sense of being let out of the cage” with a more unfettered approach, the downside is that at times people have gotten so excited about that agile is not an abandonment of discipline, but a just a different kind of discipline, said Olson.
“Although I was trained on it, I am no longer a proponent of the waterfall lifecycle approach,” said Olson. However, he noted, “I do think that this key element of data architecture and data modeling, which originated with those waterfall methodologies, is still very, very important and critical in the enterprise in terms of actually getting value out of your data assets. If you are not able to model them, you can’t actually understand them.”
For example, Olson said, “If you are pulling information out of the data warehouse and you don’t realize that the global HR database has salaries in Euros, you can make a big mistake when you are doing your planning and budgeting. This is not an academic thing.” And then, there is the famous story of NASA’s Mars Climate Orbiter, Olson added. “They spent half a billion dollars on it and sent it to Mars and when they pressed the button to do the orbital injection over Mars to start collecting all that great climate data the thing just disappeared. What they came to understand was that the flight control system was in metric units and the onboard system was in SI and so there was a units distinction, and the injection fire on the rockets was the wrong amount. That was a metadata problem.”
Metadata is necessary and it is critical for discovery, in order to find the information that is in the organization. And, then once the information is found, it is critical to properly interpret the information, Olson said. “The idea that somehow the metadata is in the code is great for folks that are code-literate but data consumers can’t use that, so there has to be some externalization of the metadata or the information is not useful in a broader context. It is just that simple."
To help address the challenges and conflicts that may arise within an orgnaization due to differing approaches, said Olson, Embarcadero has an enterprise modeling tool ER/Studio which is used to construct visualizations as part of the process and also to create metadata definitions for the data elements. “And, with the ER/Studio Team Server products, we are now syndicating that metadata so it is available much more broadly inside of organizations - both inside of tools and as an annotation to existing web systems so that you can understand the definitions and data while you are working with them.”