In addition, business teams are slow to share their data. “They rely on data engineering teams of highly skilled engineers, who are hampered by inadequate tooling and lack of visibility into data and the complex underlying technology,” said Stevenson. “They are asked to not only master complex distributed technologies but also to become an expert in every business domain. Data decays over time, if you can’t discover data, process it, and quickly share your results, you are at a disadvantage.”
The sheer complexity also hampers analytics efforts, especially with “many- sources operational applications from various competitors, each with their own data models and data validation rules” within enterprises, said Raghu Chakravarthi, senior vice president of R&D at Actian. For example, he related, one large auto manufacturer he worked with “had 60-plus application data sources they pulled from before performing analytics to answer a single business question. A simple operation such as ‘identify customer’ across operational data stores became complex when you only have a first and last name to correlate data.”
In addition, variations between applications for customer hierarchical detail—some may only have one level, while others have up to 10—result in an inordinate amount of time spent cleansing, correlating, and deduping data, Chakravarthi continued. “Typically, these still result in irrelevant data, so many enterprises wrote specific business rules and cleansing logic in ETL/ELT. These rigid practices cause the 80%.”
Prepping the Organization
Reversing, or at least easing, the 80/20 rule requires a shift in organizational priorities—and even organizational structure. Simply opening up communication channels is a great way to start. “Teams should build processes where analysts and business stakeholders meet to discuss new questions and interpretations on a weekly basis to take advantage of new datasets and business opportunities as they arise,” said Bailis.
This requires a common vision across the enterprise. “Executive mandates are not sufficient,” said Horia Tipi, head of global optimization at FICO. “Stakeholders such as product, distribution, finance, and marketing teams need to see and understand the benefits of an integrated data analytics solution in order to really engage with the process.” At the same time, he cautioned, this doesn’t mean turning on a firehose on such diverse teams. “Marketing teams are not equipped to understand finance; while product teams often have unrealistic expectations of distribution. The solution is to have a single source of truth—a holistic data picture—that is complemented by a projection of that truth onto the screens of each individual stakeholder.”
Still, in efforts to assure rapid, unimpeded flows of data-driven insights, some perspective is needed. Frequently, existing ETL processes may be enough for an ongoing data transformation flow. Organizations seek perfection too often in their drive to have the cleanest data possible, down to the transaction level, even if the data is only being used for strategic purposes such as trend analysis, observed Glen Rabie, CEO of Yellowfin. But the effort to achieve perfection may far outweigh the benefits, he noted. “Organizations need to be more efficient and prepare their data to the level that supports the detail of analysis they need to do.”
Instead of devoting too many resources to data preparation, organizations should focus on how work is allocated among data teams, Rabie continued. “Sometimes, analysts are not actually data ‘analysts’; rather, they are data preparers. As a result, they feel more comfortable working with the data than analyzing it and conducting the business analysis that the organization needs. Ensure that the right data specialists are assigned to the roles in the analytic process for which they have both the skills and inclination.” Rabie also urged enterprises to provide their data teams with the right resources. “Given the initial effort to prepare data, if the organization does not provide sufficient analysts, prepared data may not get the analysis it deserves.” Instead, he said, organizational priorities may push the analytics team to the next dataset. “Companies should appoint more business analysts to analyze the data that has been prepared.”
One View of the Data
Bringing teams together across the organization requires their involvement in a truly holistic data strategy—“understanding what data they have, understanding that all data is not equal, and then linking their data initiatives to their corporate initiatives,” said Andreas Wesselmann, senior vice president of HANA and analytics, data management and platform, for SAP. Most companies have data in many systems, in ERP and other hybrid applications, and flowing in from IoT, social, and external sources, he explained. “The data is now multi-faceted and in many cases not fully connected. Businesses must have a solid data management strategy, with data integration and orchestration at the top of the IT priorities list.”
The name of the game is also focus, applying data exactly where it is needed across the business. For example, in operations, “much of the data is first and foremost used in process control and real-time operator insight and forensic analysis,” said Richard Beeson, CTO of OSIsoft. “On the maintenance side, the data serves to schedule people, equipment, and services in the most efficient manner possible, taking into account production schedules and customer commitments which are typically in another core data source.”