As many seasoned data professionals know, moving data from sources, or between systems, can be fraught with challenges. Businesses expect data to be ready and available, and are often oblivious to the machinations needed behind the scenes to make data available on demand. This process, referred to as “data onboarding,” involves migrating customer or other third-party data into a new software system, and is an increasingly prevalent and persistent pain point for companies. For today’s real-time, information-on-demand environments, it’s a process that is too slow and complicated.
That’s the key takeaway from a survey of 5,000 managers and executives—many from outside IT departments—conducted by Flatfile, which found that more than three-quarters of respondents either “sometimes or often” run into problems onboarding data. Overall, 90% of respondents must transfer data from one system to another at some point.
Data onboarding is a crucial but painstaking process that holds back individual companies and entire industries from reaching their full potential. The last thing a company wants is to lose a client over frustration due to data migration.
The survey revealed that data onboarding is increasingly routine, with 50% of respondents citing data onboarding as a daily activity. Another 28% said onboarding is done weekly and 22% reported that they have to onboard data multiple times per day.
Problems are experienced across a diverse range of industries for a variety of data types. About half of respondents are dealing with customer relationship management data, 26% work with financial data, and 23% with ecommerce data,
Types of Data Being Imported
CRM: 52%
MarketingC32%
Financial: 26%
Ecommerce: 23%
HR/Employee: 15%
Healthcare: 13%
Logistics: 13%
Manufacturing: 7%
Real Estate: 4%
Other: 11%
The challenges with data importing are multifaceted, with more than three in four reporting data formatting issues, and another seven in 10 having problems with data validation when onboarding their data. Other significant problems include accessing data, security concerns around sensitive data, and data volume. These findings make sense, “given the potential messy data issues that can arise and the need for clean, accurate data to have a successful import,” the survey’s authors observed. “Most companies struggle with data wrangling and spend far too much time focused on cleaning data—turning IT engineers into data janitors.”
Specific Data Import Issues Being Experienced
Data Formatting:76%
Data Validation: 69%
Column-Matching:46%
Data Access: 28%
Data Sensitivity: 28%
Data Volume: 27%
Project/Workload: 24%
Version Control: 23%
Other: 1%
As a result of these issues, employee talent is often sidelined to troubleshoot problems. Nearly half of respondents said it can take weeks or months to get new data successfully onboarded, and only 4% of respondents said their data was imported without problems.
The survey also explored potential solutions to improve data import experiences. The need for integration between systems was cited by 36% of respondents, followed by greater ease of use (29%), and more automation (26%). Another frequent comment concerned the need to streamline the processes, the survey’s authors said. Self-service—the ability for users to properly import their data without extensive intervention—was another frequently mentioned response when asked how to change the current experience. “The respondents’ comments reinforce the fact that when the responsibility for data import is left to the customer, chances are it won’t go smoothly—especially using the most common current processes,” the authors indicated. “Customers today are forced to read through extensive tutorials or watch videos to hopefully figure out the process.”
Survey respondents provided insights on the workarounds they developed to manage the process. “We input the data model (manually or programmatically) and the customer is then able to import their data, map to the data model, and address any validation issues themselves,” said one respondent. Another said they would like “an integrated dashboard in our application for customers to submit data and review it. Also, a way to preview the data in a detailed manner. For example, you’re importing contact data and the final output would be a combination of companies with contacts—showing a preview.”