Today, the success of many startups hinges upon the ability to gain insights from rapidly growing data. Yet startups and smaller businesses often don’t have the resources to hire a full-scale data science team, especially considering the painful data scientist shortage that’s making it difficult for even large enterprises to find qualified candidates.
McKinsey & Company has projected that by 2018, the U.S. alone “could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.”
Here’s the good news: Building an entire new data science department from the ground up to analyze all your big data isn’t your only bet. With a combination of strategically selected manpower and the right software for your data, businesses can leverage value from big data right now--without an in-house data science team.
Here are three such examples of the different approaches companies can take to address their big and complex data analytics challenges in 2016.
3 ways companies are handling big data without hiring a data science department:
1. Choose a Part-Man, Part-Machine Solution
In the tough telemarketing world, it's a numbers game. Not just for the telemarketers but also for the sales teams responsible for new customer acquisition. One example for this approach is used by Qualifa, a UK-based global B2B telemarketing lead generation company with a customer list that includes Cisco, HP, Dell, Symantec and McAfee, wanted to use its data to discover new ways to more efficiently seal deals. Here comes the tricky part: they wanted to do it without an IT middleman, even though their data was scattered across multiple data sources.
Qualifa needed to design sophisticated models and algorithms and allow business people with limited technical knowledge to create visualizations. Using business intelligence software and working closely with their provider, Qualifa to build a scenario where the managers didn’t need a team of dedicated data scientists and provided everyone in the company hands-on access to data needed to evaluate business needs and make impactful decisions.
The company increased productivity by 43% using new ways to get prospects to purchase their telemarketing services, and put its BI insights into the hands of over 200 sales agents in near real time. Using part manpower and part business intelligence software, Qualifa was able to meet its big data opportunity and challenge without hiring data scientists.
Tip: Companies like Qualifa know they have a big data analytics challenge, but often size isn't the only dimension that makes data “complex” and difficult to analyze quickly and meaningfully. You must also take into account where the data is gathered from and if the data is structured or unstructured. All challenges that relate to big or disparate data will make your data complex.
Complex data calls for either a team of data experts to prepare, clean, and correlate data before analysis is possible, or an advanced BI tool with a back-end powerful enough to handle data integration, preparation and management of complex data to deliver data in a form that users can actually analyze.
2. Build an Eclectic Team of People
Since data scientists are hard to come by, you can create a technical team of quantitative analysts along with software engineers that have the skills to write the algorithms, program platforms, and integrate data -- instead of hiring data scientists. Many of these people may already exist in the IT department and cost minimum recruitment time and financial overhead.
For example, a company can pair a team of statisticians and computer scientists with business analysts. By coupling the data geeks with business savvy team members, the data geeks will be exposed to the business challenges throughout every step of the way and the two sides will better communicate.
According to a story published in the Wall Street Journal, Monster Worldwide created a team of data crunchers, statisticians, business analysts, computer scientists and business-minded “navigators”to explain new insights to managers.
Jean-Paul Isson, the global VP of business intelligence atMonster, was quoted in the item as saying: “It’s about understanding the business challenge, creating some valuable actionable insights to the data, and communicating their findings to the business,” about his tailored data scientist team.
Tip: When building a data team, try to find people with an innate sense of curiosity, in addition to other more technical qualities. It’s the curiosity that causes people to ask the big questions. The stronger data DNA the team and company has, the more productive and effective the effort will be.
3. Use a Simple Tool With Simple Data
In a world where it seems nearly everything is labeled big data, it becomes particularly important that businesses understand the amount and complexity of the data they actually want to crunch. Some businesses have simple data and need an Excel 2.0--a simple data visualization tool without a powerful back-end to prepare, integrate or manage data. A visualization tool can turn data into relatable, well-designed data visualizations. The question is not how big is your business, but rather, how big and complex is your data. These days, even small businesses can have very complex data needs.
There are times when a simple data visualization tool can be used to organize data initially and then move to upgrade to solutions that can handle big data when tackling complex calculations and technical charts.
The majority of businesses these days work with terabytes, not petabytes, of data. That will overwhelm basic visualization tools to handle but it doesn’t require a monster-size solution.
Tip: Early on, determine what questions you are trying to tackle and that will help you figure out whether a simple, mid-size or mega-size solution is the right fit. Even if your current needs are easy enough to satisfy, you should also consider the possibility that these needs will be very different in the future -- either because you will be dealing with more data sources, or will want to accomplish more advanced goals with your data analysis, in which case it might turn out that data visualization tools are not scalable and extensible enough to serve as a useful business intelligence solution without requiring additional tools.
The Real Challenge: Big, Disparate Data
As you can see from the examples presented above, the complexity of your data plays a major part in deciding which type of big data solution is the best fit for your particular situation. But what exactly is complex data, and how does it differ from traditional “big data”?
Complex data mainly stems from the fact that data is coming from a large amount of disparate sources, and is rapidly being updated and changing. The “size” of the data is in itself a factor that could imply complexity, but disparity is just as crucial a factor - questions such as how many sources does it stem from, how much is structured versus unstructured, and in how many silos is it stored.
The challenge presented by complex data is that it is typically more difficult to understand, and more difficult to work with. It is not normalized and does not necessarily follow a clear and consistent format or structure. It is produced by many different systems, each speaking its own “language”. To analyze this type of data, preparatory work is required to integrate the various sources and create a single source of truth.
Do I Have Complex Data?
It may surprise you, but even data in Zendesk and your CRM system is not structured in ways that are easily connected -- something seemingly simple, such as a definition of “customer”, might not be the same in one as in the other. Meaning: if you want to analyze data from a CRM with information from your help desk, you already have some complex data on your hands.
A perfect example recently featured in ComputerWorld describes the challenge when Randall-Reilly, a consulting company that helps companies finance large capital purchases, was limited by multiple large spreadsheets and lack of a data scientist team. “We could only sell to big companies who could take spreadsheets and put them into their systems, because they were the only ones who had data scientists,” said James Vogel, vice president of business analytics at Randall-Reilly.
Instead of hiring a data scientist team, Randall-Reilly started using a BI tool, and now his team can easily analyze multiple large spreadsheets as well as pull data from public, private and proprietary sources for more advanced analytics
Data Scientist Team vs. Empowered Business Analysts
Traditionally, complex data used to be considered to be exclusively in the realm of large data science teams, equipped with massive hardware infrastructure and a multitude of proprietary tools. However, as we have seen above, today more and more organizations are dealing with complex data, due to the growth of data and the increasing amount of data sources; coupled with a shortage of data scientists, the need to find alternative systems for dealing with big data becomes clear.
Fortunately, such solutions exist, mostly relying on the rapid advancements made in modern business analytics tools, which allow companies to simplify and streamline many aspects in the process of analyzing data - even when dealing with big or complex data. However, and as can also be seen, different tools are more appropriate for different tasks and scenarios.
The ability to leverage your data creates endless business opportunities that are now open not only to data scientists, but also to business analysts, or even non-technical business users. For many years, only large enterprises could get into the game. But companies no longer need deep pockets and massive resources to tap into the treasures offered by their big data.
Amir Orad, Chief Executive Office at SiSense, the business intelligence software company that lets non-techies easily analyze and visualize big datasets from multiple sources.
Image courtesy of Shutterstock.