Big Data and Information Management Predictions for 2016

What’s ahead for 2016 in terms of cloud, IoT, big data, analytics, and open source technologies?  IT executives gaze into their crystal balls, and offer their thoughts on the upcoming challenges and opportunities ahead in the coming year - and beyond.

100% of development and testing is moving to the cloud, thanks to the clear ROI

As CIOs evaluate which cloud technologies make sense to implement when, one of the first moves they make is transitioning development-and-testing to the cloud. Cloud-based app development and testing is such a clear ROI opportunity that Oracle CEO Mark Hurd predicts it will be done entirely in the cloud by 2025. When dev-and-test is done using on-premises software and hardware, IT teams have to buy, license, and configure everything from servers to databases needed to create a development environment that hopefully matches the environment an app will someday run in production. Hurd estimates that work wastes about 80% of the resources spent on dev-test, not to mention slowing the work down. Cloud dev-and-test can still fit into the reality that cloud and on-premises systems will co-exist for many years to come. IT teams often want to develop-and-test in a cloud environment and then move an application back on-premises for production, often for regulatory reasons. Look for companies to choose cloud development platforms that give them the choice to either scale up in a public cloud or run on-premises, once the app’s production ready.

IoT Will Move from Hype to Substance

“The Internet of Things has already emerged as the next mega-trend but in 2016, it will excel beyond just hype,” says Stefan Groschupf, CEO of Datameer. “We will see companies actively change their strategy and infrastructure to harness the power and insight of IoT technologies and data.”

Storage (Particularly Flash) Will Become an Extremely Abundant Resource

Next-generation, software-based storage technology is enabling multi-temperature (fast and dense) solutions, observes MapR’s CEO and cofounder John Schroeder. In particular, he notes, flash memory is a key technology that will enable new design for products in the consumer, computer and enterprise markets. Moreover, consumer demand for flash will drive down its cost, and flash deployments in big data will begin to deploy. The optimal solution, according to Schroeder, will combine flash and disk to support both fast and dense configurations. In 2016, he says, look for a new generation of software-based storage that enables multi-temperature solutions to proliferate so organizations will not have to choose between fast and dense, and will be able to get both.

Security Will Need to Get Smart to Thwart New Cyber Threats 

Many devices in the smart home are always on and always communicating. The connected home could become the simplest way to hack into people’s lives, according to Intel Security’s predictions on the future of cybercrime and security for 2016 and beyond in the McAfee Labs Threat Predictions Report. “The best hockey players navigate within the ice rink, grapple with opposing players, take advantage of opportunities when available, and, critically, as Wayne Gretzky said, always skate to where the puck is going to be—not where it has been,” stated Vincent Weafer, vice president of Intel Security’s McAfee Labs. “To address the business, technology and threat landscape realities facing them, we must help organizations get to where they need to be, using technologies that will enable and not hinder their businesses, and understand what kinds of threats could be confronting them tomorrow and far into the future.”

Spark Will Get Even Hotter

Spark is part of Hadoop distributions and is widely associated with Hadoop. Expect to see that change in 2016 as Spark goes its own way, establishing a separate, vibrant ecosystem. In fact, you can expect to see the major cloud vendors release their own Spark PaaS offerings. Will we see an Elastic Spark? Good chance,” notes Bob Muglia, former president of Microsoft’s $16 billion server and tools business and current CEO of Snowflake, in a recent blog post. “It provides an efficient, general-purpose framework for parallel execution. This is very useful in today’s world where data analysis often requires the resources of a fleet of machines working together. While Spark is still relatively immature, it has the potential to evolve into the standard framework and API for parallel algorithmic analytics and machine learning.

Protection of Data Itself Will Become Paramount

 “While hundreds of millions of dollars have been invested on perimeter and data center layer security, in-depth defense at the data layer is still in its infancy,” says  Eric Tilenius, CEO of BlueTalon, a company that delivers data-centric security for Hadoop, SQL and big data environments deployed on-premises or in the cloud. According to Tilenius, it’s still too easy for hackers to circumvent perimeter defenses, steal valid user credentials, and get access to data  records. “In 2016, as companies protect themselves from the threat of data loss, new means of data-centric security - protection of the data itself that is - will become mainstream to consistently control user access and credentials where it matters the most.”

The Demand for BI and Analytics Will Reach New Heights

"In 2016, we will see business intelligence (BI) and analytics reach new heights. As advanced data technologies emerge, businesses will process and store more information than ever before,” said Brad Peters, co-founder and chief product officer of Birst, a provider of cloud BI and analytics.  “As a result, they will be looking for a next-generation BI and analytics platform that helps them tap into the power of their data, whether in the cloud or on-premises. This ‘Networked BI’ capability creates an interwoven data fabric that delivers business-user self-service while eliminating analytical silos, resulting in faster and more trusted decision-making.”

In-Memory Processing Will Continue to Thrive

“It’s been interesting to see 2015 and there’s a lot of progress made around the adoption of in-memory,” said David Jonker, senior director of big data initiatives at SAP. “IBM’s commitment to Spark, for example, was yet another endorsement of how in-memory processing is really coming to be a central piece of everyone’s big data strategy.”

His expectation is that there will continue to be a strong growth in this area.

“The idea is by simplifying your environment and getting rid of a lot of legacy systems and bringing it onto a central in memory platform you can actually do both your operations and analytics,” Jonker said.

Analytics and Big Events Will Drive Demand Exponentially

Analytics has the potential to play as important a role in the winning of the next presidency as their policies, with data analytics going beyond polling to inform TV media buys, debate results and more, according to Laura Sellers, vice president of product management at Alteryx. Analytics also has the opportunity to change the way people interact with large-scale events, such as the upcoming Olympics in Brazil in 2016, creating an increased demand for data-driven insights.

To Deal With the Shortage of Data Scientists, Companies Will Rely More on Big Data Cloud Services

Salaries for data scientists and Hadoop administrators will see a sharp increase in 2016 as growth in Hadoop demand exceeds the growth of available talent, according to Mike Maciag, COO, Altiscale, a Hadoop as a service startup. To circumvent the need to hire more data scientists and Hadoop admins, organizations will rely on fully managed cloud services with built-in operational support, freeing up existing data science teams to focus their time and effort on analysis instead of wrangling complex Hadoop clusters, he notes.

The Shift from Fibre Channel to Ethernet Will Accelerate

"In the past, Fibre Channel's speed advantage made it the standard for enterprise storage networking. However, with the wide spread deployment of 10Gbe Ethernet and the transition to 40Gbe the speed advantage of Fibre Channel over Ethernet has evaporated,” said Jacob Cherian, vice president of product management and product strategy at storage platform provider Reduxio. “Ethernet's speed and cost advantage along with that fact that Ethernet is backed by Cisco and the other major companies in the networking space, will lead more enterprises to integrate Ethernet into their enterprise storage architectures."

Big Data Quandary Will Continue as Companies Try to Understand its Value

Data is continuing to become more and more relevant within business circles. But how many companies are taking their data and making practical business use with it. Walker White, CEO with BDNA, which provides solutions for consolidating and maintaining enterprise IT data, notes that this is an issue that will persist in 2016. “Today’s big data phenomenon can be compared to data warehousing when it first started about 15 years ago – the data collected was useless until it was put into the right context for solving specific business problems. Big data is similar in the sense that it provides the raw materials to provide significant insights, but has no value without the right tools to analyze it. Some strides are being made in data analytics, but big data is still far from the panacea that has been promised.”

Cloud Will Become the Platform of Choice for IoT

Cloud will become the de facto platform for IoT, says Marc Olesen, SVP and GM of cloud solutions at Splunk, which provides a platform for operational intelligence. As organizations increasingly bring IoT devices to market, cloud will be the leading platform for collecting and analyzing data generated by these IoT devices and to ensure their uptime and performance.

Digital Business Makes Its Way To The Corner Office

The C-suite increasingly understands the value of digital business and the data it produces, but they have yet to bring new executives permanently into the room to drive a digital agenda across their company - and that's about to change, according to Jeremy Burton, president of products and marketing with EMC. Increasingly, he notes, Chief Digital Officers will be gvien the responsibility of deploying the newest technology to make products smarter and create frictionless service experiences for customers.  And, data – the digital exhaust that these new capabilities leave behind – will need to analyzed and exploited for competitive advantage.   This is not something that will happen only in the tech industry but will be across the board in every industry.  Even though the C-suite executives know it is the right direction, they will still grapple with decisions about org charts, duties, and turf wars. Should the Chief Digital Officer report to the CEO, the CMO … or the CIO?