Big Data Becomes Fast and Accelerated with IoT

The Internet of Things (IoT) promises to be an internet of devices where the information gathered enables us to do new things. It may seem like a new concept, and it is new in the consumer world, however, the concept has been around for a while in different forms.  Retail and transportation industries have been leaders in developing “smart” applications for quite a while but it is only recently that more off-the-shelf technology is being used and the cloud is being embraced to make it smarter, more flexible and more affordable.

IoT is in the very early stages of deployment. It promises to unlock value and rapidly transform how organizations manage, operationalize, and monetize their assets. With IoT, physical assets become liquid, easily indexed and tracked, enabling identification of idle capacity or over utilization.

IoT is a complex structure of hardware, sensors, applications, analytics and devices that need to be able to communicate geographically and across all functions. Once the data is collected from numerous endpoints, the challenge then becomes converting it into actionable insight.  Sorting and analyzing the output from all of these endpoints, and making it relevant for users, is a problem on a scale we have not seen before.

As more organization s embrace the competitive advantages of IoT, they are concerned that their analytical tools and infrastructure will not support this next wave of industrial revolution. The velocity of technological change and the speed of business is outstripping IT’s ability to transform current data infrastructures and business analytics into IoT-ready systems.

It is clear that organizations understand the value of IoT data, but it is unclear how to actually get from collecting the data to deriving value from the data. Until that’s resolved, IoT projects may prove more frustrating than useful.

There are still four major challenges to gaining widespread value from the Internet of Things:

  1. Data moves to the edge of the network – According to Gartner, as we move into the IoT era, the number of connected devices is set to explode to 25 billion by 2020. Even enterprises that have a modest number of assets when compared to this huge number of devices will have to prepare their data center to handle a significant increase in data. The inbound scale issue surrounding data collection, data storage, and analytics is a major challenge. The data center architecture will need to evolve and evolve fast. As more and more devices are connected to the Internet, they will generate vast amounts of data that need to be processed, managed, stored and analyzed. Automated data processing capabilities will need to be distributed to the edge of network to process IoT real-time data feeds. As more and more data is generated, current data center gear won’t be able to provide the data and traffic flow capabilities that are required to move quantities of IoT data. By decentralizing core compute and storage resources and pushing them out to the edge, organizations can prevent massive volumes of data from crashing the data center. According to IDC, by 2018, 40% of IoT-created data will be stored, processed, analyzed, and acted upon close to, or at the edge, of the network.
  2. Diversified Data - Not only is data growing, it is also diversifying. Organizations require flexibility in handling a variety of data sets including the ability to handle structured, unstructured data, and streaming data with a comprehensive data management strategy. Most data coming from sensors will be structured, in the form of streaming data and it is likely to come from multiple streams. There will be cases where the data coming in will be in the form of images or even video – unstructured. However, all of this data is raw and truly doesn’t have context unless compared with historical data that is likely stored in the enterprise data warehouse. Organizations need to be able to take this variety of data types and multiple data sources and be able to process, organize, and load it into enterprise data warehouses quickly and efficiently. The ability to blend data from different sources including sensor data and enterprise data and aggregate it for a united view will be critical to deriving value from all of the data collected.
  3. IoT Accelerates Data Center Evolution – The data center will need to rapidly evolve to handle big data generated from IoT applications. Because of the influx of data, higher capacity transport will be required coming in to the data center. In fact, more transport capacity will likely be needed throughout the data center. Computing power and storage will de-centralize and move closer to the edge so the IoT data can be quickly processed closer to the devices and closer to the users who be using that data. Computing will become much more distributed and the delineation between edge and core will become blurred over time.  Portions of the applications will run on public clouds, while the rest will run on a private cloud – hybrid will become the new normal. Disruption will become the norm and as a result, data center will need to be fluid and more agile to handle unanticipated changes. And as a result, everything in the data center, including the network, will become virtualized.
  4. Data Security – IoT opens up new challenges with privacy and security. Each device with a sensor is a new endpoint on the network which can now be a target for a security breach. Adding hundreds or even thousands of new endpoints creates a security challenge on a scale never seen before. Issues such as how to handle device lifecycle and software updates on a large scale need to be addressed before IoT will ever become mainstream. As computing and automation of the data streams move to the edge of the network, additional layers of security will be needed. If there is a breach, there are privacy concerns on how the data will be used. The industry must look at IoT applications in a holistic way and create a layered security at the endpoints, at the edge, within the public cloud as well as in the enterprise data center. There is no silver bullet to mitigate threats.

We can expect to see every year within the next 5 years be “The Year of IoT.” Today, many enterprise organizations are just in the initial stages of exploring what IoT is and what it can do for the organization. Many do not see any business justification at this time.  Challenges abound however this will gradually change as the technology matures and more use cases across a range of industries become proven.


Subscribe to Big Data Quarterly E-Edition