Big data with its increasing volume, velocity, and variety of data is causing organizations to see their data as a valuable commodity and creating new business models. But central to that value, says Amit Walia, chief product officer of Informatica, is the ability to bring all data together to get a single view. Here, Walia discusses the three key challenges organizations face and what he sees as the biggest disruptor on the horizon.
What is happening in big data management today?
Amit Walia: Big data clearly has become something that everyone wants to do. Think about a large bank with retail banking, commercial banking, and wealth management – they want to get a single view of their customers and understand customers’ buying patterns so that they get a holistic view of them. Or, think of a large manufacturer that has a lot of data on their shop floor and wants to be able to want to bring all that data together for predictive maintenance.
How mature is it?
Walia: It is relatively early in its journey compared to other technology changes. Big data is still a relatively nascent space. If you even compare it with cloud computing, cloud is far more established than big data.
What are the key challenges customers are facing?
Walia: There are three things that customers reaching out to us and asking for help with. They want an operationally scalable, reliable, secure solution, end to end, that is holistic and operationally scalable. Number two, they need help managing the skills gap in this whole rapidly changing technology landscape. And, number three, they need business and IT to work together so that value can be created very quickly versus waiting for some long-term answers.
What are they trying to do?
Walia: First, customers are clearly trying to create value from big data. They want more than piece parts that they have to cobble together themselves.
The second thing customers are struggling with is skills. They turn to us because, in addition to big data being a new space, technology is also changing very rapidly and it will continue to change. The number of projects in Apache Hadoop keeps changing. The skills gap is massive. Large banks, healthcare providers, and retailers want to depend on a vendor that can take the whole skill set issue away.
And number three, there is a big change happening in the world of big data. Self-service - that is the whole idea. You don’t want to create some massive IT project that takes years to bear fruit, before business users can benefit from it. Business users want to be able derive value now. This whole business and IT time-to-value issue is a big thing in the world of big data because again, it’s the idea is that you make quick decisions and you can create value instantly.
How is Informatica addressing those three key issues?
Walia: Operationally scalable, reliable - that is Informatica’s bread and butter, and has been for years. Our offerings have been used by the biggest and most demanding customers across the globe and our breadth in terms of data management is obviously very broad too.
How do you offer that?
Walia: The first is top-grade big data integration, the ability to connect and bring data from a lot of places, transform it, to govern it, master that data. That is critical for large customers and the ability to secure it - that is made possible by our Big Data Management product. It is holistic and has breadth and scale. We sit on any Hadoop ecosystem - Cloudera, Hortonworks, AWS, Azure - we connect all data types - SAP, Oracle - that is the breadth and scale we already have.
How do you approach the skills gap?
Walia: Our ability to abstract the complexity away from the customer is huge. We have our own native engines, and we support Spark, Spark Streaming, MapReduce, Hive - all of those things. Our claim to our customers is that they don’t have to worry about any of that. They don’t need to learn anything new; our abstraction engine will decide whether to run this in Spark, run this in Spark Streaming, or run this in our native engine. That abstraction allows our customers to be future-proofed.
How do you deal with the need for faster time to value?
Walia: We have taken steps to deliver what we call a business-focused data solution. We announced the Intelligent Data Lake, and a version of that is the Marketing Data Lake where we are basically providing a solution that IT can give to business users and, out-of-the-box, it is a complete end user business solution enabling them to search for any kind of data across the lake or the enterprise. And, obviously, our intelligent metadata capability allows them to discover through machine learning things that they wouldn’t know. For analysts, the biggest issue is that they don’t know what they are looking for. They want to find customer data but they just don’t know where it sits in the company.
Our integrated preparation or data wrangling capability allows customers to bring data together and play with it in an easy Excel-like framework. And then, if they want, they can operationalize that by creating reports, visualize that in Tableau, Qlik, or SAP BusinessObjects through a business interface, and we also have machine learning that gives them recommendations.
As far as the cloud, hybrid seems to be the dominant approach going forward.
Walia: Absolutely. I think hybrid is the way to go. No existing customer is going to have their infrastructure be 100% on premise or 100% cloud. We hear this from every customer.
Customers may have technologies such as Cloudera and Hadoop on-prem and be leveraging our capabilities of Big Data Management and Intelligent Data Lake. And then, they want to be able leverage something like HD Insights that Microsoft Azure has or even AWS capabilities for certain workloads.
We allow them to run their workloads on-premise or in the cloud. Customers are not making an either/or decision; they are making “and” decisions based on the workloads that are best running on-prem and best running on the cloud. The world will continue to remain hybrid for years to come.
Do you find that customers need help in putting together a hybrid architecture?
Walia: We saw that a couple of years ago as the world was evolving - that clearly customers were initially looking at each of these technology disruptions - on-prem, cloud, and big data - as a separate, isolated architecture and then they quickly realized that they are all interwoven. The glue across all these architectures is their data. They can get great customer data coming from end user devices - that is IoT; and from websites – that is big data; and they have customer data sitting in their existing on-prem infrastructure.
What does that mean for their architectures?
Walia: The value is connecting that data so they can do something with it, so if they create parallel architectures, they are not going to be able to do that. They turn to us because we are focused on all things data. We started our journey 2 years ago with something called the Intelligent Data Platform, where we are bringing the whole organization’s enterprise data together and the fundamental core capability of that is our intelligent metadata. We call that the Live Data Map - it is an enterprise-wide solution that allows you to bring metadata across your entire enterprise so that then you can get a single view of wherever your data sits – whether it is in a cloud, on-prem, big data, IoT - whatever it is.
Customers want that because that is where the value is and that is why we have invested in our Data Platform’s metadata capabilities.
Integration is central to big data success.
Walia: The biggest disruption today in the global economy is data, and if you cannot bring all of your data together to get a single view, you are missing a huge trick. That is what customers are telling us.
Is there a big data technology that is yet untapped?
Walia: Big data means really big data. Even the weblogs and click data that companies are collecting about their customers is still not big enough. It is IoT that is going to disrupt everything.
Think about it - Ford is using the whole connected car initiative to move from just being a car company to knowing more about you and me when we get in the car: what are we looking at, what are our habits, what radio station do we listen to, where do we click? Basically, a car is a connected device. IoT is going to disrupt business models and be the biggest disruptor of how companies become better placed than their competitors. That is the area to watch out for, but it is still in its infancy.
If you had one piece of advice for an organization starting a big data project, what would that be?
Walia: Think of the long-term value that you want to create for your organization and invest in solutions that can deliver it. Don’t embark on science projects. They don’t create any value and when you bring science projects into your organization, you don’t instill any confidence. Think of the real value and invest in solutions and partners that can help you deliver that value. Think of what is operational, scalable, and secure versus science projects.
This interview was conducted, edited, and condensed by Joyce Wells.
Image courtesy of Shutterstock.