As companies measure the health and performance of their IT/cloud/application infrastructure and various data-in-motion streams such as metrics, events, logs, and traces, the number of monitoring tools grows, resulting in a complicated and overloaded observability stack.
Kishore Gopalakrishna, co-founder and CEO at StarTree, believes unified observability platforms can solve these pain points by providing the flexibility, data autonomy, and extensibility offered by a disaggregated model.
StarTree’s premier platform, StarTree Cloud, is a fully managed, user-facing, real-time analytics platform designed for online analytical processing (OLAP) at massive speed and scale, powered by Apache Pinot. StarTree Cloud provides enterprise-grade reliability and advanced capabilities such as tiered storage, plus additional indexes and connectors.
With a passion for solving complex problems in distributed systems, Gopalakrishna has worked on projects such as Apache Helix, a cluster management framework; Espresso, a distributed document store; and ThirdEye, a platform for anomaly detection and root cause analysis at LinkedIn. At LinkedIn, he also developed Apache Pinot, the real-time analytics datastore designed to deliver scalable and ultra-fast analytics.
Can you explain the concept of Observability (O11y) and its significance in the context of real-time analytics?
The definition of observability is quite broad and continuously changing over time. At first, it catered to the needs of observing system metrics, logs, sales, real-time analytics, etc. Now, we’re observing everything that happens in real time in the real world.
It has broadened in the last decade to include everything we do on the web. We look at comments and likes; these things need to be observed. We need to see if the information is human- or machine-generated. The next level is to go from collecting data to insights and then act on top of it.
What is a disaggregated stack approach? What are the key benefits of this approach, and why do you believe it’s more advantageous for modern enterprises?
The way to think about this is [that] these problems aren’t new. What is changing is the amount of tools companies now manage. Companies want to do additional things; they must bring additional tools, from observing to collecting to storing. This causes the duplication of data.
What organizations need is a specialized system to standardize on horizontal systems. This opens how they’re built. There are systems that transport, store, and provide queries and then systems that lie on top.
Instead of each solution being its own stack, there is one stack at every layer. The system has the same stack for visualizing data; you can go from a vertical to horizontal stack.
This gives people options in terms of what they can do. The benefits, once companies reach this scale, expand the collection of this metrics data. It doesn’t need to just be about visualization. You could do machine learning, and the solution doesn’t expose the data. The advantage is that everything is in a single pane of glass.
What are some common challenges enterprises face when transitioning to a disaggregated stack for observability and how can StarTree help them?
The biggest challenge is that the internal folks are already used to the existing tools. This needs to be a companywide initiative. It’s hard to pull off in isolation. You need buy-in from everyone. It’s important to get people, processes, and tech in place before shifting from a bunch of platforms to a disaggregated stack approach.
Then, companies must have the proper layers built in, such as Kafka, Pulsar, etc. These streaming systems act as the backbone to see these events. While building the next set of layers, you need a skillset/experience in building model data, which comes with additional overhead.
This is where StarTree comes into the picture. We are an established technology for capturing and storing real-time data. The storage layer is where StarTree is happening, forming the horizontal layer. The tech on top is the real-time analytics platform, which stores metrics and logs efficiently. We help people go from an “all or nothing” stack to that desegregated stack.
StarTree is also building connectors to existing tools. We don’t want companies to change tools at the last mile, so we provide these connectors, and end users don’t see too many changes.
What are your predictions for this space in the future?
This is an interesting space that keeps evolving. When you look at observability, you see companies move from on-prem to the cloud to slowly having systems being commoditized to adopting a disaggregated stack. There is value in what data they are capturing, and you can get a small picture of what they’re collecting. This is a democratized way people can build apps. You can open up to the next level where people can build tools on top of it.
This area is also ripe for AI disruption. You can ask a lot of questions on this data. Right now, we are only focusing on knowledge and static information. When it comes to analyzing data continuously, it’s changing in real time and that becomes more interesting with the way world is moving with AI right now. More can be built on top of data. It’s going to be exciting to see the types of apps people can come up with.