Hazelcast, a provider of an open source in-memory data grid (IMDG) platform, is unveiling Hazelcast Jet, a new platform that gives users a distributed processing engine for big data streams.
With Hazelcast’s IMDG providing storage functionality, Hazelcast Jet is a new Apache 2 licensed open source project that performs parallel execution to enable data-intensive applications to operate in near real-time.
“The simplicity of programming and deployment is going to be a real breath of fresh air in this space,” said Greg Luck, CEO, Hazelcast.
Using directed acyclic graphs (DAG) to model relationships between individual steps in the data processing pipeline, Hazelcast Jet is simple to deploy and can execute both batch and stream-based data processing applications.
Hazelcast Jet is built on top of a one-record-per-time architecture (sometimes known as continuous operators). This means that it processes incoming records as soon as possible, opposed to accumulating records into micro-batches, consequently lowering latency for applications.
Important components to the platform include an events-based architecture for low latency Transaction Processing System (TPS) applications, directed acyclic graphs (DAG) to model data flow, high level java.util.stream API for great simplicity, low level Core (DAG) API for maximum flexibility allowing direct manipulation of vertices representing data source readers, joiners, sorters, aggregators and data sinks, and distributed connectors for reading and writing: Hazelcast IMDG Map and List, HDFS, Kafka, File, Socket.
Java developers and architects will benefit the most from this platform, according to Luck.
“They can embed Jet in their application and if they do that, there are no additional moving parts above what they already have,” Luck said. “It’s very simple operationally.”
After the platform is released the company plans on having an intensive interaction experience with its community.
“We’re going to be very interested in the first use cases people apply this to,” Luck said. “We may have a cadence of releases, maybe about once a month.”
According to Luck, the company will also work on aligning the Hazelcast platform with Jet so the two platforms will work together more harmoniously.
For more information about this news, visit hazelcast.com.