Ably, a real-time experience infrastructure provider, is debuting the latest iteration of the Ably Kafka Connector, enhancing a variety of areas including throughput, scalability, error handling, and troubleshooting, as well as accreditation as a Confluent Cloud Custom Connector.
Ably Kafka Connector 3.0 continues to deliver efficient and reliable Kafka pipeline extension capabilities for developers, now with improvements toward several features.
Enhanced throughput and scalability within the solution’s 3.0 version is a result of Dynamic Channel Configuration. With the ability to generate an Ably channel—which is equivalent to Kafka topics and optimized for cross-network communication—per user or session, developers benefit from more streamlined distribution of relevant information to clients via the internet. This ensures both flexibility and scalability for message routing from Kafka topics, supporting millions of concurrent clients.
The release also highlights the connector’s newfound REST support, which allows it to record publication to Ably utilizing parallel publishing through a thread pool mechanism, ultimately enhancing overall throughput.
This version of the Ably Kafka Connector also features Kafka Connect dead letter queue (DLQ) support, where the connector can forward records to a DLQ topic. This greatly eases the debug and troubleshooting process for developers, where issues can be resolved before they lead to critical system failures or missed data sets, according to the company.
“Kafka is not designed for fine-grained data distribution over the internet to variable numbers of client devices,” said Matthew O’Riordan, CEO of Ably. “Building and maintaining an internet-facing messaging layer for real-time data distribution from Kafka to web, mobile, and IoT devices can be complex and resource-intensive. Using our Ably Kafka Connector, developers can effortlessly and securely distribute Kafka data to internet-connected clients via Ably, so they can focus more on their core engineering tasks.”
Now certified as a Confluent Cloud Custom Connector, the Ably Kafka Connector can be leveraged as a plugin on Confluent Cloud. This accelerates Ably to Kafka connection, without changes to the code, while guaranteeing high availability and performance. Logs and metrics are leveraged to monitor the health of connectors and workers, alleviating developers of the pains associated with provisioning and managing low-level connector infrastructure.
Ably emphasizes its mission to extend and improve the Ably Kafka Connector, welcoming both feedback and feature requests. The solution is available under the Apache 2 open source license.
To learn more about Ably Kafka Connector 3.0, please visit https://ably.com/.