StreamSets, provider a DataOps platform for modern data integration, is upgrading its platform with innovations that help companies efficiently build and continuously operate dataflows.
New capabilities include data drift handling for cloud data stores for improved pipeline resiliency, continuous integration and delivery (CI/CD) automation that brings DevOps-style agility to dataflow pipelines, and the ability to centrally manage in-stream data protection policies for security and compliance.
These features build on StreamSets DataOps Platform’s catalog of cloud connectors, its cloud-native architecture for easy cross-platform deployment, and its ability to elastically scale dataflows via Kubernetes.
Features such as data drift handling and in-stream data protection are powered by StreamSets’ Intelligent Pipelines capability, which inspects and analyzes data in-flow.
“As our customers embark on their hybrid cloud journey, we see first-hand their struggle to orchestrate end-to-end management of data movement across a growing range of on-premises and cloud platforms,” said Arvind Prabhakar, CTO, StreamSets. “Our DataOps platform was architected as cloud-native from the start, allowing us to easily evolve with the market. Cloud drift-handling and CI/CD for dataflows are unique enhancements that help our customers on their journey from traditional to modern data integration based on DataOps.“
With these new features, which began rolling out in late August, StreamSets DataOps Platform now offers:
- Development automation through a full-featured dataflow designer that includes “easy button” connectors for Amazon S3, Elastic MapReduce (EMR) and RedShift; Azure Data Lake Storage, HDInsight and Azure Databricks; Google DataProc and Snowflake
- Elastic scaling of cloud, multi-cloud and reverse hybrid cloud dataflows via Kubernetes
- New data drift handling, which automatically reflects updates to source schema in Amazon Athena, Azure SQL and Google BigQuery cloud data services
- A new CI/CD framework for automating frequent changes to dataflows through iterative design, test, validate and deployment steps
- New central governance of StreamSets Data Protector policies that detect and deal with sensitive data such as PII and PHI
StreamSets DataOps Platform will be at the Strata Data Conference (booth #935), September 11-13 in New York City.
For more information about these updates, visit www.streamsets.com.