Hortonworks revealed upcoming advancements of Hortonworks Data Platform (HDP) with the in-memory analytic capabilities of Spark.
Apache Spark 1.5.2 will include support for Spark SQL and Spark Streaming. Hortonworks’ commitment to Spark is focused on helping customers accelerate data science, maintain seamless data access, drive innovation at the core and ultimately scale for the enterprise.
Hortonworks has also launched Hortonworks Community Connection (HCC), a new online collaboration destination for developers, DevOps, customers and partners to get answers to questions, collaborate on technical articles and share code examples from GitHub. The new online community is an extension of Hortonworks’ open source roots and underscores its commitment to engaging with the community, and fostering community innovation and knowledge.
“We continue to see customers across all industries derive real value from using Spark with Hortonworks Data Platform,” said Tim Hall, vice president of product management at Hortonworks. “Our customers rely on us to guide them on their Spark journey, and our ability to scale Spark against massive data-sets is remarkable. With the inclusion of Spark 1.5.2 on HDP, customers can get new Spark capabilities and maximize its value for the enterprise.”
Customers can deploy modern, Spark-based applications alongside Hadoop workloads in a consistent, predictable and reliable way.
According to Hortonworks, its three main areas of focus for Spark include:
- Data Science Acceleration: Improving data science productivity by enhancing Apache Zeppelin, currently available as a technical preview, and by contributing additional Spark algorithms and packages to ease the development of key solutions. One example is Project Magellan, an open source library for geospatial analytics that facilitates geospatial queries and builds upon Spark to solve hard problems dealing with geospatial data at scale.
- Seamless Data Access: Hortonworks is improving Spark’s integration with YARN, HDFS, Hive, HBase and ORC because customers are running Spark on YARN in combination and in conjunction with many of the other popular data access engines. Specifically, Hortonworks is working to further optimize data access via the new Data Source API. This will allow Spark SQL users to take full advantage of ORC File instantiation as a table, Column pruning, Language integrated queries, and Predicate pushdown
- Innovation at the Core: Enhancing Spark’s enterprise security, governance, operations and overall readiness for real-world production deployment.
Fostering Community Innovation
More information about Project Magellan can be found within Hortonworks Community Connection, which currently hosts thousands of technical articles and FAQs on Hadoop, Spark and other big data technologies contributed by Hortonworks engineers and other technical experts.
Registration for HCC is now open to the public at http://hortonworks.com/community.
Additional information about HDP and Spark can be found at http://hortonworks.com/blog/spark-hdp-perfect-together.
Image courtesy of Shutterstock