Astronomer, the leader in modern data orchestration, is introducing a new set of Apache Airflow integrations to accelerate LLMOps (large language model operations) and support AI use cases.
Modern, data-first organizations are now able to connect to the most widely-used LLM services and vector databases with integrations across the AI ecosystem, including OpenAI, Cohere, pgvector, Pinecone, OpenSearch, and Weaviate.
By enabling data-centric teams to integrate data pipelines and data processing more easily with machine learning (ML) workflows, organizations can streamline the development of operational AI, according to the company.
Astro provides critical data-driven orchestration for these leading vector databases and natural language processing (NLP) solutions, driving the MLOps and LLMOps strategies behind the latest generative AI applications.
Supporting the entire AI lifecycle, from prototype to production, Astro provides "day two operations" that include monitoring, alerting, and end-to-end lineage, and guarantees enterprise-grade uptime to help prevent critical outages to AI operations, according to the company.
Astro also prioritizes collaboration between data and ML engineers, from traditional data pipelines and getting ML production-ready, to building AI applications on Airflow.
"Organizations today are already relying on Astro and Airflow to harness the data required to fuel LLMs and AI. With these new integrations, we are now helping organizations realize the full potential of AI and natural language processing, and optimize their machine learning workflows," said Steven Hillion, SVP of data and AI at Astronomer. "These integrations put Astro at the foundation of any AI strategy, to better process complex and distributed volumes of data with the open source and proprietary frameworks that drive the current generative AI ecosystem."
These integrations further extend the benefits of Astro and Airflow to an organization's AI strategy by:
- Improving data lineage
- Data availability
- Flexibility and agility
Astronomer is also making Ask Astro, its LLM-powered chatbot, available in the Apache Airflow Slack Channel, and sharing the source code as a reference implementation.
Ask Astro leverages a wealth of Airflow knowledge from Astronomer-specific documents across Github, Stack Overflow, Slack, the Astronomer Registry, and more, making it immediately available as a starting point for developers who are looking to operationalize their applications, according to the company.
For more information about this news, visit www.astronomer.io.