Lucid Imagination Combines Search, Analytics and Big Data to Shed Light on ‘Dark Data’

Lucid Imagination, a developer of search, discovery and analytics software based on Apache Lucene and Apache Solr technology, has unveiled LucidWorks Big Data, a fully integrated development stack that combines advantages of multiple open source projects including Hadoop, Mahout, R and Lucene/Solr to provide search, machine learning, recommendation engines and analytics for structured and unstructured content in one solution available in the cloud.

“With more and more companies being challenged by the explosive growth of information, as has been widely reported, the vast majority of that content is unstructured or semi structured text, and traditional business intelligence or traditional  analytics methodologies don’t come close to addressing the vast percentage of content,” Paul Doscher, CEO of Lucid Imagination, tells DBTA. The new solution, the company says, is targeted at the growing problem of “dark” data, which is contained in stored audio files, text messages, email repositories, log files, transaction applications, and various other content stores. According to Lucid, organizations have little to no idea how much lost opportunity is hidden in the vast amounts of data they’ve collected and stored.

With the new offering, LucidWorks Big Data, Lucid Imagination has integrated various open source projects in order to provide a cloud-based open source software stack for a broad range of use cases, giving technologists and business users the ability to initially pilot big data projects in a cloud situation, without the overhead and risk associated with setting up an on-premise deployment. By integrating these technologies and providing them in the cloud, an enterprise can provide a representative data set or use a data set that Lucid makes available to it, says Doscher. The cloud deployment enables organizations to focus solely on the delivery of business value and avoid the high cost of experimentation, including long lead times associated with infrastructure and application development lifecycles prior to placing their big data solution into production. “We have done all the integration. All you are dealing with is a set of APIs that allow you to develop your application much more quickly and take advantage of those technologies without having to become experts in them,” says Doscher.

The result is that organizations have a training environment and prototyping environment, where they can develop their skills and knowledge around the technology. “It allows them to focus on the business value they hope the big data application will provide. It enables them to validate and test, and to get user acceptance before they actually make the significant investment that these types of applications require in order to build out the hardware, infrastructure, software, and services,” says Doscher.

Lucid Imagination delivers enterprise-grade search development platforms built on the power of Apache Lucene/Solr open source search. In addition, it says, out of the 35 Core Committers to the Apache Lucene/Solr project, nine individuals work for Lucid Imagination, making the company the largest supporter of open source search in the industry. Customers include AT&T, Sears, Ford, Verizon, Cisco, Zappos, Raytheon, The Guardian, The Smithsonian Institution,, The Motley Fool, Qualcomm, Taser, eHarmony and many other household names around the world. Lucid Imagination investors include Shasta Ventures, Granite Ventures, Walden International, and In-Q-Tel

Learn more about the company at

For more information about LucidWorks Big Data, go to