The perils of constructing a big data project are many. Even in the relatively short history of businesses delving into big data projects, there is already an established history of high rates of failure.
The reasons for failure are many, but often fall under the umbrella of a lack of careful planning and a failure to effectively reconcile the panoramic visions of business objectives with technical and structural realities, according to a recent survey. Business objectives are often abstract and high level, and operationalizing those abstractions into the kind of detailed information that feeds the development of effective big data projects is often to be found at the root of that failure.
Build a foundation of business understanding
Selecting the right technical solutions for the project is of course important, but is difficult to do properly without first having a clear enumeration of the business objectives motivating the project. Those objectives are often quite diverse from one project to the next, and significantly guide the path that the development of the rest of the project will take.
Define success for the big data project concretely
Precision and measurability are the keys to defining the business objectives that will be addressed by the project. Particularly important are clear ideas about what successful implementation of the project will look like in its fully rolled-out form, and who the interested parties are that will be affected by the project. These stakeholders who should be accounted for at this stage include those who have a role to play in the planning and design processes as well as those who will ultimately benefit from the results of the project.
Among the interested parties that will play the largest day-to-day role in the execution of the project is the project team itself, which should be constructed and identified at this early stage. This process should include an in-depth understanding of the resources afforded by each team member and their specific roles and responsibilities. Identification of a project “sponsor” is a good idea. This person is responsible for being the project’s champion: removing obstacles, obtaining necessary resources, and representing the interests of the projects and its stakeholders.
Quantify the abstract business objectives
With a sound foundation in place, it is possible to begin to distill the abstract business objectives down into their more quantifiable elements. For each specific business objective, identify the relevant concrete measure. Quantifying the process allows more realistic projections for success and obstacles than working in the theoretical realm. Flawed expectations of the process can lead to many of the common pitfalls that plague big data projects: scope creep, unclear goals, passive or non-existent project management, poor communication, and starting too small.
Define the technical requirements
Identifying which technical tools are best suited for the project begins with a re-examination of the available data, and a survey of the relevant tools available. Sorting through the data involves determining what is useful to the project and what can be discarded, and how the particular qualities of the data can be used to achieve the desire goal. Descriptions of data should focus on its quality and quantity. It is important to keep in mind that positive attributes of data sources often have their negative counterparts. The higher the volume of data, for example, the higher the potential for arriving at the desired result; but higher also is the expense in time and money of processing time.
Defining how the data will be worked with is also a key step. Knowing who will be working with the data, the systems that are involved with data processing, and any related challenges is key in selecting the right technical tools for the project. The bottom line in choosing technical solutions is to keep the project goals in the forefront. The capabilities of a particular tool are only valuable if they can be effectively applied to the demands of the project.
Put the big data project in context
With the core elements of the project in place, it’s a good idea to examine how the project fits into the broader picture of the life of the business. Big data projects are often a big investment, and understanding how the project will continue to find its place in a maturing environment is one of the most important factors in ensuring its long-term success. The business and its surrounding environment may change; is the project prepared to adapt with those changes and continue to provide value to the business?
Comprehensive and long-term planning as well as meticulous fleshing out of the abstractions into their concrete and measurable details, will enable big data project planners to avoid the worst of the big data pitfalls.
About the author
Jim Kaskade, CEO of Infochimps, has led companies in cloud computing, software as a service (SaaS), online and mobile digital media, online and mobile advertising, and semiconductors from their founding to acquisition.
(Image courtesy of Shutterstock)