Big Data is no longer “hype,” but a   reality. Customers are progressing from   experimentation in small R&D groups   to implementing use cases and taking   applications into production. Companies   are identifying the gaps in the industry’s   Big Data and analytics offerings, which are quickly being   filled by the open source movement and enterprise   vendors. For example, look at the quick sequence   of capabilities that have been released around SQL,   metadata management, encryption, interactive query   processing, and analytics.

   Also, Hadoop is evolving into a heterogeneous   computing and storage platform. The platform will   continue to evolve and will become the next generation   data warehouse and visualization base,   bridging the gap between transaction   processing and analytics needs. The   industry demand will shift from a data   landing zone, serving downstream   warehouses and data marts, to a   true polystructured warehouse   with large scale in-place analytics   capabilities, built into the   warehouse substrate. This   will drive the need for a   variety of storage engines   on that warehouse base:   key value, columnar,   hierarchical, graph,   document, JSON, and   dimensional, accessed   by a common set of   interfaces ranging from SQL, REST API, Document API,   Link traversal, and Search.

  Over the next few years, I see a few   key trends in the Big Data space:
 • The Hadoop platform will mature into a heterogeneous   computing platform for batch, interactive, analytical   and real-time workloads.
 • The maturation of models like Docker and OpenStack   will make it easier to set up and manage large clusters.   • More Big Data implementations will move from being   on premise to the cloud.
• Companies will have the ability to write and run   analytics algorithms across different architectures, exploiting   MapReduce, Spark and even SQL.
 • Finally, the rise of hybrid Big Data platforms will force   a cultural shift within the enterprise. Organizations will   become much more data driven, applying insights to   everything from key business process and decisions   to the way they fundamentally operate. Rather than   relying on decisions based on gut feelings, businesses   will infuse analytics into everything that employees touch   (management systems, machine to machine processes,   daily decisions & tasks, etc.) to develop data-driven and   evidence-based cultures and workforces.

 The following are critical success factors for the   success of Big Data projects:
 • The availability of skills to design and implement large   scale projects that cuts across data management, and   analytics.
 • Breakthroughs in visualization for data discovery,   correlation and exploration of data involving complex   data.
 • Built-in governance capabilities like encryption,   masking, lineage, life cycle management, auditing in the   Big Data infrastructure.

 Criteria to evaluate Big Data  outsourcing vendors 
Whether you are evaluating Big Data development   in-house or via an outside vendor, the most   important factor to consider is longevity. Your   strategy should not be based on short-term goals,  but instead focus on long-term business growth. Companies need a technology infrastructure that will   support the Big Data and analytics capabilities of the   future, which will ensure your business’ growth and   success.

  Taking a “platform” approach is key, as it allows users to address the full spectrum of Big Data challenges.    Quickly emerging as the world’s newest resource for   competitive advantage, organizations are building   platform-based analytics approaches.

  Your platform needs a combination of a cluster   management, a domain specific application   development expertise and a data science team for a   successful deployment. Some enterprises with strong   in house development skills can corral this. Most others   have skills that are more domain-specific (like retail,   pharmaceuticals, clinical trials, fraud) and cannot handle   the scale and complexity of the deployment or might lack   data science teams. Therefore, the key factors to evaluate   and decide on doing in-house or partnering are:

 • Are you skilled in handling a complex data center   operation? If not, you should look at a managed services   provider either on a cloud infrastructure or on premise.
 • Do you have experience handling a warehouse   infrastructure with associated data management,   reporting, governance and analytics capabilities? If not,   you should get a solution integrator or a service bureau   that has the experience in designing and implementing a   Big Data platform with appropriate tools and governance.
•Ensure you have access to the right set of data and   information discovery tools, as well as visualization tools   that can run on the Big Data platform you are choosing.
 • Do you have quants and data scientists in house? If not,   gather a small team of data scientists that understand   your domain. This can be done in-house or even as a   service via specialist providers.
  • Establish a center of competency around Big Data that   can deploy the right platform with a focus on gaining   actionable insights.

 The biggest challenges towards  successful deployment of Big Data   projects
 The key to Big Data success is thinking long-term.   Companies should build a strategy that focuses not only   on how to start using Big Data, but thinks about how to   grow with Big Data. While you must consider your longterm   strategy, it is best to adopt Big Data and analytics in   small chunks. While Big Data projects can transform the   enterprise, the implementation must proceed in   The first step for companies who want to leverage   Big Data is to identify team members with the skills to   dive into the data, whether it’s through internal searching   or bringing in new, data-skilled workers. Establishing a   small, data-focused team is a good way to build skills,   best practices around platform and analytics, and   application development tools that are relevant to the   industry your company operates in.

Verticals that will benefit the most from  Big Data and Analytics
 Big Data is being adopted across industries. Big Data and   analytics technologies are used in industries as varied as   telco, retail, healthcare, e-commerce, hospitality, the auto   industry, energy and utilities and more.

 For instance, Emory University Hospital is using   streaming analytics technology as part of a research   project aimed at developing advanced, predictive care   for critical patients. The system can identify patterns in   physiological data and instantly alert clinicians of danger   signs in patients within the ICU.

 In the auto industry, PSA Peugeot Citroën is using Big Data and Analytics and mobile solutions to integrate and   analyze the massive amounts of data from cars, phones, traffic signals, lights and other sources to launch a new   era of connected vehicles. The French car manufacturer  plans to offer a range of “connected services” to its clients,   allowing them to use numerous access channels such   as websites, vehicle data, customer service or mobile   applications. Drivers will also be able to access other   sources of vehicle data and customer service information   such as better weather precision via onboard sensors of   temperature, lights, and windshield wipers from Peugeot  Citroen's cars.