There’s a lot of data about, and there’ll be a lot more in future, but organisations still have plenty of work to do if they’re to routinely turn big data into valuable business insights.
We live in an increasingly data-driven society, in which information is becoming as much of a currency as money. Many consumers use free services from internet giants like Google, Facebook, Amazon, Microsoft and Apple, for example, and in return allow these corporations to track and monetise their online behaviour.
One of the biggest questions of the day is the openness of such transactions, and the level of control that individuals have over the fate of the personal information they — sometimes unwittingly — divulge to organisations with which they interact online. Recent votes on both sides of the Atlantic have highlighted the capacity for data-savvy organisations to hoover up and profile large amounts of user data — including demographics, consumer behaviour and internet activity — in order to micro-target adverts, news stories and services in support of particular goals or causes.
Clearly, the data floodgates are now opening for businesses of all sizes and descriptions, bringing myriad opportunities for timely analysis in pursuit of competitive advantage. Although the focus is currently slanted towards customer behaviour, data is available at multiple points in the product or service supply chain, and comes in many forms — traditional (structured) , ad hoc (unstructured) , real time, and IoT- or M2M-generated, to name but a few.
Companies that implement big data analytics successfully can reap rich rewards from cost-saving efficiencies and revenue-generating innovations. This can help businesses achieve a digital transformation, allowing them to maintain competitiveness in the face of any disruptive startups — which are data-driven almost by definition — that spring up in their markets.
However, useful business insights don’t automatically flow from a torrent of heterogeneous information: actionable data must be identified, organised and analysed, and the results implemented across relevant parts of the business. That requires planning, budget and the right tools and expertise.
This overview, and the remainder of this ZDNet special report, examines the state of play in big data analytics. We may have passed ‘peak hype’ on the subject — analyst firm Gartner dropped Big Data from its Hype Cycle for Emerging Technologies back in 2015 — but has it yet delivered on its promise?
Attempts are periodically made to estimate how much data is generated worldwide every year, and in what form. Back in 2014, IDC and EMC put the ‘ Digital Universe ‘ at 4.4 zettabytes (ZB) in 2013 — that’s 4.4 trillion gigabytes — and predicted this would grow to 44ZB in 2020, more than doubling every two years. The latest estimate, from IDC and Seagate’s Data Age 2025 report, puts the 2025 figure (now dubbed the ‘Global Datasphere’) at 163ZB — a tenfold rise from the 16.1ZB created in 2016.
The IDC/Seagate report also predicts that the bulk of worldwide data creation will shift from consumers to enterprises, the latter accounting for 60 percent by 2025. Trends driving this shift, according to the report, include: the evolution of data from business background to life-critical; embedded systems and the IoT; cognitive/AI systems that change the landscape; mobile and real-time data; and security as a critical foundation.
All that data needs a home, either permanent or temporary, which explains the interest of a storage company like Seagate in this area.
In a statement launching the report, Seagate CEO Steve Luczo (soon to become Executive Chairman) said: “While we can see from this new research that the era of Big Data is upon us, the value of data is really not in the ‘known’, but in the ‘unknown’ where we are vastly underestimating the potentials today. What is really exciting are the analytics, the new businesses, the new thinking and new ecosystems from industries like robotics and machine-to-machine learning, and their profound social and economic impact on our society. The opportunity for today’s enterprises and tomorrow’s entrepreneurs to capture the value of data is tremendous, and our global business leaders will be exploring these opportunities for decades to come.”
Faced with mind-boggling quantities of data, CxOs might be forgiven for feeling overwhelmed. But, of course, not all data is suitable or available for analysis. In the Data Age 2025 report, for example, IDC estimates that by 2025 some 20 percent of the data in the global datasphere will be critical to our daily lives, and 10 percent of that will be ‘hypercritical’:
The report notes that: “The emergence of hypercritical data must compel businesses to develop and deploy data capture, analytics, and infrastructure that delivers extremely high reliability, bandwidth, and availability; more secure systems; new business practices; and even new legal infrastructures to mitigate exposure to shifting and potentially debilitating liabilities.”
AI and machine learning will increasingly be involved in big data analysis, which further restricts the amount of available data. In the Data Age 2025 report, IDC estimates that by the end of 2025 only 15 percent of the data in the global datasphere will be tagged — and therefore suitable for AI/ML analysis — and only 20 percent of that (3% of the total) will actually be analysed by cognitive systems:
At the turn of each year, experts in a variety of tech fields offer their summaries of current trends and make predictions for the next 12 months. Big data is no exception, and we’ve collated multiple 2017 contributions, assigning predictions to a range of emergent categories. Here’s how a sample of the pundit community viewed the big data landscape as 2017 got underway:
For big data industry-watchers, the most influential area for 2017 is ‘AI, machine learning, automation & cognitive systems’. Analyst firm Ovum, for example, suggests that “Machine learning is the big disruptor” and that “Analytic applications embedding machine learning are becoming the norm”. Increasing levels of automation are almost an inevitable requirement if organisations are to avoid drowning in data — or, as Enterra Systems puts it: “Artificial intelligence will grow in importance as data volume increases”.
The second-placed recurrent theme for big data experts is the emergence of ‘Data-driven business applications’ (also a key theme for this ZDNet special report) . Oracle puts it succinctly by noting that “Applications, not just analytics, propel big data adoption”, while Gartner predicts that “Data and analytics will drive modern business operations, and not simply reflect their performance”.
Other widely-cited trends and predictions for 2017 concern ‘Informatics, data science & data engineering’, ‘Big data proliferation & governance’ and ‘Cloud-based analytics & integrated data services’.
