How much data is too much?

The amount of data in the world is growing exponentially. By 2020, Cisco estimates there will 40 zettabytes of data in existence. Some of this is raw, unstructured telemetry data, which is partly turned into intelligence that is valuable to operations teams. But how do you decide if the data is past its sell by date? The stark reality is that hoarding data is both inefficient and expensive.

Analyze wisely

We are told time and again that data is a prized commodity, but it is very easy to be big data rich, yet still insight poor, if information isn’t harvested and analyzed efficiently. This is becoming increasingly important as the Internet of Things (IoT) connects more and more devices, churning out even more data.

Mining big data for actionable insight is a major challenge. Searching all your data would be like panning for gold – a long and often thankless task with little or no reward. Ultimately, you need to know what data to keep and what data to discard. This means having the right data scientists and tools in place to make these decisions and rapidly process the data. Yet there are still organizations who spend more on collecting data than analyzing it.

Organizations are aware of the strategic importance of big data and analytics, but there are still hurdles to overcome. According to PwC’s Global Data and Analytics survey, 31 per cent of senior executives say the timeliness of data in their organization is poor, while 25 per cent admit they lack the skills or expertise to make greater use of data.

In addition, 61% of executives acknowledged that their organizations should rely on data-driven analysis more, and intuition less. At the same time, they did not see their organizations as highly data driven, leaving them open to being overtaken by competitors.

PWC says that many organizations are not taking full advantage of the analytics they have or aren’t confident enough to jump to a more sophisticated solution. According to its findings, half are using descriptive or diagnostic approaches. The most sophisticated companies are using predictive approaches, which include enable automatic trading, for example.

The importance of virtualization

An increasingly important tactic to getting the most value out of your data is data virtualization. It allows organizations to link data in different formats across different platforms and protocols. Users can build a visual representation of data patterns that take in large amounts of data quickly. It provides a fast way of spotting trends and exploring data sets to find correlations, for example.

By 2020, analyst firm Gartner forecasts that 35 per cent of enterprises will implement data virtualization in some form as part of their big data strategy. “Enterprise data virtualization has become critical to every organization in overcoming growing data challenges. These platforms deliver faster access to connected data and support self-service and agile data-access capabilities for enterprise architecture pros to drive new business initiatives,” explains Noel Yuhanna, principal analyst at Forrester.

But, it is important to remember that data virtualization is only as accurate as the quality of data being visualized. This means organizations must have people and processes in place to manage big data and where necessary it must be cleaned.

Smart data

Back to the big question: how much data really is too much? The truth is that in a super connected world the data is going to keep coming. There is the argument that the more data a company has the more value can be mined from it. But to make data work, organizations need to take a smart approach. They need to work out which data they need to keep, process and analyze to make smart decisions and what can be jettisoned.