The Big Data Paradox

Download PDF

By Don Creswell, SmartOrg

The advent of the digital age has made it possible for people to collect more data than ever before. We are more information driven than ever before in human history. Where at one time, data gathering was a practice exclusive to government and business; it has bled into our personal lives too. The most efficient family households today are managed by spreadsheets for the sake of time and chore management. You would think excess of data was a good thing, right? Well, it should be. However, it is so easy to get overwhelmed by the sheer amount of information being gathered that it becomes impossible to read (never mind actually analyze) the reports, tables and other knowledge that is literally a mouse-click away.

Big DataWelcome to the phenomenon known as The Big Data Paradox. The sheer magnitude of the Internet and its abilities to research and gather data, and the technologies that have been developed to manage it, such as digital spreadsheets, have made information more important to us than ever before. Unfortunately, that magnitude has also made it harder to use because we are constantly being bombarded with the latest facts and figures.

Every business and organization gathers every nanobyte of information that exists in the universe about its industry, competitors, and potential clients. We collect data on our own business, clients, sales numbers, profits, losses, and predictions. We research, gather, compile and collect. We start spreadsheets and folders, and we dedicate entire drives of our company’s servers to hold the data we aggregate. But what do we do with all of it once we have it? All too quickly, the information overwhelms us. Not unlike the stacks of paper that used to cover our desks and fill the hallways of our offices, if we aren’t careful about how we organize the information we receive, it all becomes useless as it gets lost amid the mountains of data we keep receiving.

Data management companies were born out of the need to better manage all this data. Systems and networks were developed to break the information down into smaller chunks, making it more accessible to the end user. The paradox occurred when even those smaller chunks of information were too big for the end user to digest. Managers continue to wade through data that is irrelevant to their work to find what is relevant.

SmartOrg wants to be a part of the solution to make data more manageable and useful to decision makers. We understand how easy it is for data to get overwhelming for the end user. We created Rangal to try to prevent this from happening. No matter how much data you have, sooner or later it must be refined so that key findings become manageable and contribute to good decision making. We want big data to become small data because decisions for businesses can’t be made based on terabytes of unstructured information. Ultimately, the goal is to continue to streamline data management so that it is easier to navigate, research and analyze.