Big data is a major technology trend for middle market companies. The technology offers major opportunities, although not without significant work. That's because big data brings big data sets — the large collections of information that make the techniques and tools worth the trouble of investing in.
However, a middle market company often doesn't have the extensive IT resources of a larger business. Executives cannot necessarily delegate a big data program and all the concerns of the data sets it involves. Although you won't necessarily undertake these five steps by yourself, you should understand what must happen so you can keep from being taken by surprise if returns from big data are suddenly small.
Know what you want. Big data can easily turn into a trolling operation: You sit in a boat on a river, dropping a baited hook here and there. But unlike fishing, when you can tell relatively quickly that nothing is biting, you have to examine the data sets to see if they contain something useful. That can burn through a lot of resources. If you know the type of information you're looking for in the first place, it's much easier and faster to tell whether some data might hold something of interest.
Know where the data sets are and what's in them. You can decide what information you want in the first place, but that does little good if you don't know where to find it. Your company should document the various sources of data it has, whether in formal databases or in a spreadsheet on someone's computer. The documentation should include where on the network the information lies, a description of the data's structure (for example, the fields, or parts, of database records), and a general indication of the type of information included. The people responsible for the data and its maintenance, with their contact information, should be included. If the information is held externally, whether on a cloud service or by a third party, like a government that has open data, someone should be able to tell how to get it and what it contains.
Know how to translate them. Not all data are the same. In addition to the structure of the data in the documentation, you'll need to understand the concepts behind it because different parts of a company might use the same terms differently. A logistics group might use a first in/first out approach to inventory even as the financial department uses average inventory value. That could mean two different costs of goods, depending on whose definition you use. Things get even more slippery when the data comes from different organizations. Someone will have to translate between sources, and that means at a conceptual level, not merely a technical one.
Know what to throw out. If you wanted to explore a new type of market or product, looking at historic sales would do little good. Socioeconomic profiles of an area are a waste to analyze if you neither have nor plan distribution there. Data can be enchanting. But data sets, like the Sirens of Homer's Odyssey, can lure you to crash on the rocks. Learning what data will have no impact and which you should throw away can be tricky.
Know the applicable law. Data privacy legislation has grown in scope, reach, and force because of the many examples of companies losing control over sensitive consumer data. Depending on the country, or even state, you may have to meet different requirements of storing and safeguarding data. You might even need to get explicit permission to use personal information in some types of analysis.
Some of these steps require specific technical expertise. Others rely on intricate legal knowledge, insight into industry practices and terminology, or strategy. Making big data sets work for you is a complex process that you need to manage, or else you could find your data-fueled decisions heading off into the wrong direction.
How does your midsize company use big data?
Erik Sherman is an NCMM contributor and author whose work has appeared in such publications as The Wall Street Journal The New York Times Magazine Newsweek, the Financial Times Chief Executive Inc., and Fortune. He also blogs for CBS MoneyWatch. Sherman has extensive experience in corporate communications consulting and is the author or co-author of 10 books. Follow him on Twitter and circle him on Google+.