Information overload: prevent your company from being buried alive.

Paul Bradley, Apollo Data Technologies

Due to the economics of technology, corporations can collect and store information on nearly every interaction with their clients, with their suppliers, and on their internal operations. According to a Forrester Brief in 2001, Global 3500 enterprises spent on average $664,000 annually on database systems. Unfortunately, the ability for companies to effectively use this collected information to support strategic business decisions lags behind their ability to collect and store it. But if a company can leverage their data for decision support, they are more likely to have a competitive edge in their market sector.

At the same time, the very economics that allow corporations to collect and store details on interactions with clients, suppliers and their own processes make it more difficult to leverage this data asset to strategically improve their business and their operations.

Processing speed and storage capability

“Moore’s Law” was predicted in 1964 by Gordon Moore who later went on to co-found Intel. The law states, roughly, that computer processing power doubles every 18 months, for a fixed cost. As a result of this 'law,' we can easily purchase computers today with processing power that dwarfed the machines available even 5 years ago.

But computer storage manufacturers have out-paced their colleagues who build processors. The amount of computer storage that can be purchased for a fixed cost doubles, roughly, every 9 months.

The means of accumulating information and storing it far outstrip a company’s capacity to process, sift through, analyze. and utilize this data asset and apply it for competitive business advantage. As time passes, the gap between being able to store data (as measured by the amount of storage a corporation can obtain) and process, prepare, and analyze this data (as measured by the amount of processing power a corporation can obtain), grows at an exponential rate. If a corporation does not put processes in place to close this gap, information overload will continue. Their data assets quickly become data “tombs” – the data is simply stored and never sees the light of day.

The graph below displays the growing gap between the increasing ability to store data and processing speed. This gap between highlights the increasing amount of data generated and stored by corporations.







Experts who design and build data storage and transformation systems that efficiently support high-level strategic analytic initiatives have played the primary role in closing the ever-increasing gap between a corporation’s data and its strategic decision makers. These experts provide strategic business decision makers with solid, supportable data and trends that translate into improved customer and supplier relationship management and more efficient internal processes. These improvements will ultimately lead to improved competitive advantage.

Some forward thinking companies already are seeing their long-term investments pay dividends as they can now target their marketing and promotional efforts with unparalleled precision.

But, most aren't. That's where the challenge comes in.

The data game: the challenges businesses face

The challenge is simple: how to extract the valuable knowledge from data, quickly and effectively.

Significant progress has been made over the last three decades by academic researchers in database technology, statistics and machine learning. This work created data mining – a set of techniques and methodologies that can efficiently extract patterns and trends from large datasets. By focusing these techniques and methodologies on a corporation’s strategic business initiatives, data mining technologies and processes move into mainstream business practice. When applied correctly, data mining technologies and processes effectively close the growing gap between the business decision maker and their data assets and help corporations make informed, actionable decisions.

Historically, in-house technology staffs have been the only ones with clear access to collected data. Statistical analysts used complex statistical packages to do sophisticated manipulation and interpretation of the data. For corporations with the resources to support an IT staff and statistical analytic staff, data mining technology resided with these groups. Even in these situations, the ability to apply trends and patterns extracted from data for improved marketing, pricing, or targeting can be hampered due to political an other resource constraints.

The good news is that in the next few years, more companies will put systems in place to benchmark the ways they measure and disseminate data. According to a recent study by The META Group, by 2005 most companies will have adopted scorecards for tracking this information. By 2007, leading organizations will undergo regular information audits.

Business intelligence initiatives continue to receive higher priority as companies realize the value of using their collected data to support strategic business decisions. According to a January, 2004 Gartner study, the total business intelligence software market will have a compound annual growth rate of 8.5 percent by 2007. This has created a market opportunity for database and data mining experts to apply their skills to build the technological infrastructures needed to effectively and efficiently support the data and analytic needs of corporate strategic decision-makers.

An October, 2003 Gartner study also found that Information democracy, corporate performance management and business activity monitoring are driving mass business intelligence adoption. But, only about 35 percent of Global 3000 companies are aware that they need to maximize data. Another 45 percent of businesses operate on "reactive" data management mentality, used only in high-level strategic decisions and not spread throughout the entire business.

That paradigm is clearly shifting.

About the author

Paul Bradley, Ph.D., Principal, Data Mining Technologies, directs specification, design and solution evaluation for data analysis problems. Prior to CO-founding Apollo Data Technologies, Paul consulted on data mining algorithm integration with Microsoft Research and SQL Server, and led data analysis solution implementations for a number of Microsoft divisions. Earlier, Paul was data mining development lead at digiMine, Inc., where he focused on integrating data mining technology into the company’s service offering. Prior to digiMine, Paul was a researcher in the Data Management, Exploration and Mining Group at Microsoft Research, where he helped develop new data mining algorithms and shipped data mining components in products such as SQL Server and Commerce Server. Paul earned his Ph.D. and MS degrees in computer science and BS in mathematics from the University of Wisconsin.  www.apollodatatech.com