Trade journals are getting an advertising boost with redlined copy that encourages the elimination of information silos. Sounds pretty good, but businesses should keep in mind the saying about things too good to be true. Here are some reasons why in case anyone forgot.

While one vendor’s push to eradicate data silos and islands may lack subtlety, the desire to do so is always present in techies’ heart of hearts, which feeds other vendors and, sometimes, effects business sense. As with all things, there are trade offs, and in this case not all of them are subtle.

The appeal is understandable: Unify everything. The underlying assumption that every piece of data is connected to every other piece of data may be utopian, but the business implications are dramatic. Absolute data centralization has some absolute problems when problems arise, which always do sooner or later.

Separate data repositories, pejoratively called islands and silos, come about for any number of reasons, all relating to convenience and compliance with business practices. The main reason is that business grow and change, with existing information processes expanding and new processes added as new markets, opportunities or requirements present themselves.

Often, the quickest way to accommodate these is to deploy a purpose built process, with little reference to the existing ones because there’s nothing in common and there’s no real logical reason to do so. The main driver for this is the simple fact that existing processes are not related to the new ones.

One good example of this is the completely distinct functions of an accounting system and, say, a Customer Relationship Manager (CRM) system. Another example are the myriad private data repositories that any company has, in Excel, Access, FileMaker and others, that meet very specific and often small scale requirements that take too long to implement in the mainstream business systems, or are too clumsy to use if they are.

Both islands and silos simplify maintenance issues, because their scopes are limited, and they do not have to accommodate disparate requirements beyond and outside of their natural scope, most of which have their own change dynamics. Information management has a power law relationship to size and complexity, similar to Metcalfe’s Law (on network effects), Moore’s Law (increasing processing power at decreasing prices) and Newton’s Laws of Motion. One good example of this is the sheer size of institutional financial and credit databases, which cannot be completely and accurately backed up due to the time it takes to physically move the bits around, during which the databases cannot be taken off line.

Eliminating data islands and silos by having One Big Table database dramatically increases the head count for both operations and maintenance, due in part to the complexity driven by the power law mentioned above, and the fact that expertise in One Big Table crowds out expertise in the various functional components that are encompassed. Simply stated, management of the One Big Table’s intricacies has to be mastered before being able to access the components themselves. 

Not least, the One Big Table system is prone to failure like all systems, sometimes spectacularly like when something breaks, such as a disk subsystem, or more subtly as when information gets jumbled because of incomplete or ambiguous relationship definitions. Then there are the failures driven by a contradiction of the One Big Table’s underlying assumptions, demonstrated aptly by the collapse (in 1998) of Long Term Capital Management’s hedging to infinity and beyond, before the advent of subprime but markedly prescient.

The natural world itself has a pronounced bias in favor of diversity, bias in the sense that more evolutionary changes, driven by ease in adaptation of an existing basis or foundation, increase the chances for long term survival. Which is a fancy way of advising to not putting all one’s eggs in one basket.

Islands and silos do have problems like anything else. Perhaps the biggest one is that their separate data repositories get of sync with their contexts. Another problem is the increasing difficulty to extract meaningful information that relates to other aspects of a business enterprise, to other islands and silos. Both of these problems can be mitigated, if not directly solved, by having well defined methods of extracting tailored information, and adding new pertinent information. There are many ways of accomplishing these, with the easiest being specialized access methods that are small and easily maintained, and use a meta data exchange mechanism such as XML file or data source formats.

Islands and silos may not be sexy or state of the art, but they perform a valuable and ongoing service to the business enterprise. Listen to nature, keep things small and manageable, stay mainstream, and sleep well at night.

A Word From Our Sponser

Stop Toxic Email From The Bad Guys Before It Can Get Close To Your PC! Our  ePAL for email security is a barrier between the outside world and your regular email program

Tags: data islands, data silos, databases, design assumptions, excel, filemaker, private databases, software maintenance

This entry was posted on Tuesday, March 17th, 2009 at 0:50 and is filed under Software That Works. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.