Shadow system

Shadow system is a term used in information services for any application relied upon for business processes that is not under the jurisdiction of a centralized information systems department. That is, the information systems department did not create it, was not aware of it, and does not support it.

An organization that has a centralized Information Services department usually requires rigorous guidelines for developing a new system or application. Simultaneously, with the rise of powerful desktop applications that give savvy end users the ability to author sophisticated tools on their own, a business group often finds it more expedient to create the application themselves.

Any organization faces a multitude of pressures to change and respond to new government regulations, customer demands and action by competitors. In order to respond to these changes, organizations need to be able to understand all aspects of their business and often ask questions of itself that have never been asked before. Ongoing pressure for change creates an ongoing pressure to analyze data in new ways and get information quickly into the hands of people who need it. Only through creative and flexible reporting are businesses able to spot new trends and identify new opportunities rapidly enough to take full advantage of them. The type of data analysis that most frequently necessitates the development or purchase of Shadow Systems usually comes from the needs of the user. Since the centralized information systems department usually reports to the organization’s CFO or COO, the systems that they develop are designed for their needs. The needs of departmental managers are often quite different, requiring more detailed analysis that incorporate variables not contained in the solution designed by the central information systems department.

The greatly increased power of personal computer hardware and software analysis tools has meant that individual users now have all of the computing power they need right in front of them. Large databases containing all of an organization's customer, supplier, or accounting information; the kind that could once only be stored on a central corporate mainframe, can now be contained easily on a single laptop.

Shadow Data Systems often suffer from poor design. Errors may be hard to find, modifications may be difficult, and long term support may be troublesome.

Typically, Shadow Data Systems are only used by one or two people. Unless they are developed by experienced programmers, it may be difficult to scale them up to support tens or hundreds of users.

Shadow Data Systems are often lack adequate documentation. Knowledge about the system is passed by word of mouth and can be confined to a very small number of people. This knowledge is then lost completely if one or two staff members leave.

Around two thirds of the effort involved in professional software development is expended in testing. Shadow Data Systems undergo much more cursory testing and may have latent errors that only become apparent after a long period of production use.

Shadow Data Systems hold substantial chunks of company data and can include confidential information about customers, suppliers or staff. The access control processes for these systems are often much more lax than for a centralized company database and may not even exist at all. Physically locating sensitive data on desktop or laptop computers can leave an organization very exposed if the computer is stolen.

Data in local databases and spreadsheets can very easily be modified, either intentionally or otherwise. Once changed it can be hard to track what changes have been made and what the original data looked like. Where the system manipulates the data it can introduce more subtle errors that remain completely undetected for long periods.

Shadow systems existing on a single computer are often not regularly backed up. It is best to have these systems on a computer that is regularly backed up or on a server.

There may be many different shadow systems within an organization reporting against the same data. Each one may add filters and manipulate the data in different ways. This can lead to apparent inconsistencies in their output. Where two shadow systems disagree, either or both of them may be wrong.

When Shadow Data Systems are created by an experienced programmer or software engineer with significant input from departmental management, the resulting solution frequently exceeds the capability of those created by the organizations centralized information systems department. The experience of the programmer / software engineer easily removes most of the previously stated problems. And, when combined with input of departmental management, the resulting product actually meets the needs of the end user.