New eBusiness applications demand 100% availability, along with scalable, reliable access to enterprise data. But corporate databases are becoming so large that just maintaining traditional service levels is almost impossible. Because larger databases take more time to load, unload, search, reorganize, index and optimize, the performance of mission-critical applications often deteriorates when database size increases relentlessly. Response times get longer. Getting to decision-making information becomes more difficult. Service levels decrease.

In the past, the response to such prolific database growth was to upgrade CPU MIPS and acquire more disk storage. Faster, more powerful processors sped access to information, while the underlying databases continued to grow. However, this approach is rapidly losing its viability because MIPS upgrades have become exceedingly expensive, and they only solve linear growth. With the advent of eBusiness and the Internet, the stress on performance has increased exponentially, not linearly, thereby making additional MIPS an ineffective solution. Furthermore, the corporate data explosion has squeezed backup/restore and reorganization windows to the point where data availability is seriously threatened.

Forward-looking companies are now using a new technology — intelligent archiving — to address the fundamental issue of runaway database growth. Mission critical databases are streamlined by transferring outdated historical data to an archive. These data archives can be quickly and easily researched, and data can be restored if necessary. In the meantime, both performance and availability of production data is greatly improved. In addition, planned increases in MIPS capacity can be deferred, often saving millions of dollars.

There are significant technical challenges involved in archiving production data. This is particularly true when the data is stored in a relational DBMS such as DB2. Often, data has been normalized across hundreds of tables that are interconnected by at least that many relationships. Many relationships are "application-managed" and thus are not directly supervised by DB2. Moreover, many companies fear that historical data, once it is no longer in the active database, may be difficult to locate and restore.

Some IT organizations have attempted to build and maintain customized data archiving systems. Historically, these efforts have been set aside, after the magnitude of the undertaking began to impede other high priority programs. Creating an internal archiving solution diverts highly skilled IT resources who could be contributing to a revenue producing project, rather than infrastructure development. Not only must the project staff write and debug software for extracting 100% accurate subsets of data into an archive, they must also design and develop a means for researching and restoring archived data. Then there is the problem of managing a growing number of archive files. In addition, the software must be customized for every new application and every change to the production data model.

When faced with the prospect of devoting increasing budget and staff to a never-ending project of maintaining, supporting and enhancing an internal archiving solution, most CIOs realize that a generalized archiving solution is the more prudent strategy.

Enthusiasm for intelligent archiving is spreading, largely because it is one of those rare "dream technologies" that is conceptually simple yet incredibly powerful in what it achieves — the simultaneous improvement of performance and major cost reductions. But, IT organizations need to carefully choose the right solution for archiving, particularly when dealing with complex enterprise databases such as DB2. A complete archiving solution will streamline databases and operate inline with normal database maintenance, while providing a method for tracking archival operations and a safe, efficient way to browse and retrieve selected data from the archives, when necessary. To gain the most benefit from an archiving solution, the following best-of-breed criteria should be considered:

• With dozens or hundreds of normalized tables in the database, the solution must precisely offload not only the right data, but data that is referentially intact and complete, including those many relationships in the database that are managed by the application.
• After data is archived, the solution must have the ability to accurately delete only the correct data, leaving the other production data intact.
• When old data needs to be found, the solution must have the capability to search the archives quickly and easily, especially for situations where no one knows which archive file contains the data.
• The solution must allow data to be "selectively restored" from archive files, rather than reloading entire sets of archived data.
• When restoring data, the solution must ensure that all related rows that may span dozens or even hundreds of tables, are restored correctly.
• The solution must be flexible enough to accommodate changes to the production data model that occur between the time that the data has been archived and the time it needs to be restored.

A best-of-breed archiving solution, when it is properly implemented, addresses a critical operational need for organizations with large, complex DB2 databases. Old data can be moved to archives in a precise, scientific manner. Databases can be optimized for peak performance and precious MIPS are no longer used to search through tables and indexes that are bloated with unused historical data. Archives can be browsed and data can be selectively restored as needed. Moreover, organizations eliminate the additional expense of creating one-time programs whenever data needs to be archived and restored.

With an intelligent archiving solution, a company can maximize its investment in both its applications and its operational platform:

• Archive files can be stored on the most cost-effective medium.
• MIPS upgrades — with their attendant astronomical hardware and software license fees — can be deferred, and savings realized.
• The organization can keep production DASD expenses at low levels.
• At the same time, service levels are kept very high because production database searches and response times are minimized.
• Availability is improved because batch-processing windows are shortened and less time is required to backup and reorganize data.

In the long run, implementing a generalized archiving solution is the most cost-effective course of action. The fundamental problem — database growth — is directly addressed, while internal resources are not unduly diverted from the business to write, debug and maintain complex custom archive and restore tools. With a truly generalized, robust product, future applications and databases can be merged into an existing archiving strategy, and a full set of tools is provided for all of the attendant support operations — archival tracking, research activities and selective data restoration.

Stephen J. Gerrard is Vice President, Strategic Planning at Princeton Softech (www.princetonsoftech.com), a subsidiary of Computer Horizons Corp. (NASDAQ: CHRZ) that provides enabling software for streamlining application performance and testing of eBusiness and mission-critical solutions. Mr. Gerrard is a 30-year veteran of the IT industry, and is a frequent speaker at industry conferences in North America, Europe, and Asia. He has authored numerous papers on the topics of relational data management and application development.