Participating in a DR exercise can be quiet challenging. Long hours in windowless rooms with co-workers, little or poor sleep, close quarters, erratic meals, short tempers, all punctuated with technical problems.However, it is a privilege to be asked to help make a DR exercise successful.

This can be a light-hearted day for MIS systems administrators, as the superstition provides an excuse for deferring major activities until another day.

May all your systems be stable, and all your data secure, on Friday the 13th!

I’ve been following the Madoff scandal with some interest. This financial scandal seems to be like a neutron bomb set off inside certain wealthy neighborhoods. It destroyed many lives and fortunes, but left all the buildings intact.

The investigation is in its early days, but some interesting details and speculations are making their way into the press. The really astounding part to me is how Mr. Madoff’s alleged transgressions damaged the lives and reputations of his own sons and nieces.

The story, as currently reported in New York newspapers, is that the Madoff family ran a conventional securities firm, employing hundreds of financial professionals, and engaging in providing conventional financial services. All apparently legal. However, hidden in the inner sanctum, camouflaged by the larger business, was a large hedge fund operated as a Ponzi scheme, defrauding its specific investors. The current speculation is that the Ponzi scheme was either run only by Bernard Madoff, or perhaps only by Bernard and his brother Peter. If true, the Ponzi scheme was run by one or both senior Madoff brothers, without the knowledge of the adult-age Madoff sons and daughter. Having built careers there, the sons and daughter of the senior Madoff brothers can probably expect to never again work in the financial industry. Who of good repute would ever hire someone under such a cloud? How on earth would any parent run such an operation knowing that it would destroy the careers of his sons, nieces, and nephews?

One of the interesting questions about this scandal is the boundary between chutzpah and stupidity. It certainly takes a great deal of confidence to steal from the wealthy and well-connected.   It takes a great deal of chutzpah to circulate amongst the privileged and powerful, taking their acclaims while you are taking their money, knowing that you are not only stealing their money, but also stealing from their charities. I wonder if it felt especially bold to be taking money from Hollywood big-shots in a year where the biggest story in Hollywood touched on the ruthlessness of show business titans when crossed. One of the more recent tidbits from the Madoff stories is speculation that some of the defrauded investors are Russian oligarchs. These are people perhaps likely to respond differently than a Hollywood philanthropist after being fleeced of billions.

When I was a boy, my father gave me a lecture on the “ethical spectrum,” and the differences between activities that are illegal, legal, moral, and ethical. His main point was that one should seek to live one’s life on the highest ethical ground, and not merely seek to live on the slippery slope where rests the blurry line dividing between what is legal and illegal. I wish I had a recording of his actual words to listen to now, but his lecture was academic. He had already taught the lesson by example. I have been blessed to have a number of father figures in my life, who taught by example the lessons of character, trust, dependability, loyalty, sacrifice, and community.

In my own town, we recently laid to rest Mr. Willie Kocurek, who one might depict as the antithesis of Mr. Madoff (and all the other infamous names from Enron, WorldCom, Tyco, etc.) Mr. Kocurek was not widely known outside his home town. However, Mr. Kocurek and his kind form the bedrock of communities across America. Here at home, he earned what I would consider the supreme accolade, perhaps outranking the Medal of Honor as an indicator of selfless devotion in civilian life. Having lived his life an open book, and having his character so understood and appreciated, his community named a grade school in his honor while he still had decades of life ahead of him. Mr. Madoff serves as a classic justification for the very old tradition of not naming streets, ships, and buildings in honor of people while they are still alive. Even an honored elder statesman can succumb to human weakness, amuse the press, and embarrass themselves and their family. Mr. Kocurek had such a depth of character that the entire community was confident that nobody would ever have to explain to children why the namesake of their school was in jail or was the butt of jokes from comedians and columnists.

I hope everyone ensnared in the Madoff calamity finds security, hope, redemption, and justice. I pray that I will always be able to live up to the example of character taught by my father figures, even in times of adversity and temptation.

The CIO is caught in a bind, because a court may generally expect that producing records from a tape is similar to producing papers from a file. However, the CIO’s shop will likely no longer have the right hardware or software for reading the tapes, or the right software for recreating the information from the stream of bytes on the tape. There can be a long technical journey between having some random tapes full of data in backup format, and reproducing an e-mail or a database query from 1992.

Every system administrator knows (and can be summoned to testify in court) that good system administrators will keep a private collection of backups in readiness for the day when a heroic system recovery will be required. Likewise, every database administrator knows (and can likewise testify) that good database administrators will keep a private collection of backups in readiness for that day when a heroic database recovery will be required. (Once while implementing a new backup solution, a DBA told me that he didn’t care if we took hot, cold, daily, weekly, full, or incremental backups of his database.  In a real recovery situation, he planned to restore his database from a private backup on a flashdrive in his desk.) An experienced CIO could be expected to know that these private backup collections exist. To the sysadmins and DBAs, they are a bulwark against Murphy’s Laws. To opposing counsel, they can be treasures from Alladin’s secret cave. To corporate counsel, they are a liability, and to CIOs they are a possible future migraine.

This may be the time for a prudent CIO to establish policies on authorized and unauthorized data repositories, and policies on purging media when applications, systems, or staff are retired, replaced, or re-purposed.

Which is scarier to a CIO: an outsourced sysadmin leaving and taking his private collection of backups with him? Or a sysadmin departing and leaving his private collection of backups on a back shelf to be discovered a year or two later?

These seven (actually eight and sometimes nine) levels are frequently referred to as DR “tiers.” However, the numbering of these DR levels is reversed from the traditional numbering of MIS service level tiers. Level 1 has the lowest DR capability, as well as the lowest cost and complexity. Levels 6 and 7 have the highest DR capabilities, as well as the highest costs.

The seven levels assume that MIS operations take place in a primary data center, with the availability of an alternate data center for use in a disaster. The seven levels concentrate on how data and how application transactions are transferred from the primary data center to the alternate data center. The seven levels describe the technologies and investments needed to provide faster recovery following a disaster. Recovery times are conventionally measured as Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO.)

One of the most important concepts of the Seven Levels is that a specific desired recovery level must be designed into the specific architecture of each application. Achieving faster recovery times requires designing applications and infrastructures using specific technologies and methodologies.

One of the greatest recurring problems in Business Continuity planning is disconnection between DR expectations and the level of investment in DR designs and infrastructures. It is extrememly common for CEOs, CFOs, and business unit managers to have expectations for fast recoveries (usually corresponding to DR Levels 3-5,) while actual infrastructures have been implemented at DR Level 1 based on the available DR budgets.

As presently defined, the seven levels are tied to specific server and storage technologies and recovery methodologies, particularly technologies and methodologies popular in the 1990s and early 2000s. There have been some important new technologies (particularly server and storage virtualization) which are not directly addressed in the seven levels. It is possible that, in the future, the definitions of the existing levels will be more generalized, or there will be a few additional levels defined.

As seen in the following diagram, moving to a higher level provides faster recovery times, but requires a significantly higher investment in hardware, software, telecommunications, facilities, training, services, and management overhead.

The seven tiers have been described as follows:

Tier 0 – No data is preserved off-site. No recovery is expected.

Tier 1 – Data is preserved off-site in backup format.

Tier 2 – Data is preserved off-site in backup format, with a DR hot site available.

Tier 3 – Data is preserved off-site in backup format using electronic vaulting.

Tier 4 – Data is duplicated off-site as point-in-time copies of primary data.

Tier 5 – Applications are customized for transactional integrity across two (or more) data centers.

Tier 6 – Data is duplicated off-site as continuous copy of primary data.

Tier 7 – Data and applications are duplicated off-site, with automated recovery.

Meeting specific recovery requirements is heavily dependent on leadership, organization, documentation, staff skills, and practice exercises. At the end of the day, only DR testing will ensure that any enterprise and infrastructure is capable of meeting specific recovery requirements.

• May you live in interesting times!
• May you earn the attention of those in authority!
• May you find what you are seeking!

Chinese historians tell us that this is an American myth. However, there is no question that we live in interesting times.  This very day’s headlines describe financial, electoral, military, governmental, and ethical crises.  These are proving to be tumultuous, unpredictable times, with uncertain dangers for business, and for MIS operations.  These are the times when one wants to sleep well at night knowing that one has a well-tested plan for business continuity, reliable operations for data protection, and that one has access to the kinds of facilities and equipment needed for rapid response to unforeseen problems.

Having been involved in the resolution of a number of MIS crises, I can assure you that human ingenuity, determination, and collaboration will usually find a solution for almost any kind of crisis. I invite you to a dialog here about modern threats to MIS and about how leadership and teamwork can disarm those threats.

Perhaps these are the times to keep in mind another overused American myth about ancient Chinese wisdom.  The other myth says that the Chinese glyph for the word “crisis” contains within it the glyph for the word “opportunity.”