Issues and trends in mission critical infrastructure design, planning and maintenance

In the wake of Sept. 11’s devastation at Ground Zero in New York City, facilities executives are taking stock of mission critical operations. Facility design and construction details are receiving greater scrutiny in light of recent disasters.

Security had traditionally focused on electronic intrusion from viruses and hackers. While electronic intrusion remains a hot priority, corporations also are anxious to protect the physical infrastructure of mission critical operations. Priorities of locations that must operate seven days a week, 24 hours daily include networking technologies, power, HVAC systems, security and safety.

Meeting these priorities requires the expertise found among various members of the facility planning team comprised of facility and risk management, information technology, architecture, engineering, construction and finance, among others.

Corporate protection
Designed to withstand almost anything Mother Nature can blow its way, the Target Technology Center, one of the Midwest’s largest and most sophisticated data centers, is built to withstand high winds and even an F3 tornado. Completed in late 2001, the 115,000-square-foot N+1 facility was designed by Ellerbe Beckett, with assistance from commercial real estate firm Ryan Companies US., Inc. and Target’s information services department, along with Target’s corporate office and consultants.

To withstand the impact of an F3 tornado, the windowless building’s louvers, doors and exterior openings are designed to withstand 200-mph wind speeds. Under such conditions, the building’s brick veneer is designed to peel away, exposing the weather-resistant precast wall behind it.

The roof design had two independent systems. Security is addressed with berms or barricades at points of entry, and cameras monitor critical areas. The office complex security station also monitors the data center around the clock.

The facility is concurrently maintainable — the operations do not have to shut down to perform maintenance activities. And, the building’s design allows for future expansion of electrical and mechanical infrastructure without interruption of Target’s operations.

With 45,000 square feet of raised flooring, major new equipment installations can be assimilated by the pre-wired and pre-bussed underfloor. This plug-and-play approach helps add efficiency to the building and flexibility for future growth.

For small to medium companies with mission critical business applications, there are colocation/hosting sites across the country like the ManagedOps.com Data Center, that provide secure, climate-controlled environment designed to host mission critical business applications. Since the Data Center is a shared facility, the infrastructure investment is spread across many companies, resulting in increased economies of scale and lower costs.

The power to the data center is isolated and conditioned to prevent damage to servers, networks and HVAC equipment. Each server rack has its own dedicated power circuit and UPS unit for uninterrupted operation. If power is out for an extended time, the ManagedOps.com CAT 230KV generator automatically generates power until electrical service is restored.

The data center environment is controlled by redundant HVAC units and is continually monitored by data center personnel. The building is protected by an alarm system monitored by both an alarm company and the police. Anti-static flooring prevents static electricity from damaging data and electrical components. A multi-gigabyte Ethernet backbone and an overhead cabling system provide state-of-the-art speed and reliability. Likewise, external infrastructure elements such as domestic water (for cooling and fire suppression), sewage and access roads must be considered/protected depending on their importance.

Master planning
“The first step is to break your building into sections and look at the mission critical priorities in each,” explains Irene Miess, national sales manager for the FDA-regulated industry for Siemens Building Technologies.

“For instance, in pharmaceutical companies, the archives are critical,” says Miess, “but the office area is less critical. Then you can build your quality and redundancy according to the needs from your master plan.”

Sometimes a portion of the production process also is mission critical, explains Michael Jones, strategic planning director for S&C Electric.

“For years, the mission critical process was the data center, but with personal computers all over the building, mission critical areas are moving beyond the data center to other areas of the building. In computer chip fabrication, one crucial element takes 28 days to make. If power is interrupted during that process, the company could lose most or all production.”

Facility deployment
The recent slowdown in the nation’s economy is not stopping the development of mission critical locations, but it is changing how and where they are being deployed. According to Sriram Sivaram, president of Catalyst Power, a number of companies are taking existing building space and converting it to mission critical operations.

“This is more difficult than building from scratch,” says Sivaram, “because you must take an entire audit of the facility before starting. That means you often need to hire an expert in mission critical engineering to audit the building and see if it is possible to retrofit a segment. The expert can suggest what location is suitable. Often, the most convenient location in a building is not suitable for mission critical operations.”

Sivaram recommends that audits include a full thermal analysis. “In data centers, heat removal often is a problem,” he says. “You might have the right cooling tonnage, for example, but the ducting is not laid out for how the computers will be located within the space.”

Power architecture also plays a role in whether the building can be retrofitted for mission critical operations. Data centers demand from 30 watts to 125 watts of uninterruptible power per square foot, according to Leo Soucy, owner of Facilities Engineering Associates. “The needs of a Fortune-500 mainframe are quite different than Internet Service Provider (ISP) centers,” explains Soucy.

Even with the correct power footprint, the facility may need sophisticated rewiring, says Sivaram.

“Not every electrical contractor has the knowledge that mission critical facilities demand,” says Sivaram. Many buildings have circuit breakers, designed to trip to save lives. As a result, they may be rated for 15 amperes, but they may trip at 8 or 10 amperes, because of their design. But if a circuit breaker trips at a server farm, the whole farm could go down.

“When you recognize that issue, you know you have to test every breaker, check and recalibrate it so that you know exactly at what amperage it will trip,” says Sivaram. “Or, you have to move to fuses, which are a pain in the neck to replace when they fail. That’s why it’s so important to hire an upfront expert to take care of even the finest of details.”

For companies devoting a segment of a building to mission critical operations, redlining the building’s drawings is essential. “Even when you build from scratch, plans often don’t look like the final facility. Parts of the original design may be eliminated during the value engineering process to help control costs,” explains Miess.

Other elements also may be changed to correct for possible weak points found during factory testing. The redlined drawings show facilities executives what they really have in the building — whether it was built or renovated to include a mission critical area.

“It’s important for facilities engineers to agree that the project is complete only when the drawings match the building,” urges Miess. “Otherwise, if you need to isolate rooms because you have a power problem, an air or nitrogen leak or some other issue, you won’t know for sure that the valves you shut off or the power you redirect actually will protect the mission critical area,” says Miess.

“Everything to do with master planning for mission critical operations is about risk. You want your master plan to assess and address those risks appropriately.”

Beyond today
The master plan in 7x24 design goes beyond today’s need for backup generators, uninterruptible power supply (UPS) boxes and battery systems. It also has to look to the future, according to Leonard Ruff, program manager for Callison Architecture.

“You must project your computer needs two years, five years, 10 years from today.” says Ruff, “so that you size your mechanical and electrical capacity to support that ultimate load. You do not have to install all equipment in the beginning, but you have to allow for later additions.

“For example,” he says, “you may project that in 10 years you will need 1,000 kVa of UPS, which you will add in 250 kVa increases. But you also must add corresponding cooling load growth. In other words, you project ultimate use requirements, and then plan the space to add loads as needs increase.”

The question becomes where to put the UPS, particularly in retrofit applications where floor loads may not be able to handle battery weight. Some companies are moving toward locating UPS outside the main building in an outdoor enclosure.

As the need to backup mission critical kilowatts grows, the amperes also grow. This makes mission critical UPS batteries bigger and heavier to support the operations, adding to the building’s floor load. In addition, such UPS batteries are considered a hazardous substance, so they must be isolated in their own specially vented room, which has to be climate controlled, with hydrogen levels monitored.

Locating such batteries outdoors frees building space and allows the UPS to operate on protected circuits without requiring new circuit wiring throughout the existing building. “By going outdoors, you remove some of the cooling load on the HVAC for the building,” says Jones. “These outdoor enclosures have their own heating and cooling units to keep the UPS building in the 60 degree to 90 degree range. That keeps the batteries in their optimum operating range so they last their expected lives.”

Remote sites
For some companies, redundancy includes remote locations “The concept that a natural or manmade disaster would hit both McLean, Va., and San Mateo, Calif., simultaneously is very remote,” observes Bill Mazzetti, Jr., principal of Mazzetti & Associates. “So some companies are storing data, mirroring it and then repeating the storing and mirroring in two completely separate centers. In essence, they are creating four levels of backups to crucial information.”

According to Syska & Hennessy’s OnlineEnvironments, preparedness lies not just in securing a remote site, but also in knowing precisely the people, infrastructure and systems necessary to restore critical functions.

“It may seem like a monumental task to imagine the unthinkable, but there are six essential questions to ask that will identify the key processes and plans that underlie true preparedness,” says Jim McEnteggart, associate partner:

1. Have you rated each facility by how critical it is to supporting business operations?
   
2. Within each facility or data center, have you identified the components most essential to operations?
   
3. What is the threshold for the hours or days that these different operations can be down before essential services must be moved to a new site? Typically, customer call centers need to be back online immediately. In contrast, an internal function such as accounting may be offline longer without a significant impact on operations.
   
4. Are documented plans and processes in place to transition from the damaged facility to the new site? If personnel cannot easily transfer to the remote location, skilled labor should be available nearby to get business back on track.
   
5. Have time limits been set for recreating essential support services? A trader may have phone lines, but he can’t operate without access to market data or customer accounts, for example.
   
6. Are regular drills performed to keep the staff prepared for any disaster? The disaster plan should be tested to see how it would have held up under conditions similar to the tragic events of Sept. 11, 2001.
   

When such backup is crucial, the two 7x24 sites may serve different functions, except in emergencies, says Phil Michel, president, St. John’s Consulting Group. A training center, for example, might serve as the backup to another critical operation.

Some companies, including Microsoft’s data center, also have redundant communications access, with alternate routes and two separate telecommunications carriers — US West and GTE — so that telecommunications can remain connected, even during a disaster. The telecommunications systems, in turn, use redundant land lines, fiber optic or microwave routes to make sure messages get in and out without a glitch.

In addition to equipment redundancy and power reliability issues, mission-critical facilities also must approach fire safety and security differently. For companies with mixed operations — some devoted to mission critical functions — and others less critical, balancing the needs of each is complex.
“Normally, the data floor is well controlled,” says Soucy. “Many companies use card access systems. Some use biometric scanners. Some companies extend this heightened level of security to their utility rooms, as well.”

Sprinkler systems are designed to be very sensitive to fire conditions, and liquid detection systems often are located under or at floor level, according to Soucy.

In 7x24 facilities, ancillary systems may include fire alarm and suppression, pre-act sprinkler systems, incipient fire detection systems, hydrogen alarm and exhaust systems, liquid leak detection systems, and even full supervisory control and data acquisition systems.

Ancillary systems, as well as HVAC and power elements, are factory tested for most mission critical locations. They also undergo rigorous deployment testing before the building is opened to make sure the redundancy and transfer of operations from one chiller to the backup chiller, for example, proceed seamlessly and that both work equally well under utility power and backup generation.

“When Microsoft was moving into its new building, the company put in space heaters to simulate the heat generated by the computers. They plugged into the UPS system to see at what point the heat could cause the UPS to fail,” explains Mazzetti.

The Achilles heel of mission critical design is the human element. A well-commissioned, well-designed mission critical facility is only as good as its weakest link. “Sixty to 70 percent of downtime in such facilities is purely human error,” says Sivaram.

The group maintains a Web site publishes newsletters, holds two national conferences each year and has recognized chapters in Arizona; Atlanta; Delaware Valley; Kansas City; Metro New York; Ohio; Colorado; San Francisco; Seattle; Southern California; and Washington, DC. Work is under way to establish chapters in Boston, Canada, Minnesota and Texas.

The group works with recognized IS/IT, networking and facilities trade publications such as Building Operating Management and collaborates with peer organizations to further enhance dialogue on mission critical reliability issues.

For additional information, visit 7x24 Exchange’s Web site for a list of contacts.