Managers Discuss Data Center Maintenance Challenges
OTHER PARTS OF THIS ARTICLEPt. 1: This Page
The role of maintenance and engineering managers in the day-to-day operations of data centers and mission-critical facilities is becoming more critical to organizations. The toughest challenge facing managers with institutional and commercial facilities is ensuring these areas remain operational and reliable around the clock.
How has your department's involvement with data centers changed over the years?
AMICO: We've gone from no involvement to engineering design and daily monitoring and support. In the past 17-plus years, I have seen data centers range in size from information technology (IT) closets to greater than 1,000-square-foot rooms. Each facility I provide support to has a dedicated data room that we routinely inspect. The larger, more critical rooms are supplied with remote-monitoring capabilities.
KAMAU: As technology progresses, we increase the development of our in-house applications in order to automate a lot of our processes. This creates a need to have proper backup and recovery systems in case of a failure. A higher level of technological skills must be utilized in the correct places to ensure a smooth flow of operations in case something fails.
TURNER: As the security requirements for server rooms and data centers have increased over the years, access to them has become extremely restricted. The county's maintenance team no longer has direct access. Facilities personnel are no longer the first to be contacted when emergency alarms are activated from inside a data center. IT personnel are at the top of the first responder call list. IT responders call in facilities maintenance to address certain type of systems failures.
What power monitoring and metering strategies does your department use for data centers?
AMICO: We monitor temperature, humidity and voltage with remote alarming capability, as well as a 24/7 switchboard. Should a parameter go out of band, a remote page or e-mail shall be sent to on-call personnel to address the issues.
KAMAU: Not all building and departments are neatly metered, so we must rely on a combination of meters and department square footage to assess use. We also add a multiplier for spaces that contain more equipment and use more power. Since a data center uses more power than an average facility, they get an appropriate multiplier added to compensate so that usage statistics are more realistic.
TURNER: Power-monitoring strategies are limited to manual recording of electricity consumption in each facility. Separate metering of data centers and fiber optic head-ends isn't currently being done. Funding approval for such things has been difficult to come by in the last five-six years due to budget restrictions. We acknowledge the value and utility of such systems and plan to continue to press for funding to acquire them.
How have technology changes affected maintenance and monitoring of backup generators and uninterruptible power supplies (UPS)?
AMICO: With the increased dependency on electronic data over the past 12 years or so years, I have had to install additional monitoring points to the building automation systems to monitor data-center temperature, humidity, power supply voltage for primary and redundant power sources and, where utilized, UPS and emergency-generator status through remote monitoring in the case of a fault. When a parameter goes out of band, a message is sent via remote paging or e-mail to contact on-call personnel.
KAMAU: Technology has helped eliminate the amount of downtime that incurred in the past during a power outage. The UPS system kicks in immediately and allows enough time for the backup generator to power on. We get notified as soon as that happens. This has really helped with protecting our equipment and data, particularly as more equipment and systems become dependent on technology and people become more dependent on the data they generate.
What is the data center's impact on your HVAC system?
AMICO: Luckily, every data center that I provide support to has its own dedicated or supplemental HVAC system, with critical centers having HVAC on emergency generator power. However, not all HVAC systems are on emergency power.
KAMAU: Our primary data center has no impact on the HVAC system as it has its own standalone chillers and circulation system that the Office of Information Technology (OIT) is responsible for maintaining. However, our secondary data center rooms have very little equipment and thus it hardly impacts the HVAC system. However, as additional equipment is added, we get notified so as to ensure that our HVAC capacity can handle the increased heat load.
TURNER: Data centers, fiber optic head-ends, and larger server rooms have dedicated HVAC systems. The climate in those areas is controlled even when power is offline in the rest of the building.
Where does improving data center efficiency rank on your list of priorities?
AMICO: It has a high importance, given the types of data that resides on server and network systems. Some areas don't have the needed redundancy that they should and require contingency planning in case of a power or HVAC failure.
KAMAU: It's at the top of our list, as most of our in-house production applications run from there. We're always looking for better ways to be more energy-efficient with our data centers. For example, our primary data center managed by the OIT uses a combination of hot- and cold-aisle containment layout, which is intended to prevent cool exhaust air from mixing within the server room. Facilities management constantly monitors changes in the HVAC industry to take advantage of more efficient equipment, energy-reduction techniques, and rightsizing of systems.
TURNER: Until we have the capability to capture the utilities consumption of individual data centers, we'll continue to face an uphill challenge in proving that green equipment choices are worth the additional first cost. Given the long string of budget reductions in recent years, gaining approval to spend some of our limited resources on sustainable upgrades in our data centers in the absence of hard data to back up our assertions isn't likely to gain any traction.
What kinds of training challenges do you face regarding data center maintenance?
AMICO: My challenge has been the reassurance with IT staff that all will go well when it's time to perform maintenance on data center HVAC equipment or power supply.
KAMAU: Staffing with the appropriate expertise is our main challenge. The recession caused massive cuts to our budget, which reduced our ability to staff with properly credentialed full-time employees for these positions, so we've had to rely on IT students to help support our technology needs. Though inexpensive to hire, they require large amounts of training, which usually occurs under fire. Their schedules are also erratic because we must work around their school schedule, which causes us to schedule maintenance and repair work accordingly. On the bright side, we are enhancing their education and preparing them for real-world situations. Having this kind of work experience should help them land great jobs after graduation.
TURNER: With the recent string of budget cuts, our training budget has been reduced to that which is required for staff to maintain licenses and certifications to continue to do their jobs. We participate in as many free training opportunities as we can — teleconferences, webinars, and videoconferences that are usually provided by our vendors and equipment manufacturers.