Debra Vieira, CH2M
Over the past several years, mission critical clients seem to be asking the same series of questions regarding data center designs. These questions relate to the best distribution system and best level of redundancy, the correct generator rating to use, whether solar power can be used in a data center, and more. The answer to these questions is “It depends,” which really doesn’t help address the root of their questions. For every one of these topics, an entire white paper can be written to highlight the attributes and deficiencies, and in many cases, white papers are currently available. However, sometimes a simple and concise overview is what is required rather than an in-depth analysis. The following are the most common questions that this CH2M office has received along with a concise overview.
What is the Best System Topology?
There isn’t a single “best” system topology. There is only the best topology for an individual data center end user. The electrical distribution system for a data center can be configured in multiple topologies. While the options and suboptions can be myriad, the following topologies are commonly deployed (see Figure 1).
The above topologies assume a low-voltage UPS installation. However, similar systems can be developed using a medium-voltage UPS. Beyond the redundancy configuration, these low-voltage UPS topologies also can be evaluated on ease-of-load management, backup power generation, their ability to deploy and commission initially and when expanding, first costs and total cost of ownership, physical footprint of the equipment comprising the topology, and time to construct the initial installation as well as expansion of the system.
A commonality between the different topologies presented is the need to transfer load between systems. No matter the system topology, the requirement to transfer load between electrical systems—either for planned maintenance activities, expansions, or failure modes—must be done. Load management refers to how the load is managed across multiple systems.
2N topology. The premise behind a 2N system is that there are two occurrences of each piece of critical electrical equipment to allow the failure or maintenance of any one piece without impacting the overall operation of the data center IT equipment. This configuration has a number of impacts:
Distributed redundant (3M2) topology. The premise behind a 3M2 system is that there are three independent paths for power to flow, each path designed to run at approximately 66.7% of its rated capacity and at 100% during a failure or maintenance event. This configuration is realized by carefully assigning load such that the failover is properly distributed among the remaining systems.
This configuration has a number of impacts to the distribution:
N+1 shared redundant (N+1 SR). The premise behind the N+1 SR system is that each IT block is supported by one primary path. In the event of maintenance or a failure, there is a redundant but shared module that provides backup support. The shared module in this topology has the same equipment capacities and configuration as the primary power system, minimizing the types of equipment to maintain.
For example, if six IT blocks are to be installed, then seven distribution systems (substations, generators, and UPS) will need to be installed for an N+1 system. This N+1 system can easily be reconfigured to an N+2 system with minimal impact (procuring eight systems in lieu of seven). This reconfiguration would allow the system to provide full reserve capacity even while a system is being maintained.
This configuration has several impacts to consider:
N+1 common bus (N+1 CB). The premise behind the N+1 CB system is there is one primary path that supports each IT block. This path also has an N+1 capacity UPS to facilitate maintenance and function in the event of a UPS failure. The system is backed up by a simple transfer switch system with a backup generator.
This configuration has a number of impacts on the distribution:
The above topology descriptions only highlight a few systems. There are other topologies and multiple variations on these topologies. There isn’t a ranking system for topologies; one isn’t better than another. Each topology has pros and cons that must be weighed against the performance, budget, schedule, and the ultimate function of each data center.
What Generator Rating Should be Used for a Data Center?
Generators need to be able to deliver backup power for an unknown number of hours when utility power is unavailable. To help select the appropriate generator, manufacturers have developed ratings for engine-generators to meet load and run time requirements under different conditions. The International Standards Organization (ISO) Standard 8528-2005, Reciprocating Internal Combustion Engine Driven Alternating Current Generating Sets, tries to provide consistency across manufacturers. However, the ISO standard only defines the minimum requirements. If the generator is capable of a higher performance, then the manufacturer can determine the listed rating. To complicate generator ratings even more, some industries have their own ratings specific to that industry and application. These various ratings can make selecting the correct generator type complicated.
There are four ratings defined by ISO-8528:
The generator industry also has two additional ratings that are not defined by ISO-8528: mission critical standby and standby. Mission critical standby allows for an 85% load factor with only 5% of the run time at the nameplate rating. A standby-rated generator can provide the nameplate rating for the duration of an outage assuming a load factor of 70% and a maximum run time of 500 hours/year.
Data center designs assume a constant load and worst-case ambient temperatures. This does not reflect real-world operation and results in overbuilt and excess equipment. Furthermore, it is unrealistic to expect 100% load for 100% of the operating hours, as the generator typically requires maintenance and oil changes after every 500 hours of run time. Realistically during a long outage, the ambient temperature will fluctuate below the maximum design temperature. Similarly, the load in a data center is not constant. Based on research performed by Caterpillar, real-world data center applications show an inherent variability in loads. This variability in both loads and ambient temperatures allows manufacturers to state that a standby-rated generator will provide nameplate power for the duration of the outage and it’s appropriate for a data center application. However, if an end user truly desires an unlimited number of run hours, then a standby-rated generator is not the appropriate choice.
What Type of Transformer is Best?
The type of transformer to be used for a data center is constantly questioned and challenged by end users trying to understand if they should invest in a high-performance transformer or not. There are two categories of distribution transformers: dry-type and liquid-filled. Within each category, there are several different types. The dry-type category can be subdivided into five categories with the following features:
For liquid-filled transformers, various types of fluids can be used to insulate and cool the transformers. These include less-flammable fluids, nonflammable fluids, mineral oil, and Askarel.
When put into the context of a mission critical environment, two transformers stand out: the cast-coil transformer due to its exceptional performance and the less-flammable liquid-immersed transformer due to its dependability and longevity in commercial and industrial environments. While both transformer types are appropriate for a data center, each comes with pros and cons that require evaluation for the specific environment.
Liquid-filled transformers are more efficient than cast coil. Because air is the basic cooling and insulating system for cast coil transformers, they will be larger than liquid-filled units of the same voltage and capacity. When operating at the same current, more material and more core and coil imply higher losses for cast coil. Liquid-filled transformers have the additional cooling and insulating properties associated with the oil-and-paper systems and tend to have lower losses than corresponding cast coil units.
Liquid-filled transformers have an average lifespan of 25 to 35 years. The average lifespan of a cast coil transformer is 15 to 25 years. Because liquid-filled transformers last longer than cast coil, they save on material, labor to replace, and operational impact due to replacement.
Recommended annual maintenance for a cast coil transformer consists of inspection, infrared examination of bolted connections, and vacuuming of grills and coils to maintain adequate cooling. Most times, cleaning of the grills and coils requires the transformer to be de-energized, which often leads to this maintenance procedure being skipped. The buildup of material on the transformer grills and coils can lead to decreased transformer efficiency due to decreased airflow.
Maintenance for a liquid-filled transformer consists of drawing and analyzing an oil sample. The oil analysis provides an accurate assessment of the transformer condition and allows for a scheduled repair or replacement rather than an unforeseen failure. This kind of assessment is not possible on a cast coil transformer. Additionally, omitting the oil sampling does not decrease the transformer efficiency.
Cast-coil-type transformers have a history of catastrophic failures within data centers due to switching induced transient voltages when switched by upstream vacuum breakers. There has been significant research by IEEE committees, which resulted in guidelines for mitigating techniques (i.e., resistive-capacitive [RC] snubbers) published in IEEE C57.142-2010: IEEE Guide to Describe the Occurrence and Mitigation of Switching Transients Induced by Transformers, Switching Device, and System Interaction. Liquid-filled transformers seem less susceptible to this problem, as there is no published data on their failure. Regardless of the transformer type installed, best industry practice is to perform a switching transient study and install RC snubbers on the systems if warranted.
When a transformer fails, a decision must be made on whether to repair or replace it. Cast coil transformers typically are not repairable; they must be replaced. However, there are a few companies who are building recyclable cast coil transformers. On the other hand, in most cases, liquid-filled transformers can be repaired or rewound.
When a cast-coil transformer fails, the entire winding is rendered useless because it is encapsulated in epoxy resin. Because of the construction, the materials are difficult and expensive to recycle. Liquid-filled transformers are easily recycled after they’ve reached the end of their useful life. The steel, copper, and aluminum can be recycled.
Cast-coil transformers have a higher operating sound level than liquid-filled transformers. Typical cast coil transformers operate in the 64 to 70 dB range while liquid-filled transformers operate in the 58 to 63 dB range. A decibel is a logarithmic function and sound pressure doubles for every 3-dB increase.
Liquid-filled transformers have less material for the core and coil and use highly effective oil and paper cooling systems, which allow them to be small in physical footprint and weigh less than the corresponding cast coil unit. Because cast coil transformers are air-cooled, they are often larger than their liquid counterparts assuming the same voltage and capacity (kVA rating). Cast coil transformers have more core material, which implies higher costs and losses.
Dry-type transformers have the advantage of being easy to install with fire-resistant and environmental benefits. Liquid-filled transformers have the distinct disadvantage of requiring fluid containment. However, advances in insulating fluids, such as Envirotemp FR3 by Cargill, a natural ester derived from renewable vegetable oil, is reducing the advantages of dry-type transformers.
For indoor installations of transformers, cast coil must be located in a transformer room with minimum 1-hour fire-resistant construction in accordance with NFPA 70-2017: National Electrical Code (NEC) Article 450.21(B). However, if less-flammable liquid-insulated transformers are installed indoors, they are permitted in an area that is protected by an automatic fire-extinguishing system and has a liquid-confinement area in accordance with NEC Article 450.23.
Traditionally less-flammable liquid-filled transformers are installed outdoors. However, both types can be installed outdoors. This option of outdoor installation has the additional advantage of reducing data center cooling requirements. In this case, cast coil transformers need to have a weatherproof enclosure and cannot be located within 12 in. of combustible building materials per NEC Article 450.22. The liquid-filled transformer must be physically separated from doors, windows, and similar building openings in accordance with NEC Article 450.27.
The choice between a cast coil and a less-flammable liquid-filled transformer can be a challenging one to make. A liquid-filled transformer is a solid choice for a data center application because it is more efficient, physically smaller and lighter, quieter, recyclable, and has a longer lifespan. However, if the demand for high electrical and mechanical performance is of the utmost concern, then cast coil would be the appropriate choice.
What IT Distribution Voltage Should be Used?
By now it’s well understood in the data center industry that 3-phase circuits can provide more power to the IT cabinet than a single-phase circuit. However, the choice of distribution voltage between 208 Y/120 V or 415 Y/240 V depends on the answers to several questions, such as:
Let’s start with the power of a 3-phase circuit. A 208 Y/120 V, 3-phase, 20-amp circuit can power up to a 5.7-kVA cabinet. Per NEC Article 210.20, branch-circuit breakers can be used up to 80% of their rating, assuming it’s not a 100%-rated device. Therefore, a 208 V, 3-phase, 20-amp circuit can power a cabinet up to 5.7 kVA (20 amps x 0.8 x √3 x 208 V). Now, if that same 20-amp circuit was operating at 415 Y/240 V, 3-phase, then that circuit could power a cabinet up to 11.5 kVA (20 amps x 0.8 x √3 x 415 V). That’s more than twice the power from the same circuit for no extra distribution cost.
If the specifications for the IT equipment can be tightly controlled, the decision to standardize on 415 Y/240 V distribution is a pretty simple one. However, if the IT environment cannot be tightly controlled, the decision is more challenging. Currently, most IT power supplies have a wide range of operating voltage, from 110 V to 240 V. This allows the equipment to be powered from numerous voltage options while only having to change the plug configuration to the power supply. However, legacy equipment or specialized IT equipment may have very precise voltage requirements, thereby not allowing for operation at the higher 240 V level. To address this problem, both 208 Y/120 V and 415 Y/120 V can be deployed within a data center, but this is rarely done as it creates confusion for deployment of IT equipment.
The follow-on question typically asked is if the entire data center can run at 415 V, rather than bringing in 480 V and having the energy loss associated with the transformation to 415 V. While technically feasible, the equipment costs are high because standard HVAC motors operate at 480 V. Use of 415 V for HVAC would require specially wound motors, thus increasing the cost of the HVAC equipment.
Must We Install an Emergency Power-off System?
Emergency power-off (EPO) buttons are the fear of every data center operator. With the push of a button, the entire data center power and cooling can be shut down. Because of the devastation that activation of an EPO can cause, EPOs typically are designed with a two- or three-step activation process, such as lifting a cover and pressing the button or having two EPO buttons that must be activated simultaneously. These multistep options assume that the authority having jurisdiction has provided approval for such a design. However, EPOs are not necessarily required. The need for an EPO is typically triggered by NEC Article 645.10, which allows alternative and significantly relaxed wiring methods in comparison with the requirements of Chapter 3 and Articles 708, 725, and 770. These relaxed wiring methods are allowed in exchange for adding an EPO system and ensuring separation of the IT equipment’s HVAC occupancies from other occupancies. The principle benefit of using Article 645.10 is to allow more flexible wiring methods in the plenum spaces and raised floors. However, if the wiring is compliant with Chapter 3 and Articles 708, 725, and 770, the EPO is not required.
Can We Use Photovoltaic Systems to Power Our Data Center?
Corporations and data center investors are demanding sustainability be built into the data center. The positive impact on public relations by showcasing a sustainable data center can’t be underestimated, especially considering how much of a power hog data centers can be. Additionally, many utility companies will offer incentives for the use of energy-efficient and sustainable technologies. An often-questioned item is whether photovoltaic systems can be used to meet some of the sustainability requirements in a data center environment. The answer is yes, but a good understanding of photovoltaic systems and the limitations and impacts on a data center are required prior to making the investment (see Figure 4).
The power production of photovoltaic equipment varies considerably depending on type and location of the system installed. There are three main types of solar panel technologies. Crystalline silicon (c-Si) is the most common photovoltaic array type, along with thin-film and concentrating photovoltaic. Thin-film is generally less efficient than c-Si, but also less expensive. Concentrating photovoltaic arrays use lenses and mirrors to reflect concentrated solar energy onto high-efficiency cells. Concentrating photovoltaic arrays require direct sunlight and tracking systems to be most effective and are typically used by utility companies.
Solar cells are not 100% efficient. In the infrared region of light, solar cells are too weak to create electricity; and in the ultraviolet region of light, solar cells create heat instead of electricity. The amount of power that can be generated with a photovoltaic array also varies due to the average sunshine (insolation, or the delivery of solar radiation to the earth’s surface) along with the temperature and wind. Typically, photovoltaic arrays are rated at 77˚F, allowing them to perform better in cold rather than in hot climates. As temperatures rise above 77˚F, the array output decays (the amount of decay varies by type of system). Ultimately, what this means is that the power generation of an array can vary over the course of a day and year. Added to this are the inefficiencies of the inverter, and if used, storage batteries.
The physical space required to install the photovoltaic array can be significant. A simple rule is to assume 100,000 sq ft (about 2.5 acres) for a 1-MW photovoltaic-generating plant. However, this does not include the space required for access or other ground-mounted appurtenances. The total land required is better estimated at about 4 acres per MW. This estimate assumes a traditional c-Si photovoltaic array (without trackers). Increase this area by 30%, for a total of about 6 acres, if thin-film technology (without trackers) is used due to the inefficiencies of the technology.
A photovoltaic system may or may not provide power during a utility power failure, depending on the type of inverter installed. A standard grid-tied inverter will disconnect the photovoltaic system from the distribution system to prevent islanding. The inverter will reconnect when utility power is available. An interactive inverter will remain connected to the distribution system, but it is designed to only produce power when connected to an external power source of the correct frequency and voltage (i.e., it will come online under generator power). Typically, interactive inverters include batteries to carry the system through power outages, therefore the system should be designed such that there is enough photovoltaic-array capacity to supply the load and charge the batteries.
Most data centers do not have the necessary land to install a photovoltaic system, which substantially offsets the power demand. Then there is the question of what happens when the photovoltaic system is generating low or no power. Interactive inverters and deep-cycle storage batteries can be installed to cover these low-photovoltaic production periods, but this introduces new equipment, maintenance, and space requirements into the data center, thus creating more costs and more maintenance than may have been originally envisioned. Generally, data center sustainability is addressed more directly through efficient cooling and electrical distribution systems. Sustainability achieved through solar power, while nice to have, is generally not the focus of data center investments.
The trend is to provide a photovoltaic system that offsets some of the noncritical-administration power usage. These systems are typically small (less than 500 kW) and can be located on building rooftops, carports, and on the ground. They use a standard grid-tied inverter connected through the administration electrical distribution system, which ultimately ties into the site-distribution switchgear where the utility meter resides. A grid-tied inverter system will disconnect from the utility if there is a failure or when on generator power.
Because the grid-tied inverter connection is downstream from the utility revenue meter, a billing mechanism known as net metering generally is used. With net metering, owners are credited for any electricity they may add to the grid when the photovoltaic production is greater than the site usage. Although in most data centers, the critical load dwarfs the noncritical load; therefore it’s rare that a photovoltaic system would generate power on the grid. There are differences between states and utility companies regarding the implementation, regulations, and incentives for net metering. Furthermore, there are some utility companies that perceive net metering as lost revenue and will not allow connection to their system.
A great resource for photovoltaic and renewable energy in general is the National Renewable Energy Laboratory. The NREL website provides information on photovoltaic research, applications, publications as well as a free online tool, photovoltaic Watts, which estimates the energy production and cost of energy of grid-tied photovoltaic systems throughout the world. The photovoltaic Watts tool easily develops estimates of potential photovoltaic-installation performance.
A medium-voltage Alternative to Low-voltage UPS
Design topology evaluation also should consider the medium-voltage uninterruptible power supply (UPS). Like the topologies using the low-voltage UPS, the medium-voltage UPS can be deployed in 2N, N+1, and 3N/2 configurations. Regardless of the topology used, medium-voltage UPS systems offer advantages over low-voltage UPS systems. They generally are installed outdoors in containers, thereby minimizing the conditioned building footprint. While not required, medium-voltage UPS topologies are typically used for full-facility protection rather than using independent information technology and mechanical-cooling UPS systems, further reducing the building footprint. Medium-voltage UPS systems are large systems, starting at 2.5 MVA and scalable up to 20 MVA per UPS. Different manufacturers have different voltage offerings, but medium-voltage UPS systems can range from 5 kV up to 25 kV, with medium-voltage diesel rotary UPS systems going as high as 34.5 kV.
In early 2018, Michigan State University is expected to complete construction on a new 25,000-sq-ft data center with 10,600 sq ft of server space and initially hosting about 300 server racks. This $46 million facility will use a medium-voltage UPS system, starting with 2.5 MW of critical power and the ability to increase in critical power as needed. The utility infrastructure is built to support an increase of load up to 10 MW. Figures 5, 6, and 7 highlight the outdoor power electronic switch, switchgear, and the medium-voltage UPS installed by the university.
Debra Vieira is a senior electrical engineer at CH2M, Portland, Ore., with more than 20 years of experience for industrial, municipal, commercial, educational, and military clients globally.