Providing continuous, high-quality electrical power is among the most important obligations of the co-location data center owner. Electrical engineers must consider many factors when designing power/electrical systems for these co-location facilities. Issues such as backup, standby, and emergency power systems must be incorporated. Co-location facilities differ slightly in that they may have metered/submetered power systems or power based on occupancy measurements.
Co-location data centers are facilities where data center rack space is rented out to companies that require secure and reliable data center space. They provide secure computing space for companies so that they can spend time on their core business rather than operating data centers. Co-location building owners’ business model requires that they push the envelope of design to remove unnecessary components, increase efficiency, and prioritize design decisions to meet the terms of their contracts. The design engineer for co-location owners must understand these drivers and design a system that supports the co-location business model.
The co-location business is highly competitive, and the decision to build a new co-location facility is largely based on how much it costs to build and operate versus how much the co-location owner can charge its clients for that space. As such, co-location owners know how much the project needs to cost to achieve their return on investment (ROI) and how much land can be used per kilowatt of information technology (IT) space, among other things.
Prior to beginning design, find what the key conditions of satisfaction are to meet the cost metrics. Questions to answer include:
Codes and standards
It is critical that the system designer determine the codes and standards that apply to most co-location designs. Codes and standards that apply directly to the co-location design include:
Mission critical installations:
Service–level agreements (SLA)
When determining the programmatic requirements of a co-location, the first step is to understand the terms of the SLA for the co-location owner. The SLA will define the terms of the contract between the co-location owner and their clients and will include items and metrics that will directly influence the design. The SLA may include items such as:
Clearly, an SLA that allows for no planned outage time per year would result in a drastically different design than one that does. Similarly, if the SLA details where power monitoring will be installed for billing purposes, it will have a direct effect on the location of meters, current transformers, and voltage transformers. Understanding the key contractual metrics between the co-location owner and their clients will assist in making proper design decisions.
Often during the design phase, the co-location owner will not have a customer under contract for the space. When designing the co-location space absent firm client requirements, determine and document the following IT load assumptions:
These answers will help determine the size and quantity of the distribution from the UPS to the data center space.
HVAC/mechanical system design
The HVAC system design is beyond the scope of this article; however, the general rule is that the electrical system design must support the redundancy requirements of the mechanical system. If the mechanical system is designed to be concurrently maintainable, the electrical system that supports it typically is as well.
The mechanical systems may be fed off of the critical systems or a centralized system designed to service the entire co-location facility. Regardless of which approach is used, a careful analysis is required of both the mechanical design and the associated supporting electrical design, to ensure that the required level of redundancy is provided to the co-location.
While there are certainly many design considerations required to develop a safe design, for the electrical designer it is critical to anticipate arc flash mitigation upfront. It is paramount to design systems that eliminate energized work as much as possible, and where it is not possible, to reduce the incident energy.
For elimination of energized work, refer to the section in this article titled “Redundancy requirements and topology.” Experience with arc flash studies indicates that the most common areas for high-incident arc energies are the secondary transformers and the associated line side of the secondary switchgear as shown in Figure 1.
Primary-side overcurrent protection will rarely detect a secondary fault fast enough to significantly lower arc flash incident energy. To mitigate this, make sure to specify equipment that has the capability of mitigating or eliminating the arc flash danger. Recommended approaches include:
Most co-location facilities are part of a large campus development, which will include several separate co-locations. If this is a new site development, then a few basic questions need to be resolved:
Medium-voltage equipment type and topology
This section assumes that the substation at the transmission level is owned by the utility or will be designed by a firm that specializes in that type of design. The different co-location operators have a wide variety of medium-voltage topologies that they employ, but they are generally a variant of two key types.
Loop system: The loop system, as shown in Figure 2, is economical and reduces the overall number of switchgear sections located at the service and reduces the number of feeders from the service switchgear to the data center. This type of system often employs switch fuse construction at the transformer to allow for segmentation of the loop.
Loop systems reduce the breaker count and reduce the number of medium-voltage feeders. The disadvantages of this system include:
Radial feeders with ties: To avoid some of the disadvantages with the loop system, many owners choose to employ a radial system with ties located switchgear as shown in Figure 3. As with the loop-fed system, any one utility feed can be powered down and power will be maintained through the tie breakers. This eliminates the need for complicated switching downstream from the medium-voltage switchgear and allows for reduction of arc flash energy on the secondary side of the transformer. The disadvantages include:
The generator plant is key to the operation of the co-location. Generator plants are so critical to the operation of any data center that the Uptime Institute considers the generator plant the primary source of power and the utility as an economic backup. Generator selection is a separate topic by itself; please refer to Designing Backup, Standby, and Emergency Power in High-Performance Buildings for additional information.
The key factors to determine for data center generator plants are:
Uninterruptable power supplies
UPS technology and its application has evolved significantly, and co-location operators have driven much of this change. UPS in the past had longer-lasting batteries (10 to 15 minutes) and had both a kilowatt and kVA rating (typically 0.9 power factor), lower efficiency below the nameplate rating, smaller frame sizes, and built-in transformers. Most co-location operators are now using smaller UPS batteries (2 to 5 minutes) and either have systems rated at unity power factor or are exploring other technologies, such as rotary UPS.
Additionally, because most UPS are loaded at 40% to 66%, the co-location operators are pushing vendors to supply more efficient systems under part-load conditions. Lastly, the transformer that was often supplied with the UPS is now pushed down to the power distribution unit (PDU).
PDUs typically are electrical devices used to provide distribution in the data center. They include a transformer to step down to the data center rack voltage, secondary breakers for distribution to either racks or remote power panels (RPP), and metering. Like a UPS, PDUs usually are loaded well under their nameplate rating, so co-location owners have pushed improvement in part-load efficiency of PDU transformers. Also like UPS, PDU sizes have increased. PDUs were commonly 225 or 300 kVA in the recent past, but now are commonly 600 kVA or larger.
Co-location facilities will have extensive metering—generally more meters or more precise metering than a standard data center. The extent of the metering will largely be determined by the requirements of the SLA, but at a minimum, the facility will provide metering at:
In addition to the above, additional metering may be required based on the configuration of the system. Additional locations required may include:
Redundancy requirements and topology
The SLA will most likely define the redundancy level required for the design. The two most common redundancy requirements are:
It is important to avoid confusing these with Uptime Institute tiers or TIA 942 levels. It is entirely possible to have a concurrently maintainable system that does not meet Uptime Institute Tier III or TIA Level 3 requirements. Avoid using shorthand terms like “Tier 3 compatible”; they lead to confusion and do not have technical definitions.
Closely tied to the redundancy requirements is the electrical topology. The two most common types that are used are the distributed redundant and reserve “catcher” schemes, refer to Figure 4 and Figure 5, respectively.
In the distributed redundant system, the critical loads are split nearly evenly between systems. When one system fails, the load it is supporting transfers over to the remaining systems. Note that Figure 4 shows only two transformer generator pairs, but is a 4+1 system based on the failure transfers at the static transfer switches. The key advantages to this topology are that equipment is highly used during normal operating conditions and when one system fails, the load transfers are distributed to multiple systems.
For example, assuming that the systems in Figure 4 are fully loaded, the typical load for all five systems will be 80%. However, distributed redundant systems are difficult to expand once they are installed and require careful load tracking by operations. It is easy to assign loads on a distributed redundant system where the overall system is properly loaded, but a single system failure could result in a point overload in the system.
To overcome the shortfalls of the distributed redundant system, many co-location owners have adopted some form of the catcher system, as shown in Figure 5. In this system, the reserve system is normally unloaded and is waiting to accept load from any single failed system. The system in Figure 5 is a 2+1 system for simplicity, but many co-location operators are increasing N to 6 or higher; i.e., 6+1 to maximize the overall use of their equipment. It is relatively easy to increase the value of N in catcher systems; i.e., expansion from a 4+1 system to 5+1. The only additional equipment required is sufficient reserve breakers and the transformer, generator, and low-voltage distribution.
Co-location facility design is rapidly changing as new technology enters the marketplace and as co-location providers work to trim excess equipment out of their designs to lower their cost structure. Designs that were considered standard just a few years ago are continuing to evolve with the changing marketplace. The designs will continue to change, but understanding the fundamental design principles required to complete a basic co-location design will allow the design engineer to adapt to the market.
Power-usage effectiveness (PUE) is still not uniformly applied and referenced consistently across all clients and data centers. If you are provided a PUE number, ask to have the calculation methodology explained to you. For assistance in understanding this metric, please refer to the Green Grid’s White Paper #49, PUE: A Comprehensive Examination of the Metric.
Target power–usage effectiveness (PUE)
PUE is the ratio of the total energy consumed at the co-location over the energy delivered to the servers including power conditioning. In an ideal world, a co-location would have a PUE of 1.0—i.e., 100% of the power would be used to power the IT load. Understanding both the peak and average PUE projected for the facility is critical in determining the final medium- and low-voltage design. The client will give you the IT load, so the majority of the remaining load will be in the mechanical systems.
PUE can be complicated to calculate for co-location facilities and will determine the location of metering. In particular, UPS, cooling, and other facility systems that are shared between clients complicate the metering and the calculation. Submetering may be required for many systems. PUE is used as a metric by both the co-location’s owner and clients, so it is important to ask at the beginning of design where PUE will be measured and how it will be reported.
Some co-location owners will refer to both peak and average PUE numbers, but if possible, avoid using these terms and just use PUE. Note that systems must be sized to the “peak PUE,” so discussions of average PUE are largely marketing language. These two numbers may be very close to each other if the same mechanical cooling method is used throughout the year. However, if the climate allows for free cooling for the majority of the year but relies on other cooling methods during times of high temperature and humidity, the average may be considerably lower than the peak. PUE values vary considerably, ranging from close to 1.1 to 1.8 in some locations. Co-location PUEs are generally lower than those for owner-operated data centers.
Read more about PUE online at www.csemag.com.
Brian P. Martin, PE, Jacobs, Portland, Ore.
Author Bio: Brian P. Martin is a senior electrical technologist in the advanced facilities—electronics department at Jacobs. He is a member of the Consulting-Specifying Engineer editorial advisory board.