The OpEx Dilemma

The myriad advantages of deploying Linux on today’s commodity-based X86 servers — superior flexibility, scalability and cost-effectiveness — remains a driving factor in the success of open architecture designs in the data center. But those efficiencies are but one part of the overall data center equation, and highest ROI can only be achieved with careful attention to the post-purchase operations of those systems.

The operational challenges of today’s data centers have
become a gloomy issue for IT managers and CIOs around the globe. As
hundreds of thousands of Linux- based
x86 servers are deployed to keep pace with
ever-growing business computing needs, data center managers are
faced with increasing operational costs, namely power, cooling, and
maintenance. Today, a data center spends just 30 percent of its
budget on capital expenditures, while a whopping 70 percent is
consumed by operational expenses.

The natural shift away from antiquated RISC/proprietary,
Unix- based servers and the movement toward
high performance, open architecture, open source solutions has
enabled countless data centers to proportionally reduce capital
expenditures while increasing overall system performance. But while
the price/performance ratio has improved, the cost to operate and
maintain all those servers has spiraled out of control.

Linux continues to take hold as a strategic operational
advantage in the data center, largely due to key benefits such as
administrator skill portability, vendor diversity, and the wide
range of supported hardware providers. The myriad advantages of
deploying Linux on today’s commodity-based X86 servers
— superior flexibility, scalability and cost-effectiveness
— remains a driving factor in the success of open
architecture designs in the data center. But those efficiencies are
but one part of the overall data center equation and highest ROI
can only be achieved with careful attention to the post-purchase
operations of those systems.

According to industry analysts, power and cooling have become a
leading operational expense for the data center, second only to IT
payroll costs. Power expenditure is linked to two basic activities:
the movement of electricity from outside data center walls to power
computer systems, and the amount of power consumed by HVAC
infrastructure to evacuate heat expelled by those systems.

Today’s x86 server designs have enabled dramatic
improvements in density, with Linux-based, rack-mount systems
packing more, ever-faster processors into a cabinet. Advancements
such as half-depth form factor and back-to-back mounting enable
density levels as high as 80 1U rack-mount servers per cabinet. The
newest multi-core processors from AMD and Intel are enabling
never-before-seen CPU core densities, and the introduction of
quad-core computing will increase compute levels to an
unprecedented 640 processing cores in a cabinet just 7-feet tall
and a little more than 3-feet deep.

Density levels that high, with all those processing cores in
such a small footprint, is a dream-come-true for Linux data
centers, resulting in exponentially greater compute power in a
small amount of space. (For example, see Figure
) But higher density comes with a price: a very high
price in the form of power, cooling and maintenance. Thus, while
delivering the flexibility and scalability that leading Internet
and High Performance Computing (HPC) environments are clamoring
for, dense server clusters raise power consumption levels and
amplify thermal management challenges.

With data centers restricted by how much power can physically be
delivered to the site, it isn’t uncommon for a data center to
simply run out of power capacity long before running out of space.
Similarly, a data center’s HVAC system is limited in how many
BTUs of cooling power can be delivered. In public co-location
facilities, a fixed, predetermined level of wattage and heat
dissipation is assigned per rack, making this an even greater
challenge for companies hosting their servers in a public data

Given that data centers have among the highest density of
energy-consuming equipment of any modern building and use 100 times
the electricity of a typical office building on a per square foot
basis, the issue of consumption continues to make headlines. It is
estimated that for every 100 watts used to power a server,
typically an additional 60-70 watts of power is required to cool

So how will a data center supply enough electricity to power row
after row of server cabinets each populated with today’s
fastest CPUs? Tack on the cooling costs, and the data center will
simply run out of power.

According to a recent study by Pacific Gas& Electric, a
California data center using standard CRAC units to for cooling
spends 54 percent of its overall power consumption on the HVAC
infrastructure, with just 38 percent of power used to run the
servers themselves.

The problem is viewed to be so pervasive that through year-end
2008, Gartner predicts that cooling requirements for servers will
prevent 90 percent of enterprise data centers from achieving
maximum theoretical server density.

With the cost of power rising yearly, currently as high as
$.12/KWh in some United States states, the vicious cycle of power
consumption and cooling costs cannot be ignored.

A Look Inside the Chassis

One of the most important steps in minimizing power consumption
in the data center is to carefully specify what goes inside the
servers. Component selection has a major impact on overall system
power draw, since each and every component not only requires power
to function, but also exhausts heat.

At a basic level, systems should contain only what the IT
administrators specify, with no extras, such as unused CD-ROM or
floppy drives, unnecessary motherboard components, or overspecified
memory. While it seems obvious to buy systems that fit a given
power envelope and are customized to a data center’s needs,
some x86 providers force pre-configured choices to the end user,
leaving little room for customization or the elimination of
unnecessary parts.

For example, a 250-watt power supply would be appropriate for a
server with a 200-watt requirement, yet some manufacturers may use
a power supply of up to 500 watts to satisfy the same 200-watt
need. That would result in poor power efficiency given that typical
power supplies function most efficiently when driven at a high
percentage of their maximum rating. While a difference of tens of
watts per server is not significant in small installations, in
large deployments it drives up power consumption considerably, as
well as the amount of heat that’s dissipated.

Once the proper’ recipe’ for a specific server
configuration has been identified to match performance and power
requirements, the next step is to consider ways to reduce power
consumption by choosing high efficiency, lower wattage components.
This becomes especially important when selecting processors and
power supplies.

CPUs and AC power supplies draw more power than any other
components inside a system. Subsequently, they give off significant
amounts of heat and are generally the first parts to fail inside a
server. The processor drives a server’s overall power draw
requirement; the fastest processors consume the most power and
generate the most heat.

Today’s lower wattage, dual-core CPUs from AMD and Intel
continue to achieve high levels of performance with decreased power
consumption. Higher CPU clock speeds will require higher wattage
draws, which may be necessary in HPC and other compute-intensive
environments. Regardless of what CPU is selected, the system should
fit within the data center’s so-called’ energy
footprint’ and systems should be designed to be cooled

Power supplies are quite possibly the most problematic element
of any computing device. Traditional servers use AC-DC power
supplies, designed to convert AC power into the DC voltages
required by the system components. Most AC power supplies operate
at roughly 75 percent efficiency (on a good day), meaning that 25
percent of all the energy consumed by a server is wasted
(subsequently converted to heat) within the power supply itself.
Typically, however, these power supplies operate at between 30 and
60 percentof the full load, which drops the efficiency to around 50
to 65 percent.

In terms of reliability, MTBF ratings for traditional AC power
supplies hover around a mere 100,000 hours, largely due to the
cooling fans that ironically decrease their reliability by
contributing moving parts that are most likely to fail. With a 10
degree Celsius rise in a system’s internal temperature
halving component life, this means that any strategies that
increase power efficiency not only reduce power costs but also
significantly increase system reliability.

Recent advancements in AC power supply technology have increased
efficiency to upwards of 85 percent or higher, which can translate
to an immediate power savings of hundreds of thousands of dollars
per year in a large Linux data center deployment.

With every percentage efficiency gain, hard dollar savings are
realized, through both the reduction of power consumption and the
subsequent increase in component reliability that comes from lower
heat output. Making the initial investment in the more efficient AC
power supply is ultimately worth every penny over the lifespan of
the server.

But what if the AC-DC power supply could be eliminated
altogether in favor of a more efficient, reliable solution?

Introducing DC Power

In 2003, the first distributed DC power technologies became
available for large-scale data center server deployments. Long a
standard in the telecommunications industry, DC power options
provide a more efficient power standard, eliminating the need for
wasteful AC-DC power conversion inside a traditional AC power
supply at the system level.

By replacing a standard AC power supply with a 93 percent
efficient DC power card inside each server, power losses at the
server level are instantly reduced. Instead of converting AC power
to DC power inside the server, the conversion happens at the
cabinet-level via redundant AC-to-DC rectifiers. By converting the
power within the external rectifiers — which have an MTBF of
250,000 hours apiece — 20-40 percent less heat is dissipated
within the servers themselves. The subsequent efficiency gain
results in power savings while increasing overall system

In this rectified DC scenario, AC power is brought to the server
cabinet through standard power distribution mechanisms. Rectifiers
within the cabinet, which take up a mere 2U of rack space in a 44U
cabinet populated with 80 systems mounted back-to-back, distribute
redundant DC power to each system via DC bus bars housed inside the
cabinet. (See Figure Two. Power savings of
10 percent and higher are achieved with rectified DC solutions,
which can amount to millions of dollars in savings each year alone
for a large-scale Web farm.

While the power savings achieved using DC solutions are
certainly compelling, the real operational advantages are in the
dramatic increase in system reliability. Higher MTBF in the DC
power cards, coupled with a 20-40 percent reduction in heat inside
the chassis, means higher uptime and reduced maintenance costs
associated with system failure. Built-in redundancy at the cabinet
level ensures system stability in the event of a power failure on a
given circuit. And because rectified DC solutions can easily be
deployed in any existing AC-based data center, no costly
infrastructure changes are required.

Reducing Losses from the Doorstep to the

While carefully specifying low wattage server components such as
high efficiency CPUs and DC power cards can reduce power
consumption at both the server and cabinet levels, a tremendous
amount of power is incrementally lost from the moment it arrives at
the data center’s doorstep to the time it reaches the server

In a typical AC power path, there are four major points of
efficiency loss as electricity moves from the power grid to the
servers themselves. Efficiency losses begin when power moves from
the grid through a transformer (98 percent efficient), through an
Uninterruptible Power Supply (UPS, 74-94 percent efficient), into a
Power Distribution Unit (PDU; 98 percent efficient), and finally
into the AC power supply inside a server (71-75 percent efficient).
Total efficiency for this power path is a mere 39-63 percent. With
upwards of 40% of power wasted simply in the AC-to-DC conversion
process, it’s no wonder that today’s data centers face
insurmountable power bills.

New technologies enable a far more efficient power path for data
centers, increasing overall power efficiency to levels as high as
79 percent for the data center as a whole. By leveraging
large-scale rectifiers and exterior DC power converter units,
AC-to-DC conversion is reduced to just two points on the electrical
path, culminating in data center-wide DC power distribution
directly to DC-based servers.

The increasing popularity of easily deployed rectified DC
solutions and the even greater efficiencies achieved via
data-center level DC power distribution have prompted IT leaders to
consider DC as the new data center standard. New,” green field”
data centers are increasingly being built with DC input power in
lieu of AC to achieve the highest possible efficiency and lowest
possible power loss from the doorstep to the servers.

Keeping It Cool

Reducing power consumption at both the server and data center
levels clearly impacts operational expenditures. Reduced power
consumption means lower power bills and less heat at the component
level. Less heat in the chassis means greater system reliability
and lifespan. And greater reliability means less human capital and
money wasted on maintenance and repair.

But after all that attention on lower wattage components and
better power distribution techniques, heat will naturally be a
by-product of any machine, and remains the number one killer of
systems in a data center. Effective heat evacuation helps reduce
the burden on HVAC infrastructure while increasing server life

Traditional data centers rely upon a” hot aisle/cold aisle”
layout, where standard-depth, rack-mount systems draw in cool air
in the front and expel hot air out the back of the rack. Expelling
heat back into the data center environment naturally warms the air
and taxes existing HVAC infrastructure to adequately cool the data
center. Moreover, if not positioned properly, hot air expelled from
these traditional AC servers can make its way into a neighboring
row of servers, subsequently increasing the ambient temperature of
those systems as well as the likelihood of failure.

With cooling costs typically calculated in ton-hours of heat
removed, it’s not uncommon for an electric air conditioner to
draw 1.3KW/ton. With HVAC expenses accounting for such a
significant portion of power costs, improved heat evacuation can
become a very strategic maneuver in the quest to reduce operational

One best practice is achieved through the use of half-depth
servers mounted back-to-back inside the cabinet. Back-to-back
mounting not only improves serviceability by positioning I/O, ports
and connectors on the easily accessible front of each server, but
it also creates a natural plenum-or chimney-inside the cabinet.
Fans inside each server and carefully placed louvers built into the
cabinet direct air flow upwards through the plenum, evacuating heat
out the top. This eliminates both the hot aisle/cold aisle
challenge as well the issue of re-circulating hot air exhaust into
the data center, enabling more efficient evacuation of heat
directly into the HVAC infrastructure.

Some x86 data centers have designed special shrouds that seal
the top of the cabinet and connect directly to HVAC ducting,
enabling the movement of heat directly out of the cabinet into the
cooling framework without ever entering the data center. By
changing the direction of the fans and louvers, heat can also be
directed downward-toward the floor-and evacuated through in-floor
cooling devices. Some data centers leveraging these unique heat
evacuation techniques actually re-circulate the carefully contained
hot air to warm office space during the winter months.

Server providers have gone to great lengths to develop novel
cooling techniques. The industry has seen recent introductions in
the form of liquid cooling at the component level, and even
rack-level cooling via water pipes. However, the most practical
cooling strategies begin inside the server with careful component
selection and power considerations, and end with the way servers
are racked and heat is evacuated at the cabinet-level.

Keep It Cool

The data center, the site of computing innovation for four
decades, is continually challenged by ever-increasing operational
costs driven by business demands for increased compute performance.
With the rapid expansion of rack-mount, x86 systems, many data
center managers are struggling to survive within their existing
power envelope, while keeping data center labor costs to a

Facilities built as little as two years ago are ill-equipped to
hold today’s high-density server configurations, largely due
to inherent limitations in their power distribution and cooling
infrastructure. Thermal management issues have become more than
just a facilities challenge. Today, they very visibly impact the
bottom line in any budget.

With these key drivers in mind, reducing power consumption,
lowering heat output and increasing system reliability have become
key factors for IT purchasing decisions. Every data center needs to
consider ways to optimize existing infrastructure and streamline

*Consider the
power draw of the system.
Do your servers contain any
unnecessary, extraneous components (all of which, of course, draw
power and expel heat)? Has your server vendor selected low wattage
components that fit within your energy footprint? Do you have
enough power per square foot in your energy footprint to support a
fully-populated cabinet of servers or even a fully-populated data

*Choose wisely
when it comes to power supplies and CPUs.
Were your systems
built with the most efficient power supply to meet your needs? Did
you specify the lowest possible wattage in the processor without
compromising performance?

*Calculate the
losses associated with getting power to the servers.
you considered using rectified DC- or even direct DC-based systems
in your AC data center? How can you reduce efficiency losses at
every possible point along the distribution path?

forget about cooling.
Is heat being evacuated in the most
effective way possible? What cabinet-level designs will help
maximize density while reduce the amount of heat that pollutes your
data center?

*Finally, think
about your people.
How much of your team’s time is
spent maintaining overheating systems? What level of frustration do
your people experience as a result of operational inefficiencies?
As your top operational expense, how can you optimize your data
center to reduce the cost of human capital?

The operational challenges of today’s and tomorrow’s
data centers are not getting any simpler. As energy consumption and
its associated costs continue to rise, IT organizations need to
focus on eliminating every operational bottleneck in the data
center environment.

The clear advantages of Linux and today’s innovative, open
architecture X86 server designs will continue to impact their
widespread adoption in the world’s leading data centers, but
careful consideration must be given to the growing post-purchase
challenges as the sheer volume of deployments continues to

Linux data centers squarely focused on solutions that reduce
power, cooling and maintenance costs will reap the benefits time
and time again.

Comments are closed.