Model 7965-S42 water cooling manifold (Feature codes ECR3 and ECR4)

Overview

The 7965-S42 water cooling manifold provides water supply and water return for 1 - 20 servers that are mounted in a 7965-S42 42U slim rack. The manifold is mounted on the right side of the rack (when viewed from the rear of the rack) and extends for 40U. Power distribution unit (PDU) pockets on the right (when viewed from the rear) are not accessible and cannot be populated in the water-cooled configuration. The manifold does not interfere with the placement of servers or other I/O drawers. Quick connect fittings are located every 2U on the manifold for water supply and return that provides 20 pairs of fittings.

Requirements

You must obtain a cooling distribution unit (CDU) and water that meets cleanliness, filtration, and chemical requirements that are listed in Planning for water cooling.

CDUs are available from suppliers such as Motivair and Nortek. CDUs deliver water at a proper flow rate and temperature to cool the servers, while it maintains the temperature above the dew point to avoid condensation. CDUs are also essential to control the closed loop of the water that flows through the servers to maintain proper water cleanliness, filtration, and chemistry while it controls the wet materials in the loop.

Configurations

The manifold can be used only with a 7965-S42 rack. The water cooling feature of IBM servers cannot be configured in a non-IBM rack.

The manifold can be used only in the rack to cool servers, or it can be used with a rear door heat exchanger (RDHx) in series flow to cool the remaining heat that is transferred to the air. For more information about RDHx information, see Model 1164-95X rear door heat exchanger.

Manifold only configurations

FC ECR3 can be used to order the manifold with water input and output at the top of the rack. Since the hose exits through the top of the rack, the top 2U must be left vacant. All 2U drawers must be populated into the rack on odd EIA increments.

Figure 1. Rack and manifold with supply and return hoses exiting through the top of the rack

FC ECR4 can be used to order the manifold with water input and output at the bottom of the rack. Since the hose exits through bottom of the rack, some space must be left open on the bottom. When 1U of bottom space is left open, all 2U drawers must be populated into the rack on even EIA increments. When 2U of bottom space is left open, all 2U drawers must be populated at odd EIA increments.

Figure 2. Rack and manifold with supply and return hoses exiting through the bottom of the rack

Top hose exit locations

The following graphics show the location of the hose exiting from the top of the 7965-S42 rack.

Figure 3. Top hose exit locations

Figure 4. Top hose exit locations (when viewed from the top)

Bottom hose exit locations

The following graphics show the floor cutout locations and dimensions that are required for water hoses that are routed out through the bottom of the rack and under the floor. Power cables can also use this cutout.

Figure 5. Bottom hose exit locations

Figure 6. Bottom hose exit locations (when viewed from the bottom)

Specifications

The rear door heat exchanger can be used with this rack. For more information about rear door heat exchangers, see Model 1164-95X rear door heat exchanger specifications.

Hoses

The servers are connected to the manifold through quick-connects. The manifold has one cold water inlet that leads to the rack and one warm water outlet. Hoses are supplied by IBM. Hoses can be cut to length, but first must be cleaned so that no particles are inside the hose before installation. Some slack must be kept in the hose for easier installation. For more information about recommended clamping tools and specifications, see the Oetiker website.

The manifold has 185.4 cm (73 in.) inner diameter flexible hoses on the supply side and 185.4 cm (73 in.) inner diameter flexible hoses on the return side. After accounting for hose length inside of the rack, there is approximately 91.4 cm (3 ft) of hose for each of the supply and return hose that exits the rack.

Figure 7. Hose kit

The customer provided interconnection to the rack (under-the-floor manifold, CDU, and so on) must have a 25.4 mm (1 in.) female NPT fitting for each manifold supply and return connection. The 25.4 mm (1 in.) NPT male barb fitting from the hose kit must be threaded onto the NPT female fitting on the customer CDU plumbing. A thread sealant must be used to create a leak-free connection. Teflon tape cannot be used, as Teflon tape particles might enter the water stream.

To make the connection from the hose to the barb fitting, the facility hoses must first be cut to length. If the CDU plumbing fitting requires a longer hose than the 426.72 cm (14 ft) facility hose, the plumbing must be altered to bring the fitting close enough such that the 426.72 cm (14 ft) facility hose is sufficient. The end of the hose must be cleaned so that no particles are inside the hose before installation. The clamp is inserted over the hose and then the barb is inserted into the hose. Position the clamp near the hex portion of the fitting (not over the barb) and tighten the clamp with an Oetiker clamp tool. For more information about the clamp tool, see Forged steel Standard Jaw Pincers. The ears of the clamp must be tightened so that they make contact with each other. When the clamp is released, the ears relax and a small space is left between. This small space is normal. Figure 8 shows dimension s that must be fully closed during the clamping process.

Figure 8. Clamp ear

The supply and return labels must be applied on both ends of the facility hose to indicate the function of each hose. The quick connects on the end of the facility hose can now be connected to the quick connects on the manifold. For more information about mating quick connects, see step 3 at Replacing the water manifold in the 8335-GTW or 8335-GTX system.

The following graphics show the CDU schematics of facility hose connections.

Figure 9. CDU schematic with facility hoses that connects directly to the CDU

Figure 10. CDU schematic with facility hoses that connects to an under-the-floor manifold

Water flow

Water does not flow through the manifold unless the supply and return circuits are connected. This situation normally occurs when a water-cooled server is connected to the manifold. An option to flow water through the manifold before servers are connected is to attach the bleed tool that is included along with the manifold. If you want to commission the system before the servers are connected, the bleed tool can be used to allow water to flow. With the bleed tool connected, a limited amount of water (7.5 - 11.3 liters (2 - 3 gallons) per minute per rack) can flow through the manifold. It is advised to continue to have the water flow until the servers arrive to avoid stagnant water from sitting in the manifold for an extended time. For instructions on how to connect the bleed tool, see Bleeding air from the manifold.

Initial filling of the rack and servers with water

Figure 11. Opening the cap in the bleed tool

Figure 12. Attaching the bleed tool to the top of the rack

Bleeding air from the manifold

Air must be bled from the system when the servers are first connected and when the water starts to flow. This step must be repeated if more servers are added to the rack, or if a new cold plate is introduced to the water loop, such as a cold plate FRU replacement.

To bleed the air from the loop, follow the procedure regarding the initial filling of the loop. Ensure that you connect the two quick connects on the ends of the hoses of the bleed tool to the top-most quick connects on the supply and return manifolds. If possible, temporarily increase the water flow rate to the rack during the bleeding process up to an average of 5.6 liters (1.5 gallons) per minute per drawer to help free bubbles from internal surfaces. Never increase the flow rate to the point that the inlet pressure to the rack exceeds 6894 pascals (40 pounds per square inch (psi)).

The bleed valve must be left in place for as long as practical. The more slowly the water is initially added to the loop, the shorter the time that is needed to bleed the air. Keep the bleed tool connected for at least 24 hours after the initial filling of the system. The bleed tool can be connected for a longer time if there is a concern about bleeding more air. The bleed tool must be removed during the normal operation of the servers to prevent less water from flowing through the servers than intended. The bleed tool can be disconnected from the manifold by actuating the collar on the quick connects (on one manifold the collar is pulled and on the other manifold the collar is pushed in).

The bleeding process must be repeated after a high heat load is introduced into the system for the first time. After a high heat load is introduced into the system for the first time, reconnect the bleed tool and leave it connected for at least 12 hours. The air is now purged from the loop. Keep the bleed tool on hand in case it needs to be reconnected if a new server or a new cold plate is added to the loop. The bleed tool can be reconnected at a future point if there is a concern that air is in the loop and needs to be purged.

Manifold and RDHx configurations

The 7965-S42 manifold can be used in conjunction with the 1164-95X RDHx to remove heat from the air that is passed through the servers. To make this connection, the supply water must be connected to the supply port of the RDHx first. The supply port of the RDHx is the quick connect coupling that is closest to the outside of the door. The outlet of the RDHx (the quick connect that is closest to the rack) must be connected to the supply of the manifold. The return of the manifold must be connected to the return of the CDU plumbing. This step ensures that the coldest water enters the RDHx first, in order to maximize the performance of the RDHx.

The manifold and RDHx can be configured with hoses that exits from the top of the rack or the bottom of the rack. The following graphics show the RDHx that is connected to the manifold in a top exit orientation.

Figure 13. Top exit configuration

Figure 14. Top exit configuration (top-down view)

When the RDHx and the manifold are oriented with the hoses down, the hose loop from the RDHx to the manifold can either be managed below the raised floor or between the floor and the bottom of the rack. A loop can be formed directly under the rack. The following figure shows the RDHx quick connect locations. The order of plumbing is the same as the top exit configuration.

Figure 15. Bottom exit configuration

Facility hose for the RDHx

The same facility hose kit that is included with the manifold is included with the RDHx. The same process to connect the hose to the manifold must be followed with the RDHx. If the RDHx is used in the same loop as the manifold, then there is an extra facility hose kit that is not used.

The following graphics show the CDU schematics of facility hose connections of the CDU, the RDHx, and the manifold connected together in one loop.

Figure 16. CDU schematic with facility hoses for a manifold and RDHx that connects directly to the CDU

Figure 17. CDU schematic with facility hoses for a manifold and RDHx that connects to an under-the-floor manifold

Table 4. Water volume

Part Description	Water volume
Manifold (supply tube, return tube, and hoses)	6 liters (1.6 gallons)
Rear door heat exchanger	9 liters (2.4 gallons)
Each 30.4 cm (1 ft) of the facility hose	0.15 liters (0.04 gallons)

Two different loops (hot and cold water)

If you want to run warm water through the manifold and servers and cold water through the RDHx, you can plumb two loops to each rack. The water connections to the manifold and RDHx are performed separately as though the other entity does not exist. Facility hose kits are provided for each manifold and RDHx. You must plan carefully to ensure that adequate space is provided for two hose kits per rack.

Cooling loop requirements

Floor cutout

Racks with water hoses and power cords that exit from the bottom of the rack require a floor tile cutout of at least 30.48 cm (12 in.) long by 22.86 cm (9 in.) wide. Due to the hose bend radii, the hole must be positioned towards the side of the rack without the manifold (the left side of the rack when looking at the rear of the rack). The left edge of the hole must be at least 11.43 cm (4.5 in.) from the side and 3.81 cm (1.5 in.) from the back edge of the rack (not including doors). Hole placement on the tile depends on the location of the rack, tile size, and tile load limitations.

Figure 18. Floor cutout