Model 7965-S42 water cooling manifold (Feature codes ECR3 and ECR4)

Learn about the water cooling manifold that is available for model 7965-S42 racks with feature code (FC) ECR3 or ECR4 installed.

Overview

The 7965-S42 water cooling manifold provides water supply and water return for 1 - 20 servers that are mounted in a 7965-S42 42U slim rack. The manifold is mounted on the right side of the rack (when viewed from the rear of the rack) and extends for 40U. Power distribution unit (PDU) pockets on the right (when viewed from the rear) are not accessible and cannot be populated in the water-cooled configuration. The manifold does not interfere with the placement of servers or other I/O drawers. Quick connect fittings are located every 2U on the manifold for water supply and water return that provides 20 pairs of fittings.

Note: This solution is only available for use with IBM® water-cooled servers.

Requirements

You must obtain a cooling distribution unit (CDU) and water that meets cleanliness, filtration, and chemical requirements that are listed in Planning for water cooling.

CDUs are available from suppliers such as Motivair and Nortek. CDUs deliver proper water flow rate and temperature to cool the servers, while it maintains the temperature above the dew point to avoid condensation. CDUs are also essential to control the closed loop of the water that flows through the servers to maintain proper water cleanliness, filtration, and chemistry while it controls the wet materials in the water loop.

Configurations

The manifold can be used only with a 7965-S42 rack. The water cooling feature of IBM servers cannot be configured in a non-IBM rack.

The manifold can be used only in the rack to cool servers, or it can be used with a rear door heat exchanger (RDHx) in a series flow to cool the remaining heat that is transferred to the air. For more information about RDHx information, see Model 1164-95X rear door heat exchanger.

Manifold only configurations

FC ECR3 can be used to order the manifold with water input and output at the top of the rack. Since the hose exits through the top of the rack, the top 2U must be left vacant. All 2U drawers must be populated into the rack on odd EIA increments.

Figure 1. Rack and manifold with supply and return hoses that exits through the top of the rack

FC ECR4 can be used to order the manifold with water input and output at the bottom of the rack. Since the hose exits through bottom of the rack, some space must be left open on the bottom of the rack. When 1U of the bottom space is left open, all 2U drawers must be populated into the rack on even EIA increments. When 2U of the bottom space is left open, all 2U drawers must be populated at odd EIA increments.

Figure 2. Rack and manifold with supply and return hoses that exits through the bottom of the rack

Top hose exit locations

The following graphics show the location of the hose that exits from the top of the 7965-S42 rack.

Figure 4. Top hose exit locations (when viewed from the top)

Note: Approximately 0.91 m (3 ft.) of hose is available after the hose exits the top of the rack.

Bottom hose exit locations

The following graphics show the floor cutout locations and dimensions that are required for water hoses that are routed out through the bottom of the rack and under the floor. Power cables can also use this cutout.

Bottom hose exit locations (when view from the bottom) — Figure 6. Bottom hose exit locations (when viewed from the bottom)

Note: Approximately 0.91 m (3 ft.) of hose is available after the hose exits the bottom of the rack.

Specifications

Table 1. Manifold specifications
Manifold characteristics	Properties
Manifold weight - dry	13.6 kg (30 lbs.)
Manifold weight - with water	17.5 kg (38.6 lbs.)
Manifold volume	6 L (1.6 gal)
Note: For more information about the rack weight, see Model 7953-94X and 7965-94Y rack.

The rear door heat exchanger can be used with this rack. For more information about rear door heat exchangers, see Model 1164-95X rear door heat exchanger specifications.

Hoses

The servers are connected to the manifold through quick-connects. The manifold has one cold water inlet that leads to the rack and one warm water outlet. Hoses are supplied by IBM. Hoses can be cut to length, but first must be cleaned so that no particles are inside the hose before installation. Some slack must be kept in the hose for easier installation. For more information about suggested clamping tools and specifications, see the Oetiker website.

The manifold has 182.9 cm (6 ft.) of 25.4 mm (1 in.) inner diameter flexible hoses on the supply side and 182.9 cm (6 ft.) of 25.4 mm (1 in.) inner diameter flexible hoses on the return side. After accounting for hose length inside of the rack, there is approximately 91.4 cm (3 ft) of hose for each of the supply and return hose that exits the rack.

Table 2. Hose kit dimensions
Hose information	Dimensions or type
Hose length	426.72 cm (14 ft)
Hose machine end	Quick-connect
Water supply end	25.4 mm (1 in.) National Pipe Thread Taper (NPT) male barb and clamp¹
Bend radius	203.2 mm (8 in.)
Hose inside diameter	25.4 mm (1 in.) plus or minus 0.5 mm (0.02 in.)
Hose outside diameter	34.54 mm (1.4 in.) plus or minus 0.76 mm (0.03 in.)
Notes: The facility hose kit is supplied in a separate box from the rack and contains the following items: Two 182.9 cm (6 ft.) hoses with preattached quick-connects for connection to manifold. At one end of the hose, the quick-connect mates with the quick-connect at the end of the hose that returns from the manifold. The other end of the hose is a bare cut end. Two 25.4 mm (1 in.) NPT male barbs. One end of the fitting is a 25.4 mm (1 in.) barb to fit on the inside of the 25.4 mm (1 in.) inner diameter hose. The other end of the fitting is a 25.4 mm (1 in.) NPT male barb. Four Oetiker hose clamps 16703242 (two hose clamps are required and two hose clamps are extra). Three supply labels (only two supply labels are required). The supply labels need to be installed on the supply end of the hose after it is connected to the facility. Three return labels (only two return labels are required). The return labels need to be installed on the return end of the hose after it is connected to the facility. ¹You must provide a 25.4 mm (1 in.) NPT female barb fitting on the facility hoses.

The customer provided interconnection to the rack (under-the-floor manifold, CDU, and so on) must have a 25.4 mm (1 in.) female NPT fitting for each manifold supply and return connection. The 25.4 mm (1 in.) NPT male barb fitting from the hose kit must be threaded onto the NPT female barb fitting on the customer CDU plumbing fitting. A thread sealant must be used to create a leak-free connection. You cannot use Teflon tape, as Teflon tape particles might enter the water stream.

To make the connection from the hose to the barb fitting, the facility hoses must first be cut to length. If the CDU plumbing fitting requires a longer hose than the 426.72 cm (14 ft) facility hose, the plumbing fitting must be altered to bring the fitting close enough such that the 426.72 cm (14 ft) facility hose is sufficient. The ends of the hose must be cleaned before the installation so that no particles are inside the hose. The clamp is inserted over the hose and then the barb is inserted into the hose. Position the clamp near the hex portion of the barb fitting (not over the barb) and tighten the clamp with an Oetiker clamp tool. For more information about the clamp tool, see Forged steel Standard Jaw Pincers. The ears of the clamp must be tightened so that they make contact with each other. When the clamp is released, the ears of the clamp relax and a small space is left between the ears. This small space is normal. Figure 8 shows the dimension S that must be fully closed during the clamping process.

The supply and return labels must be applied on both ends of the facility hose to indicate the function of each facility hose. The quick connects on the end of the facility hose can now be connected to the quick connects on the manifold. For more information about mating quick connects, see step 3 at Replacing the water manifold in the 8335-GTW or 8335-GTX system.

The following graphics show the CDU schematics of facility hose connections.

Figure 9. CDU schematic with facility hoses that connects directly to the CDU

Figure 10. CDU schematic with facility hoses that connects to an under-the-floor manifold

Table 3. Water volume
Part Description	Water volume
Manifold (supply tube, return tube, and hoses)	6 liters (1.6 gallons)
Each 30.4 cm (1 ft) of the facility hose	0.15 liters (0.04 gallons)

Water flow

Water does not flow through the manifold unless the supply and return circuits are connected. This situation normally occurs when a water-cooled server is connected to the manifold. An option to flow water through the manifold before servers are connected is to attach the bleed tool that is included along with the manifold. If you want to commission the system before the servers are connected, the bleed tool can be used to allow water to flow. With the bleed tool connected, a limited amount of water (7.5 - 11.3 liters (2 - 3 gallons) per minute per rack) can flow through the manifold. It is advised to continue to have the water flow until the servers are connected to avoid stagnant water from sitting in the manifold for an extended time. For instructions on how to connect the bleed tool, see Bleeding air from the manifold.

Initial filling of the rack and servers with water

To initially fill the water to the rack with servers that are connected to the manifold, complete the following steps:

Ensure that the bleed valves and CDUs are open in your facility infrastructure.
Ensure that the cap on the bleed tool is loosened as far as possible, but without falling off. For more information, see Figure 11.
Place the air bleed valve on top of the rack and secure it to the top of the rack. A screw must be used to secure the bleed tool to the rack. For more information, see Figure 12.
Connect the bleed tool to the top most quick connects on both manifolds. Connecting the bleed tool to the manifold before you add water, removes the excess pressure that is included in the components.
Make sure that all servers have quick connects plugged to the supply and return manifolds in the rack.
Fill the rack as slowly as possible to create less mixing of the water that enters the bottom of the rack with the air that exits from the top of the rack.

Figure 11. Opening the cap in the bleed tool

Figure 12. Attaching the bleed tool to the top of the rack

Bleeding air from the manifold

Air must be bled from the system when the servers are first connected and the water starts to flow. This step must be repeated if more servers are added to the rack, or if a new cold plate is introduced to the water loop, such as a cold plate field replaceable unit (FRU) replacement.

To bleed the air from the water loop, follow the procedure regarding the initial filling of the loop. Ensure that you connect the two quick connects on the ends of the hoses of the bleed tool to the top-most quick connects on the supply and return manifolds. If possible, temporarily increase the water flow rate to the rack during the bleeding process up to an average of 5.6 liters (1.5 gallons) per minute per drawer to help free bubbles from internal surfaces. Do not increase the water flow rate to the point that the inlet pressure to the rack exceeds 6894 pascals (40 pounds per square inch (psi)).

It is important not to leave the bleed tool connected to the system during normal operation. You must allow twice the amount of water flow through the bleed hose as through a server, which decreases the amount of water that flows through the servers. For example, when there are 18 servers in a rack with the bleed tool that is attached to both manifolds, then following water flow rates apply:

If you are delivering 68.13 liters (18 gallons) per minute to the rack, then 7.57 liters (2 gallons) per minute goes to the bleed tool and 3.33 liters (0.88 gallons) per minute goes to each server.
Note: 3.33 liters (0.88 gallons) per minute is calculated by 60.56 liters divided by 18 servers (16 gallons divided by 18 servers)..
If you increase the water flow to 75.7 liters (20 gallons) per minutes to the rack, then 7.57 liters (2 gallons) per minute goes to the bleed tool and 3.78 liters (1 gallon) per minute goes to each server.
If you are delivering 34.06 liters (9 gallons) per minute to the rack, then 3.78 liters (1 gallon) per minute goes to the bleed tool and 1.66 liters (0.44 gallons) per minute goes to each server.
If you increase the water flow to 37.85 liters (10 gallons) per minute to the rack, then 3.78 liters (1 gallon) per minute goes to the bleed tool and 1.89 liters (0.5 gallons) per minute goes to each server.

The bleed valve must be left in place for as long as practical. The more slowly the water is initially added to the water loop, the shorter the time that is needed to bleed the air. Keep the bleed tool connected for at least 24 hours after the initial water filling of the system. The bleed tool can be connected for a longer time if there is a concern about bleeding more air. The bleed tool must be removed during the normal operation of the servers to prevent less water from flowing through the servers than intended. The bleed tool can be disconnected from the manifold by actuating the collar on the quick connects (on one manifold, the collar is pulled and on the other manifold, the collar is pushed in).

The bleeding process must be repeated after a high heat load is introduced into the system for the first time. After a high heat load is introduced into the system for the first time, reconnect the bleed tool and leave it connected for at least 12 hours. The air is now purged from the water loop. Keep the bleed tool on hand in case it needs to be reconnected if a new server or a new cold plate is added to the water loop. The bleed tool can be reconnected at a future point if you feel that air is in the loop and needs to be purged.

If there is a troublesome rack, then the following procedure can be used to jump-start this process, before following the preceding bleeding air from the manifold process. This process is not realistic to deploy on every rack in a large installation. The goal is to remove a large amount of air that might be trapped at the top of either manifold. The following clean process prevents the air from being reabsorbed into the water.

Ensure that the cap on the bleed tool is loosened as far as possible, but without the cap falling off.
Start with the return manifold (the rightmost manifold as viewed from the rear of the manifold). Connect the bleed tool to this manifold only. Lift the bleed valve as high as possible, assuring that the hose monotonically increases in height from the quick connect coupling to the bleed valve. Gently shake the bleed valve and the hose near the quick connect. This step might allow a large air bubble to travel up the hose and escape out of the bleed tool. You might hear the valve as the air is expelled or see a tiny amount of moisture on the valve. This situation does not occur always.
Continue to gently shake the bleed valve. Another air bubble might travel up the hose to the bleed valve. Expect approximately 30 seconds for the bubble to travel upwards and escape from the bleed tool.
Repeat this step 10 times.
Note: If you continue to hear air that is escaping or see water that is escaping from the bleed tool, repeat the step until it no longer occurs.
Remove the quick connect from the return manifold.
Attach the other hose to the supply manifold (the leftmost manifold as viewed from the rear of the manifold).
Repeat steps 1 - 4 for the supply manifold.
Proceed to Bleeding air from the manifold.

Manifold and RDHx configurations

The 7965-S42 manifold can be used in conjunction with the 1164-95X RDHx to remove heat from the air that is passed through the servers. To make this connection, the supply water must be connected to the supply port of the RDHx first. The supply port of the RDHx is the quick connect coupling that is closest to the outside of the door. The outlet of the RDHx (the quick connect that is closest to the rack) must be connected to the supply of the manifold. The return of the manifold must be connected to the return of the CDU plumbing fitting. This step ensures that the coldest water enters the RDHx first. This maximizes the performance of the RDHx.

The manifold and RDHx can be configured with hoses that exits from the top of the rack or from the bottom of the rack. The following graphics show the RDHx that is connected to the manifold in a top exit orientation.

Figure 14. Top exit configuration (top-down view)

When the RDHx and the manifold are oriented with the hoses pointed down, the hose loop from the RDHx to the manifold can either be fitted below the raised floor or between the floor and the bottom of the rack. A hose loop can be formed directly under the rack. The following figure shows the RDHx quick connect locations. The order of plumbing is the same as the top exit configuration.

Facility hose for the RDHx

The same facility hose kit that is included with the manifold is also included with the RDHx. The same process to connect the hose to the manifold must be followed for the RDHx. If the RDHx is used in the same hose loop as the manifold, then an extra facility hose kit might not be used.

The following graphics show the CDU schematics of facility hose connections of the CDU, the RDHx, and the manifold connected together in one hose loop.

Figure 16. CDU schematic with facility hoses for a manifold and RDHx that connects directly to the CDU

Figure 17. CDU schematic with facility hoses for a manifold and RDHx that connects to an under-the-floor manifold

The following table shows the water volume through each part.

Table 4. Water volume
Part Description	Water volume
Manifold (supply tube, return tube, and hoses)	6 liters (1.6 gallons)
Rear door heat exchanger	9 liters (2.4 gallons)
Each 30.4 cm (1 ft) of the facility hose	0.15 liters (0.04 gallons)

Two different loops (hot and cold water)

If you want to run warm water through the manifold and servers and cold water through the RDHx, you can plumb two hose loops to each rack. The water connections to the manifold and RDHx are performed separately as though the other entity does not exist. Facility hose kits are provided for each manifold and RDHx. You must plan carefully to ensure that adequate space is available for two hose kits per rack.

Cooling loop requirements

A secondary cooling loop, separate from the main site cooling loop, is required for the manifold.
Cooling distribution units are available from suppliers such as Motivair and Nortek.
The secondary cooling loop must meet the requirements that are outlined in the water chemistry specification.

Floor cutout

Racks with water hoses and power cords that exit from the bottom of the rack require a floor tile cutout of at least 30.48 cm (12 in.) long by 22.86 cm (9 in.) wide. Due to the hose bend radii, the hole must be positioned towards the side of the rack without the manifold (the left side of the rack when looking at the rear of the rack). The left edge of the hole must be at least 11.43 cm (4.5 in.) from the side and 3.81 cm (1.5 in.) from the back edge of the rack (not including doors). Hole placement on the tile depends on the location of the rack, tile size, and tile load limitations.