I have been wondering why the lower end POWER servers have local and far memory and not local and near. Perhaps you wondered too! Well at the Miami Power Technical University, I got to talk to Dr Joel Tendler (IBMer) and a POWER processor guru and put the question to him. He covered this sort of architectural topic in his presentation at the event and I learnt a lot in this area by listening to the "expert". Below is some background and the explanation too.

As we have covered in this series, sometimes the memory is not local but connected to a different POWER chip. When a process (program) has to access non local memory it needs to first check it is not cached somewhere and if not, read it from the POWER chip that has the memory controller for the RAM.  At the Technical University, I also attended a session from Jeff Stuecheli on POWER Cache Coherence i.e. the protocols for determining the best (meaning up to date) copy of a cache line, how to lock it for write access and how to ensure the update gets back to the memory - he is the chief memory architect and designer.  Well, I can't say I understood much of it (may be 10%) as it was advanced deep stuff but the impressions I got was "it is way more complicated than I thought but we really have loads of clever people working on the world leading technology. It explains how we get to 256 way machines with good performance".  Perhaps, I will go to his session again in the next Technical University and try to remember 20% of it this time :-)

On a practical side, each POWER has what is called a fabric bus controller (FBC) built into the chip, which is used to communicate with other POWER chips for control (finding cache lines and access modes) and for transport (moving the cache line to the POWER needing it).

• XYZ bus - up to three fabric port connections (hence three letters)
  • These are only used in the Enterprise server models like Power 780/780/795, E880 and E980 to directly connect each POWER processor in the same CEC drawer. 
• AB bus - up to two fabric port connections (hence two letters)
  • These are used by lower end models for POWER chip to POWER chip connections
  • In the large Enterprise servers Power 780/780/795, E880 and E980 these are use to connect the four CEC drawers.

In diagram form, a single POWER chip has five buses as below: