IBM Support

POWER System Firmware Warnings & Red Lights on the Dashboard

How To


Summary

Do you ignore red lights on your car dashboard? I think not and you should not ignore Firmware Updates either. They are there for your safety.

Objective

Nigels Banner

Steps

This was first published on DeveloperWorks in 2012 and was read by 28358 people.
The message is still true today.
Earlier today I received email from a customer reporting their large POWER7 based machines where on firmware 720_64 to 720_90 and their reluctance to take the outage to upgrade it. They were asking for fine details of newer firmware levels and what advantages this would bring to "justify the outage to their user departments".
 
To be blunt this is a horror story:
I lay awake at night in a cold sweat about stories like this. The customer has the whole "running computers plan upside down".  The question should be "can I risk not having the latest firmware" and user departments should expect either:
  • High Availability switch overs (few minutes outage) or
  • Live Partition Mobility (zero minutes outage) or
  • Regular planned outage (an hour). 
If the computer stops and it has firmware that is 2 years out of date ... it is the computer departments fault!  Particularly, if the firmware dates back to the initial release - this level would not even have CHARM enabled! As that arrives roughly 3 to 6 months after GA.

IBM does not issue new System Firmware levels for the fun of it!  It costs IBM many millions of dollars and man-years in effort in coding and particularly testing. Sure they make available new Hypervisor functions that you could regard as optional but the bulk of it is to increase reliability, ensure the integrity of your system,  the data in memory and the contents of your disks (to avoid corrupting your RDBMS), to avoid unexpected outages and to increase performance.
 

 
Large machines running POWER Systems Firmware on the initial generally availability releases of firmware often dating back to many years - running this in my humble opinion "bonkers".   Take a metaphor from maintaining your car:
  • When a red light starts blinking on the dashboard of your car - Do you ignore it for two years? 
  • What if you have 4 red lights, 2 orange lights and the words "Warning!!" flashing on the dashboard?
  • What would it take to get you to take the car to the garage to get it fixed? A complete break down 50 miles from town? In computer terms, a failure that takes days to fix.  A major life threatening accident with your family in the car? In computer term, you are sacked for incompetence!
The details of Systems Firmware history and fixes can be found on IBM FixCentral. You may need yout Machine-Type-Model (MTM). On AIX this is found at the first line of the lsconf command's output along with your Firmware level further down:
$ lsconf
System Model: IBM,8408-E8E
Machine Serial Number: 21D494V
Processor Type: PowerPC_POWER8
Processor Implementation Mode: POWER 8
Processor Version: PV_8_Compat
Number Of Processors: 4
Processor Clock Speed: 3724 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 27 w3-blue
Memory Size: 16384 MB
Good Memory Size: 16384 MB
Platform Firmware level: SV860_205
Firmware Version: IBM,FW860.70 (SV860_205)

. . .
Excellent webpage that defines all the terms we use in POWER land:
Exercise for the student:
Find your current Firmware Level then go up the FixCentral web-page counting the number of "Severity:  HIPER - High Impact/PERvasive, Should be installed as soon as possible."  warnings to the current best level.  This is the number of red lights on the dashboard.

My customer has five red lights on the dashboard and they are asking "Do we really need to update?".  My customer is "bonkers".  If they have an "incident" on these machines, the urge to say "we told you so!" will be overpowering.
 Good News:
I am told they can take a non-disruptive update of their firmware to older firmware (a better sane level without downtime) but they really should get to the latest version or one level back from that is it has just come out.
 The Smart Money:
Larger changes from (upping the major number like in our firmware above SV860 to SV870 are likely to be disruptive)  - but I would very highly recommend this too and plan this in the next update cycle and certainly before the end of the year at the very latest.  There are many fixes and performance tweaks in the later Firmware levels, that I would highly recommend and especially for the large machines Power 770, Power 780, Power 795, E850, E880, E950 and E880.
I would put this in the "don't phone or email me with a performance issue unless you are on the latest firmware because you would be wasting my time addressing already fixed issues".
 
  Conclusions
I know System Firmware updates are boring and hard to schedule but it is not an option but ignoring the problem is not an option either.  Just like driving your car with your eyes closed so you can ignore the red lights on the dashboard is not an option.
Live Partition Mobility has been available for 10+ years and HACMP (PowerHA) available for decades and these make it relatively painless.
If you don't regularly update firmware, you are playing a high stakes poker game with
  • your employers assets and
  • company profits, and
  • your own career. 
I don't think I can express that more clearly.
 

Additional Information


Other places to find content from Nigel Griffiths IBM (retired)

Document Location

Worldwide

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"Component":"","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"HW1W1","label":"Power -\u003EPowerLinux"},"Component":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG60","label":"IBM i"},"Component":"","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB57","label":"Power"}}]

Document Information

Modified date:
14 June 2023

UID

ibm11125111