AIX Service Strategy and Best Practices

AIX Level Naming

AIX Levels are named based on Release, Technology Level, Service Pack, and Build Sequence Identifier

  RELE-TL-SP-BBBB

The command ’oslevel –s’ will print out the current AIX Level.

  # oslevel –s  7100-01-07-1316
                |    |  |  '--------------------- Build Sequence Identifier
                |    |  '-----------------Service Pack 7
                |    '------------ Technology Level 1
                '-- Release 7.1  

Note: `oslevel –s` shows the latest full Service Pack that has been applied.
If individual fileset updates have been applied, consider the –g or –q options along with -s.
See oslevel man page in the Information Center for more details.

Releases

AIX Releases contain new software features as well as the latest fixes for defects and exploitation of new hardware.

New Releases are shipped approximately every 4 years

They are supported for usage and debug for 10 years under standard SWMA, plus 3-5 years of optional, separately priced, extended support.
The Software Lifecycle page shows the End of Support dates when they are published.

Fixes are available for supported releases on all Technology Levels that are still active for Fix Support (see below).

Technology Levels (TLs)

Technology Levels contain software enhancements, new hardware exploitation, and defect fixes.

New Technology Levels generally ship once a year in the Fall.
They are supported by fixes provided through Service Packs and iFixes for 3 years.
The AIX TL Lifecycle page shows the details Release dates and End of Fix Support dates for each TL.

Usage and debug support is still available for a TL even after the End of Fix Support, until the Release reaches End of Support.

Service Packs (SPs)

Service Packs contain fixes for defects impacting customers, fixes for critical defects found in internal testing, and can contain enablement for new hardware.

In general, changes that are allowed in a Service Pack are minimal defect fixes that do not change default behavior nor add new functionality.

New Service Packs are usually released twice a year for the latest TL in a release.

Build Sequence Identifier

The Build Sequence Identifier is used to determine valid update targets when moving to a new Technology Level. See more under ‘Planned Updates – Other Considerations’.

It is possible to for a single Service Pack to have multiple iterations, which can be differentiated by their Build Sequence Identifier. The most common reason this happens is to include late-breaking fixes in the Service Pack released on Fix Central that were found after the media was finalized.

APARs
(Authorized Program Analysis Reports)

An APAR is a record of a problem reported with AIX, and also associated with the fix for that problem. Each APAR applies to a specific Technology Level. If the same problem is reported and/or fixed on other Technology Levels, there will be different APARs for each of them. It is important to understand this when tracking and managing a fix across multiple levels of AIX.

When an APAR has shipped, the Technology Level to which it applies is noted in the Abstract.

  APAR=iz98260  request for vgsa update of concurrent vg hangs  applies to aix 6100-06  

APARs fixing the same problem on other Technology Levels are noted in the Comments.

  COMMENTS:  6100-05 - use AIX APAR IZ98619  6100-06 - use AIX APAR IZ98260  

iFixes
(Interim Fixes)

iFixes are temporary relief for defects, which can be used to fix especially critical problems that cannot be avoided until the permanent fix can be installed. These are custom built one-off fixes that patch only the required files. iFixes are tested for functionality and some regression, but generally not exposed to full regression testing or system test.

There can only be one iFix installed at a time for any given file, though there can be multiple iFixes installed for the same fileset. Installing an iFix will lock the entire fileset from being updated except by an update containing the fix for the same APAR(s) resolved by the ifix.

More information about managing iFixes is available in this document: Managing Interim Fixes on AIX.

Security iFixes

Security iFixes are a special set of iFixes published when a security vulnerability in AIX is fixed. The published Security Vulnerability Alert will include an initial set of proactively published iFixes to provide immediate relief for systems running one of the latest 3 Service Packs of each active Technology Level. It is important to stay current to have immediate access to security iFixes.

These iFixes have had limited functional and regression testing but not the full regression testing that takes place for Service Packs. See the ‘Reactive Maintainence’ section below for more information on security vulnerabilities.

iFixes for earlier Service Packs, or custom iFixes for certain systems can be requested by opening a case with IBM.

iFix Naming

iFixes are labeled as follows to help quickly identify some key information:
Label: <APAR>[s|m]<SP><seq>, for example:

  IV24230s5a  ^^^^^^^ APAR Number (specific to the Technology Level on which the iFix can be applied)         ^ (s)ingle or (m)ulti fix (if it’s multi, inspect the iFix description or aparref with emgr for the full list of APARs)          ^ Service Pack (iFix will apply to this SP at least, possibly others if the code is common across multiple SPs)           ^ sequence letter (to ensure each ifix has a unique label)

Sometimes this naming convention cannot be followed and the detailed iFix description should be consulted for more details about the contents. This is viewable with:
`emgr –l -v3 –L <iFix Label>` For installed iFixes, or
`emgr –d –v3 –e <iFix filename>` For iFix package files.

Debug Packages

Sometimes debug/test patches are sent out in the same packaging that iFixes use, but not following the standards above. These packages will be labeled beginning with “dbg” to distinguish them and will not usually contain APAR numbers. These should be removed as soon as debug/testing is complete and replaced by an Interim Fix if appropriate.

Service Pack Updates

Planned Service Pack updates are driven by requiring a software fix and/or new hardware enablement that is shipped in a Service Pack.

Contents of new Service Packs should be reviewed as they are published by IBM by examining the ‘Fix details’ on Fix Central. If the Service Pack contains fixes for problems (documented by APAR numbers) that might be encountered in your particular environment, then a planned update to apply that Service Pack should be considered to avoid encountering the problem. Important Security, HIPER, and PE resolving APARs are highlighted for careful consideration. More information on these ‘Critical APARs’ below.

New hardware enablement might also exist in a new Service Pack and would be documented in the hardware announcement. More information about the levels required to support a specific system is located in the System to AIX maps page.

New Service Packs require a reboot.

When updating Service Packs, the release date should be considered. The Fix Level Recommendation Tool (FLRT) should be used to plan the update. IBM will generally mark an AIX Service Pack recommended 90 days after it is released. However, Security Vulnerability and HIPER APARs should also be evaluated before updating to a recommended level. FLRT has links to “AIX/VIOS Security Tables” and “AIX HIPER Tables” which provide a very useful view of what HIPER or Security problems each release is exposed to, and where to get fixes.

Technology Level Upgrades

Technology Level upgrades are initiated to receive the new software enhancements and new hardware exploitation they contain; or because your current Technology Level is approaching or past the End of Fix Support date.

Contents of new Technology Levels are available on Fix Central just as for Service Packs. New features and benefits of updating to a new TL can be found in the AIX Technology LinkedIn Group.

If your current Technology Level is no longer active for new fixes, or is about to become inactive, a Technology Level upgrade should be considered. This will ensure that if any unexpected problems are encountered, a fix for the problem can be applied without any major update being required.

New Technology Levels require a reboot.

When upgrading Technology Levels, the release date and active life should be considered. When a new Technology Level is released, testing continues both internally and externally, and the first Service Pack is usually mandatory. The Fix Level Recommendation Tool (FLRT) should be used to plan the upgrade. IBM will generally mark a Technology Level recommended 90 days after its first Service Pack has been released. However, the same evaluations of critical HIPER/PE/Security fixes should be made, as described above for Service Pack Updates.

Applying the Updates

Technology Levels must be applied as a group, using the ‘smitty update_all’ or ‘install_all_updates’ commands. Installing a partial Technology Level will not be recognized from a support standpoint. Technology Level updates should always be committed and cannot be rejected. Because of that, you should always create a backup and plan on restoring that backup if you need to roll back to your previous level, or use tools like alt_disk_install or multibos as a way to get back to your previous level. After the Technology Level has been successfully applied and tested, a backup should be taken for disaster recovery situations.

Service Packs should generally be applied as a whole to simplify inventory and for risk reduction, because the filesets in a Service Pack are regression tested as a group, not as individual updates. Service Packs can be applied but not committed at first to ensure there are no problems after updating.

Other Considerations

When moving up to a new Technology Level, you must move to a Service Pack that has the same or later Build Sequence Identifier than your current Service Pack. The Service Pack number you are updating to may not be equal to or higher than your current Service Pack, but an equal or higher Build Sequence Identifier will indicate that the Service Pack was built at the same time or later, and therefore includes the same fixes. The update process will not allow an earlier built Service Pack (with a lower Build Sequence Identifier) to be applied, to avoid the chance of regression.

When updating in any form, any installed iFixes should be evaluated to know if the permanent fix will be installed in the new update, or if another iFix will need to be requested, to be applied after updating. iFixes will be automatically removed by the update if the Technology Level or Service Pack being applied contains the permanent fix (APAR) that the interim fix was patching. An installp preview will output which iFixes will be automatically removed.

Doing a “Fix search” on Fix Central for an APAR number will usually present a “Sibling information” page in the search results. This page shows, in one place, all the Service Packs which contain fixes for the same problem, and what the specific APAR numbers are for each Technology Level. This can be useful especially when planning an update or upgrade if you have iFixes installed and would like to know which Service Packs (on various Technology Levels) contain the permanent APAR fix.

Release Migrations

When migrating the operating system to a new version or release of AIX, you must be careful to not down-level the operating system by migrating to a previously built level (that may not contain all your current fixes). When applying updates, we use build dates to prevent down-leveling, but a migration ignores build dates because once it’s started, it must complete as cleanly as possible. So you must manually verify build dates before migrating, to be sure the target media build date is equal or greater than what is currently running. Compare the build dates of what is currently installed on your system (using `oslevel`, see ‘AIX Level Naming’ above) with what is printed on the media to be used for migration.

If using a NIM lpp_source for migration, you must check the build date of the media it was created from. If your lpp_source was created from, for example, 7100-00 base images, and has updates for 7100-01, then you are actually migrating to 7100-00, and then updating to 7100-01. So be sure the base install level you are migrating to is a later build date.

Always run the pre_migration script on the system you are planning to migrate, and the post_migration script after migrating. This will catch some potential failures before migrating, and tell you of any software that did not migrate correct afterwards, which can possibly be corrected. These two scripts are found on physical media in <mount_point>/usr/lpp/bos, or in a NIM SPOT in <spot_location>/lpp/bos.

As always, back up the system before migrating in case of an unrecoverable failure in the migration of the operating system.

iFixes

When an especially critical problem is encountered that cannot be avoided by some workaround, and the permanent fix cannot be applied via Service Pack update, an iFix can be requested for temporary relief.

iFixes are available by request for active Technology Levels only. If the fix requested is shipped in a later Service Pack on your current Technology Level, you will be told to update to that Service Pack instead of getting an iFix. If you cannot update to the Service Pack at this time, and the fix is very critical, the iFix request can sometimes still be made. Once a request is made it will be evaluated for feasibility. IBM will make every effort to supply an iFix for a particular issue; however, there may be some that cannot be provided due to risk or complexity.

It is recommended to update to the Technology Level or Service Pack containing the permanent fix during the next planned maintenance window.

Critical APARs

IBM highlights three categories of APARs as especially critical for customers to be aware of:

• Security APARs
  These resolve security vulnerabilities in AIX. They are reported as soon as IBM is aware and a fix is available.
• HIPER (High Impact PERvasive) APARs
  These resolve defects which have a combination of both high impact and pervasiveness such that IBM thinks all customers exposed to the issues should proactively apply the fix to avoid encountering the problem.
• PE (PTF in Error) resolving APARs
  These resolve issues where a prior PTF caused a regression in functionality, breaking something that used to work.

Fix Central has information on all these critical APARs for each Service Pack in the ‘Fix details’ link. This page lists all APAR fixes in the Service Pack, starting with Security APARs, HIPER APARs, and APARs resolving PEs. At the bottom of the page, filesets in this Service Pack that cause regressions are listed as ‘Known problems with this package’, along with a link to the PE resolving APAR.

My Notifications is the primary way that IBM will notify customers of these critical APARs. Signing up for each AIX release will ensure you are notified of all Security, HIPER, and PEs reported. When a notification is received, it should be evaluated immediately to determine if it applies to your environment. If so, the fix will be available either via iFix or APAR, as documented in the notification.

FLRT (Fix Level Recommendation Tool) - can help determine if a system is exposed to one of these critical APARs by inputting the current system level or examining the “AIX/VIOS Security Tables” and “AIX HIPER Tables”.

Base Operating System

The base AIX Operating System contains the core AIX filesets.

Installation media is available from Entitled Systems Support.
Follow these tips to find a specific oslevel ISO image on the IBM ESS website.

Updates are available via Fix Central by entering product/release and selecting “Fix packs” and via the Service Update Management Assistant (SUMA), which is part of the base OS.
Interim Fixes are provided for security vulnerabilities, and for high-impact and pervasive (HIPER) issues. Bulletins are issued via My Notifications when these interim fixes are available.

Expansion Pack

The AIX Expansion Pack contains additional software that extends the base OS capabilities. These are software packages that may have more distribution restrictions than the base OS or which have been removed from the base OS media but may still be of value to customers. For example, cryptographic filesets and Java versions no longer required for base OS function can be found on the Expansion Pack.

The Expansion Pack is available from Entitled Systems Support.

Web Download Pack Programs

The Web Download Pack Programs site contains additional application and tools. Some of the content overlaps that of the Expansion Pack, but the AIX Web Download Pack Programs can be updated more quickly than the Expansion Pack release cycle allows.

AIX Toolbox for Open Source Software

The AIX Toolbox provides 250+ open source and GNU packages built for AIX.

See more details below under Open Source Strategy.

Base Operating System

IBM provides a small number of open source packages with the base AIX media, for example: OpenSSL, perl, python, NTP3, sendmail, and rpm.

This software is generally for use by the OS but may be used by 3rd party applications as well. However, these packages are usually only updated on major new releases of AIX (exceptions may be made to address critical security issues).
It is recommended to consider the AIX Toolbox for Open Source Software if the software package is available there, since those packages are generally updated more frequently.

Expansion Pack / Web Download Pack Programs

IBM provides a few other open source packages on the Expansion Pack and Web Download Pack Programs site. This includes packages like OpenSSL, OpenSSH, and NTPv4.

These packages are updated more frequently on the Web Download Pack Programs site.

AIX Toolbox for Open Source Software

For many more open source packages compiled for AIX, the AIX Toolbox provides 250+ open source and GNU packages built for AIX.

All the tools are provided in RPM package format and a yum repository is available to easily keep them up to date. More details available in the site’s readme file.

These packages are built to install alongside any base OS version of the same software that is currently installed.

Software provided on the AIX Toolbox is provided “as is” and is not supported by IBM Support at this time. However, there is a very active AIX Open Source Software Forum where the AIX development team can be contacted to help resolve issues with the porting/packaging or to request porting a new open source package to AIX. Announcements are made on the forum when packages are updated with the latest fixes / security patches.

IBM Support for Open Source Software

Read more about how Open Source Software for AIX is supported in this document.

Fix Central
http:// www.ibm.com/support/fixcentral/
Central site for AIX update distribution. Here you can find details of Technology Levels and Service Packs that have been released, and you can download them. There are also links to this Best Practices doc and other service-related documents.

My Notifications
https://www.ibm.com/systems/support/myview/subscription/css.wss/
Sign up for Update Notifications for AIX and other products. This is the primary way customers will be notified of any critical AIX fixes including HIPER and Security Vulnerabilities. 

Fix Level Recommendation Tool (FLRT)
http://www.ibm.com/support/customercare/flrt
Tool for recommending upgrades and updates for AIX as well as other products. This is a good place to check when planning what level to roll out in the next planned maintenance window.
FLRT also has links to the “AIX HIPER Tables” and “AIX/VIOS Security Tables” which provide a very useful view of what HIPER and Security problems each release is exposed to, and where to get fixes. This includes links to the Bulletin with more information and links to download the fix either as an Interim Fix or Service Pack.

AIX Installation Tips
AIX 7.2 - https://www.ibm.com/support/pages/aix-72-installation-tips
AIX 7.1 - https://www.ibm.com/support/pages/aix-71-installation-tips

Managing Interim Fixes on AIX
http://www.ibm.com/support/docview.wss?uid=isg3T1012104
Detailed document for how to install and manage Interim Fixes on AIX

IBM System to AIX maps
http://www.ibm.com/support/docview.wss?uid=ssm1platformaix
Information on the AIX levels required to support different hardware

AIX Technology Level Lifecycles
http://www.ibm.com/support/docview.wss?uid=isg3T1012517
Life Cycle of AIX Releases and Technology Levels, including dates for release dates and end of support dates.

Product Support Lifecycles
http://www.ibm.com/software/support/systemsp/lifecycle
Life Cycle information for AIX and other products.

Entitled Software Support (Ordering Product Media)
https://www.ibm.com/servers/eserver/ess/ProtectedServlet.wss
The place to order product media or download ISO images.

AIX IT Infrastructure page
https://www.ibm.com/it-infrastructure/power/os/aix
AIX roadmap and promotional material.

AIX Support Center Tools
http://www.ibm.com/support/aixtools
Tools that IBM support will ask you to install and run to help gather information to aid in problem determination.

IBM Ideas
https://ideas.ibm.com/
Ideas portal to submit and vote on requests for enhancement (RFEs) of AIX and other IBM products.

VIOS Best Practices
https://www.ibm.com/support/pages/ibm-virtual-io-server-support-power-systems-5
Best Practices for VIOS. This information is also available in VIOS Release Notes.

VIOS SSP Best Practices
http://ibm.biz/viossspbp
Best Practices for VIOS Shared Storage pools (SSP).

Service and Support Best Practices for Power Systems
https://www.ibm.com/support/pages/service-and-support-best-practices-power-systems
Other Best Practices guides for Power Systems including Performance.

A Brief History of AIX
https://www.ibm.com/support/pages/brief-history-aix
History of "What's New" in each major AIX release.

Hints and Tips for Migrating Workload to IBM POWER9 Processor-Based Systems
https://www.ibm.com/downloads/cas/39XWR7YM
White paper from performance and development teams with tips and recommendations when migrating to Power 9.

AIX 7.3 Migration
https://community.ibm.com/community/user/power/blogs/rae-yang/2022/03/03/aix-73-migration
Article describing how to plan for and execute an operating system migration to AIX 7.3.