IBM Support

padmin login fails with "INIT: failed write of utmp entry"

Troubleshooting


Problem

The PowerVM Virtual I/O Server (VIOS) prime administrator (padmin) user fails to log in through the network and through the console.

Symptom

Network login attempt to the VIOS through ssh or telnet and through the HMC terminal (vterm) console session fails with the following error:
 
     INIT: failed write of utmp entry: " cons"

Cause

The error is symptomatic of disk I/O problem to file systems in the VIOS rootvg volume group.  This is most commonly due to the VIOS boot device no longer being accessible.  When a disk in VIOS rootvg loses connectivity or goes into read-only mode, the file systems no longer have access to the underlying disk(s) or experience write failures, which can then lead to I/O errors such as  INIT: failed write of utmp entry: " cons".  In such cases, JFS2 automatically switches the file system to read-only mode to prevent any potential corruption caused by these I/O errors.  

Without access to the error log and snap data, diagnosing the exact cause may prove challenging. However, if all file systems appear to be in read-only mode, it usually suggests problems with the boot disk itself.
To resolve this issue, it is crucial to determine the boot disk type and ensure that the disk has read-write access.
Contacting your storage support representative for verification of storage provisioning to the VIOS might be necessary.

Environment

VIOS V3.1 and 4.1 managed by an HMC.

 

Diagnosing The Problem

Common causes can include (but are not limited to) any of the following:

  1. Permissions issue for /etc/utmp or /var/adm/wtmp
  2. Full rootvg file systems.
  3. Hardware failure related to VIOS rootvg, such as disk or adapter failure.
  4. rootvg file systems are in read-only mode
  5. Disk in VIOS rootvg is SAN-attached and logged disk reservation conflict errors
Sometimes, it may be possible to identify one of these common causes by running remote commands to the VIOS from the HMC.  
Determine whether the VIOS partition has console access.  To do so, try to open a console session for the VIOS partition in question by logging in to the HMC using the GUI or Command Line Interface (CLI) as hscroot user.  Note: This requires the HMC to have Secure Shell access enabled.
 
Using HMC Enhanced GUI
> Click Resources
> All Systems
> Click the system name hosting the VIOS partition in question
> On the left pane, click "Virtual I/O Servers" under Power VM
> Select VIOS partition name
> Click Actions > Console > Open Terminal Window
 
Using HMC Classic GUI
In the navigation pane:
> Open Systems Management
> Click Servers
> Click the Managed System name hosting the VIOS partition in question
In the work pane:
> Select the VIOS partition
> Click Tasks > Console Window > Open Terminal Window
 
Using HMC CLI
At HMC command line, type vtmenu
> Enter Number of the Managed System name
> Enter number of VIOS partition in question
If the console window is nonresponsive, you might see something similar to the following:
 
Opening Virtual Terminal On Partition t720vio1 . . .
Open in progress
Open Completed.
IBM Virtual I/O Server
login: padmin
/dev/vty0: You must "exec" login from the lowest login shell.
INIT: failed write of utmp entry: "          cons"
INIT: failed write of utmp entry: "          cons"
INIT: failed write of utmp entry: "          cons"
INIT: Command is respawning too rapidly. Check for possible errors.
 
If you receive these errors, determine if VIOS commands can be run remotely using HMC command, viosvrcmd, to see if the problem can be narrowed down to one of the common causes discussed on this document.
 
You first need to set up the required HMC user to run the viosvrcmd on the VIOS remotelty from the HMC following " HMC viosvrcmd fails with HSCL2970 " .

Method #1 - To run padmin commands :
viosadminuser@<HMC_hostname>:~> command="ioslevel"    <---- This is the padmin command to run
viosadminuser@<HMC_hostname>:~> viosvrcmd -m Server-8406-70Y-SN06BFF6A -p PS700_VIOS_Epic -c "$command" 
4.1.1.10     <---- This is the command output
 
Method #2 - To run command via oem_setup_env/AIX root shell :
 
viosadminuser@<HMC_hostname>:~> command="ls -l /etc/utmp"        <--- This is the root command to be ran
viosadminuser@<HMC_hostname>:~> viosvrcmd -m Server-8406-70Y-SN06BFF6A -p PS700_VIOS_Epic -c "$command" --admin   <---added the "--admin" for root
-rw-r--r--    1 root     system        34992 May 11 16:25 /etc/utmp    <---  This is the output
 
viosadminuser@<HMC_hostname>:~> command="ls -l /etc/wtmp"    
viosadminuser@<HMC_hostname>:~> viosvrcmd -m Server-8406-70Y-SN06BFF6A -p PS700_VIOS_Epic -c "$command" --admin   
-rw-rw-r--    1 adm      adm         3906792 May 11 16:26 /var/adm/wtmp

If the VIOS is reachable using HMC viosvrcmd command, next, try to check the following Probable Causes.
On the other hand, if the VIOS is not reachable through viosvrcmd, it might be in a "hung" condition.  For how to troubleshoot a VIOS hung condition, refer to MustGather Testcase for VIOS Crash or VIOS Hang Condition.

Resolving The Problem

Read the remaining document to its entirety before taking any corrective measures.
 
IMPORTANT
If the console errors are seen after an LPM operation completed OK, the loss of disk I/O to AIX rootvg is commonly due to LUN masking issue. In other words, it is a result of the storage not being provisioned to the inactive WWPNs (which post migration become the "active" wwpns).  
The following resource explains why a mobile LPAR could lose access to the storage after LPM:
https://community.ibm.com/community/user/blogs/hariganesh-muralidharan1/2020/06/16/npiv-storage-validation-options-for-lpm
Contact your SAN admin to verify the storage LUN masking for all active WWPNs.
 

Probable Cause #1 - Permissions issue for /etc/utmp or /var/adm/wtmp
 
viosvrcmd method #2 reflects the expected permissions for /etc/utmp and /var/adm/wtmp.
So if you get similar output, then your permissions are correct. 
If your permissions are different, correct them.  Then try to log in again.
 
 
Probable Cause #2 - Full rootvg file system
 
Check for full file system through viosvrcmd by substituting the command with df -g. If there are any file systems full or nearly full, address the issue before re-attempting the log in. For more details see Diagnosing Full File Systems in PowerVM VIOS.
 

Probable Cause #3 - Hardware failure related to VIOS rootvg
 
Substitute command in viosvrcmd with errlog (padmin) or errpt command (if in oem_setup_env/AIX root shell) to determine whether there are disk or adapter errors related to rootvg.
If there are no hardware errors related to rootvg, check if there are any LVM or file system errors associated with rootvg.
 


Probable Cause #4 - rootvg file systems are in read-only mode

 
This can happen if the system loses its path to the VIOS rootvg disks.  To test this, check for ioscli.log in /home/padmin and try making a copy of the file (cp ioscli.log test123.log) by using viosvrcmd to see if it completes successfully or if it fails with errors similar to the following:
 

     HSCL2970 The IOServer command has failed because of the following reason:

     Unable to open file: /home/ios/logs/ioscli_global.trace for append 
     Error from cliCheckFile:-1
     Unable to open file: /home/ios/logs/ioscli_global.trace for append 
     Error from cliCheckFile:-1
     ...
     Unable to open file: /home/ios/logs/ioscli_global.trace for append 
     Error from cliCheckFile:-1
     cp: test123.out: Read-only file system     <-------
     rc=1
     ...

Note: If the VIOS partition boots from SAN disk, contact your SAN administrator to check if the disk might have been put in read-only mode. This is known to happen when disk copy utilities are used by the storage subsystem.

If no hardware errors were found in the error log, but a "read-only file system" message is encountered, run mount command using viosrvcmd method #2. The mount command should show how many read-only file systems there are.  Possible output can include:

    node    mounted    mounted over    vfs       date        options
   ------ ----------- --------------  ------ ------------ ---------------
          /dev/hd4    /                jfs2  Mar 15 09:47 rw,log=/dev/hd8
          /dev/hd2    /usr             jfs2  Mar 15 09:47 rw,log=/dev/hd8
          ...
 
          OR
    node    mounted    mounted over    vfs       date        options
   ------ ----------- --------------  ------ ------------ ---------------
          /dev/hd4    /                jfs2  Mar 03 01:25 ro,degraded
          /dev/hd2    /usr             jfs2  Mar 03 09:47 ro,degraded
 
If all file systems are in read only mode, this generally happens when the file system does not have access to the underlying disks and/or when writes to the disks fail (basically I/O errors). In that situation, JFS2 puts the file system in read-only mode on purpose so that there is no further writes to the disks to avoid file system corruption due to I/O errors.  In such case, the VIOS should be booted in maintenance mode to verify the file system integrity by doing a thorough fsck to bring them to a clean state and to make sure the disk is reachable. If the system is in such state, it is very likely that the boot disk has issues but without access to the error log and snap data, it is difficult to confirm categorically.  Since the only option at that point is to reboot the VIOS to maintenance mode and run fsck on the file systems, a VIOS system dump might be worth taking for further investigation while bringing down the VIOS.
 
IMPORTANT
Depending on the VIOS state, the VIOS partition might need to be booted to maintenance mode. 
 
Related Reference
 
 
Probable Cause #5 - Disk in VIOS rootvg is SAN-attached and logged disk reservation conflict errors

This probable cause can be explored if/when VIOS login access has been reestablished, which sometimes may be possible after the VIOS is restarted (preferably with the "Dump" option if root cause is needed).
Examine VIOS errlog for SC_DISK_ERR10/RESERVATION CONFLICT errors.
 
SAN disks in VIOS rootvg must have the disk reserve policy set to single_path rather than no_reserve, so I/O is exclusive to the VIOS host.  To prevent disk reservation conflict, ensure reserve policy is single_path for all SAN disks in VIOS rootvg.
 
To list disks in VIOS rootvg
$ lsvg -pv rootvg
 
To list disk reserve policy value
$ lsdev -dev hdisk0 -attr reserve_policy
 
To change it to single_path requires VIOS reboot for change to take effect
$ chdev -dev hdisk0 -attr reserve_policy=single_path -perm
$ shutdown -restart
 

[{"Type":"MASTER","Line of Business":{"code":"LOB68","label":"Power HW"},"Business Unit":{"code":"BU070","label":"IBM Infrastructure"},"Product":{"code":"SSHQYKG","label":"PowerVM Enterprise Edition"},"ARM Category":[{"code":"a8m50000000L0KcAAK","label":"Archive-\u003EPowerVM VIOS-\u003EPADMIN\/CLI"}],"ARM Case Number":"TS005645356","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
03 November 2025

UID

isg3T1023923

Page Feedback