IBM Support

IBM Cloud Pak System (or IBM PureApplication System): Virtual Machine status changes to SAFERUNNING

Troubleshooting


Problem

A virtual machine's status (VM Status) in the Virtual System Instance page changes to SAFERUNNING. See the following figure for reference:
image 1332

Symptom

Generally speaking, a virtual machine's status (VM Status) might change to SAFERUNNING when a failure is encountered during the initialization script (/0config/0config.sh) execution.
A virtual machine's status (VM Status) might change to SAFERUNNING when the agent process that is running in the virtual machine crashes or encounters a failure.

Diagnosing The Problem

To diagnose the problem, let us consider the following permission issue as an example, which can be one of the many potential causes of this problem.
The following permission denied error message might be seen during attempts to reinstall the itmos or iwd agent when you rerun /0config/0config.sh. Check the /0config/0config.log on the virtual machine for a similar failure message:
                              
Traceback (most recent call last):
  File "/0config/nodepkgs/common/scripts/itmos/iwd_agent_install.py", line 234, in <module>
    install_iwdagent()
  File "/0config/nodepkgs/common/scripts/itmos/iwd_agent_install.py", line 192, in install_iwdagent
    call([os.path.join(TMPDIR, 'ks1output', 'installIraAgent.sh'), CANDLEHOME])
  File "/usr/lib64/python2.7/subprocess.py", line 524, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied
[Tue 04 Jun 2019 12:47:33 PM CDT] invoker An error occurred; see trace.log for details.
[Tue 04 Jun 2019 12:47:33 PM CDT] invoker Traceback (most recent call last):
  File "/0config/nodepkgs/common/python/invoker.py", line 227, in execute
    execfile(pyfile, _pyglobals)
  File "/0config/nodepkgs/setup/setup.py", line 379, in <module>
    maestro.check_status(rc, 'Monitoring - Failed to setup ITM OS and/or IWD agents')
  File "/0config/nodepkgs/common/python/maestro/__init__.py", line 330, in check_status
    raise debugUtil.CallError(rc, message)
CallError
[Tue 04 Jun 2019 12:47:33 PM CDT] invoker Monitoring - Failed to setup ITM OS and/or IWD agents
To avoid the problem, the execute permission (it is the default) must be allowed for the /tmp directory, and related scripts to be run successfully.
For example, see the following script location:
 /tmp/installiwd/ks1output/installIraAgent.sh

Resolving The Problem

To reinstall the agents cleanly and recover from SAFERUNNING, see the following procedures:
Recover from safe mode on Red Hat Linux systems:
  1. Switch to root context.
  2. Set execute permission on /tmp.
  3. Enter these commands to avoid 0config.sh running twice during restart:
    rm -f /etc/init.d/maestro_config
    echo '#!/bin/sh' > /etc/init.d/maestro_config
    echo "# chkconfig: 35 99 99" >> /etc/init.d/maestro_config
    echo "# description: Maestro bootstrap" >> /etc/init.d/maestro_config
    echo 'if [ "$1" != "start" ]; then' >> /etc/init.d/maestro_config
    echo "    exit 0" >> /etc/init.d/maestro_config
    echo "fi" >> /etc/init.d/maestro_config
    echo "/0config/0config.sh" >> /etc/init.d/maestro_config
    chmod 755 /etc/init.d/maestro_config
  4. Enter these commands to disable any additional IBM Tivoli Monitoring (ITM) agent that is running:
    chkconfig ITMAgents1 off
    chkconfig ITMAgents2 off
  5. Enter these commands to update the virtual machine:
    rm -rf /0config/itlm/foundation
    rm -rf /0config/safemode
    rm -rf /opt/IBM/maestro/agent/safemode
  6. Restart the virtual system instance (VSI) from the IBM Cloud Pak System Software user interface.
    Navigate to the Virtual System Instance page, locate the specific virtual system instance, and click Stop and Start in the right panel.
Ensure that the status of the virtual system instance displays as "STOPPED" before you start the instance.
Recover from safe mode on Microsoft Windows systems:
If the Microsoft Windows machine migration gets stuck in the Launching state for an hour or more, then do the following steps on the target system
  1. Log in to Windows virtual machine.
  2. On Windows command prompt, run the following command:
    C:\IBM\maestro\maestro.deployment.ui\zero stop
  3. Click Start > Programs > IBM Tivoli Monitoring > Manage Tivoli Monitoring Services.
  4. Right-click Monitoring Agent for Windows OS and select Stop.
  5. Right-click Monitoring Agent for Workloads and select Stop.
  6. Open Task Manager and end the following processes:
    • Right-click the two Python processes (python.exe) individually and click End task.
    • Right-click the two IBM Java processes (java.exe) individually and click End task. To verify the process file location, right-click the process and click Open file location.
  7. Take a backup of the C:\IBM\maestro\agent folder.
  8. Delete the following folders:
    • C:\0config\itlm\foundation
    • C:\0config\safemode
    • C:\IBM\maestro\agent\safemode
  9. Restart the virtual system instance (VSI) from the IBM Cloud Pak System Software user interface.
    Navigate to the Virtual System Instance page, locate the specific virtual system instance, and click Stop and Start in the right panel.
Ensure that the status of the virtual system instance displays as "STOPPED" before you start the instance.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSFQSV","label":"IBM Cloud Pak System Software"},"ARM Category":[{"code":"a8m0z000000cwm2AAA","label":"Product Components"}],"ARM Case Number":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"All Versions"}]

Product Synonym

PureApp; Cloud Pak; PureApplication;

Document Information

Modified date:
25 July 2022

UID

ibm10960033

Page Feedback