Troubleshooting
Problem
The following problems are sometimes observed for some instances on the Virtual System Instances page:
- The pattern status on the Virtual System Instances page continues to show the status as "Error/Launching/Stopping/Stopped/Starting", but virtual machines and middleware also called roles are actually in "Running" state.
- The CPU and Memory status appears as "Unknown" on the Virtual System Instances -> Virtual Machine Perspective page.
- When middleware is not shown in the Middleware perspective section of the Virtual System Instances page.
- When history is not up-to-date on the Virtual System Instances page.
Cause
One of the possible causes for these issues might be due to status synchronization between Platform System Manager (or PureSystems Manager) and the Maestro agent that are running on those virtual machines.
Resolving The Problem
You can resolve them by restarting the Maestro agent to avoid virtual machine and the middleware restart on those virtual machines. However, there is no impact on the middleware or the virtual machine.
You can restart 0config manually to synchronize the status between Maestro agent and IBM® Workload Deployer. The process to do a manual restart is applicable to Linux and AIX based instances.
- Log in on each virtual machine of the instance with this command:
- sudo -i
Note: This command requires root access. - Run the following commands one by one, and ignore any error if a command fails. For example, Process not found.
/opt/IBM/maestro/maestro.deployment.ui/zero stop
/opt/IBM/maestro/ITM/bin/itmcmd agent -f stop all
/opt/IBM/maestro/ITM/bin/cinfo -r
<< ensure all agents are in stopped state. If you find the agent(s) is still running, run the above command again and again. Proceed next only when the agent is stopped.>>
/opt/IBM/maestro/ITCAMWAS/bin/itmcmd agent -f stop all
/opt/IBM/maestro/ITCAMWAS/bin/cinfo -r
<< ensure all agents are in stopped state. If you find the agent(s) is still running, run the above command again and again. Proceed next only when the agent is stopped.>>
/opt/IBM/maestro/ITCAMDB2/bin/itmcmd agent -f stop all
/opt/IBM/maestro/ITCAMDB2/bin/cinfo -r
<< ensure all agents are in stopped state. If you find the agent(s) is still running, run the above command again and again. Proceed next only when the agent is stopped.>>
/opt/IBM/maestro/ITCAMHTTP/bin/itmcmd agent -f stop all
/opt/IBM/maestro/ITCAMHTTP/bin/cinfo -r
<< ensure all agents are in stopped state. If you find the agent(s) is still running, run the above command again and again. Proceed next only when the agent is stopped.>>
/opt/IBM/maestro/ITCAMWMB/bin/itmcmd agent -f stop all
/opt/IBM/maestro/ITCAMWMB/bin/cinfo -r
<< ensure all agents are in stopped state. If you find the agent(s) is still running, run the above command again and again. Proceed next only when the agent is stopped.>>
/opt/IBM/maestro/ITCAMWMQ/bin/itmcmd agent -f stop all
/opt/IBM/maestro/ITCAMWMQ/bin/cinfo -r
<< ensure all agents are in stopped state. If you find the agent(s) is still running, run the above command again and again. Proceed next only when the agent is stopped.>>
/opt/IBM/maestro/ITCAMBN/bin/itmcmd agent -f stop all
/opt/IBM/maestro/ITCAMBN/bin/cinfo -r
<< ensure all agents are in stopped state. If you find the agent(s) is still running, run the above command again and again. Proceed next only when the agent is stopped.>>
ps -ef | grep exec_vm_tmpl | grep -v grep | awk '{print $2}' | xargs kill -9
ps -ef | grep alpine | grep -v grep | awk '{print $2}' | xargs kill -9
ps -ef | grep 9_agent | grep -v grep | awk '{print $2}' | xargs kill -9
pushd /0config
./0config.sh - Wait for the agent to start.
- Verify the status by running these commands:
ps -ef | grep -i cold
The output can be as follows:
"root * * * * /opt/IBM/ibm-java-x86_64-70/bin/java -Xms32M -Xmx1024M -javaagent:lib/bootstrap-agent.jar -jar lib/com.ibm.alpine_1.0.jar --cold"
tail -f -n 20 /0config/0config.log
The output can be as follows:
Note: Refer to the statement "update status to RUNNING" in the following output:******************************************************************************* ACCEPT tcp opt in eth0 out * ::/0 -> ::/0 tcp spt:20000 state RELATED,ESTABLISHED private is 0.0.0.0/0 iptables_cmd is ip6tables --iptables_cmd is ip6tables remove matched rule command to run: ip6tables -D OUTPUT -p tcp -d 0.0.0.0/0 --dport 9443 -j ACCEPT -o eth0 -firewall rule command to run: ip6tables -v -I OUTPUT -p tcp -d 0.0.0.0/0 --dport 9443 -j ACCEPT -o eth0 ACCEPT tcp opt in * out eth0 ::/0 -> ::/0 tcp dpt:9443 --firewall rule command to run: ip6tables -D INPUT -p tcp -s 0.0.0.0/0 --sport 9443 -m state --state RELATED,ESTABLISHED -j ACCEPT -i eth0 ---firewall rule command to run: ip6tables -v -I INPUT -p tcp -s 0.0.0.0/0 --sport 9443 -m state --state RELATED,ESTABLISHED -j ACCEPT -i eth0 ACCEPT tcp opt in eth0 out * ::/0 -> ::/0 tcp spt:9443 state RELATED,ESTABLISHED ln: failed to create symbolic link ‘/opt/IBM/maestro/agent/usr/servers/Registry_Node.11572495217964/python’: File exists update status to RUNNING CURL_SSL_OPTION = --cacert /0config/cert.tmp/cert_1.pem updateURL: inputURL = https://[fd8c:215d:178e:999:290:fa71:fabe:312a]:9443/services/deployments/d-a7ea92b4-7f43-4d1a-ab2d-ede4230a7385/command updateURL: newURL = https://IBMWorkloadDeployer:9443/services/deployments/d-a7ea92b4-7f43-4d1a-ab2d-ede4230a7385/command newURL = https://IBMWorkloadDeployer:9443/services/deployments/d-a7ea92b4-7f43-4d1a-ab2d-ede4230a7385/command X-IWD-Authorization : { "userdata" : { "tokens" : [ "eyJhdHRyaWJ1dGVzIjoie1wib3duZXJQcm9jZXNzVHlwZUlEXCI6XCJBR0VOVFwiLFwidXNlcklEXCI6XCJkLWE3ZWE5MmI0LTdmNDMtNGQxYS1hYjJkLWVkZTQyMzBhNzM4NVwiLFwiYXV0aG9yaXphdGlvbkF0dHJpYnV0ZXNcIjp7XCJyb2xlc1wiOltcIjEyXCJdfSxcImlzc3VlclByb2Nlc3NUeXBlSURcIjpcIkFETUlOXCIsXCJvd25lclB1YmxpY0tleVwiOlwiQUdFTlRcIixcImlzc3VlclB1YmxpY0tleVwiOlwiQURNSU5cIixcImRlcGxveW1lbnRJRFwiOlwiZC1hN2VhOTJiNC03ZjQzLTRkMWEtYWIyZC1lZGU0MjMwYTczODVcIixcInR5cGVcIjpcImFnZW50XCJ9Iiwic2lnbmF0dXJlIjoiQkowOE5lTGJKcHZ3bk94ZG1pRFB2VHBOOWhORldxWGUweVFQY2N6NWt5NVFcL1hFRU5MWHNIMjVXS0ttaW5ZdnpRWG0rVFRVVUtLSzBTaVwvcENnbTJaZjg4VGhsMEJrZDVwcVlmcTRjQVVYemZxNlp0XC9tRm84T2FYNjl6SWcyUTB4MmR4MWlvOEFYZlhST09GVENGWWZCSUkwNzEyU1FYbDZONXpDbnByVXNGdWFLdm40YktmeEhnc2JzckZiV2tkVnhvWWJqVWdjZlRqMWpPTmJ6R1R1amRxZG9Fb3BoZ3hXZzRhY3EwNTloc2lXXC9TeVY1bXZxMkZ2SW9DU3FhalBhcnFkb3BDY0pxY01cLzYzQndpMXNHa3V6ZkJMNEE5c2tIUTRVcTJKRjlWemNrYVRCeHhicWNSbG01VDV4SERudzhjSEFDUVB4RjYrdk5OV1lPQmJGM0E9PSJ9" ], "date" : "Thu, 31 Oct 2019 05:44:43 GMT" }, "signature" : "WbzVwk6EkJPIy4Q14BfmVu/nQimjLQ38aLw1+g2FDnVrrXIs1INFmq52kSyxEeCkguYmUvkqZncXDhCa2jA/9eX6XD6zPSIrYN4ahvoZFgdP3o32CvGVVJkXXwC/if7mTOHOG5fVZnnuBhsAc5q6w30yImr6/SHQCD5QgTc5JJidYgPFtagDOJafxHC2Q2Y7uO1+HUJEReowEQvBrKH5dcug4qC6Rq7Ynt3p6UYgUSU6O5PZ5XLyiKkobArJX/Nin5BNSMFOKRUXNH76wb0WZ4lSoESH4UaKFImKgqZ0ERcVFxakxHqFLWTddnR9IU4vGlJTobnsSVTxNJeEPsNYNw==" } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 95 0 0 100 95 0 342 --:--:-- --:--:-- --:--:-- 342 ******************************************************************************* - Verify the pattern instance status on the Platform System Manager (or PureSystems Manager) web console.
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
11 September 2020
UID
ibm11103763