Troubleshooting a PowerVC region

Use this topic to solve issues that might occur in a PowerVC region.

Unable to create or delete volumes in PowerVC

When the openstack-cinder-powervc service fails, due to lack of SSL certificate or another issue, the openstack-cinder-volume service might also fail. To solve the problem, restart the openstack-cinder-volume service on the OpenStack Controller.

Authentication errors communicating with either keystone of PowerVC

• Confirm OpenStack credentials are correct.
• Confirm PowerVC credentials are correct.
• Confirm PowerVC SSL certificate is at /etc/pki/tls/certs/powervc.crt on the OpenStack Controller.
• Confirm that the PowerVC can communicate with the PowerVC server by the means specified by the ssl certificate. This can be by IP address, host name, or FQDN, depending on the installation of PowerVC. This is resolved by either ensuring that both OpenStack Controller and PowerVC server are set up on the same DNS server correctly. Alternatively, put an appropriate entry in the /etc/hosts file of the OpenStack Controller.

Authentication errors regarding staging user/tenant

Confirm that the staging user is a member of the staging tenant.

Live Migration does not occur

• When performing a live migration, the log file states that the resource monitoring and control is down on the virtual machine. The resource monitoring and control is a program that hardware management console uses to communicate with the LPARS. This is standard on AIX® deployments, but is not on Linux on Power®.
• Confirm that the resource monitoring and control is running on the virtual machine to be migrated. This can be confirmed by an OK health status by the nova show command or by viewing the PowerVC user interface. If the resource monitoring and control is not running on an AIX deployment, reset it. The image must also be recaptured as the resource monitoring and control often gets disabled by the Network Installation Manager deployment process.

  # /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc  -check status

  • Stopping and starting resource monitoring and control without erasing configuration:
    • # /usr/sbin/rsct/bin/rmcctrl -z - Stops the daemons
    • # /usr/sbin/rsct/bin/rmcctrl -A - Adds entry to /etc/inittab and it starts the daemons
    • # /usr/sbin/rsct/bin/rmcctrl -p - Enables the daemons for remote client connections
  If it is an image for Linux on Power, there are instructions on how to install the resource monitoring and control in the PowerVC documentation.

Volume Resize Operations do not occur or fail

SSP backed storage does not support resize of disk:

2014-11-11 04:49:34.916 4750 ERROR oslo.messaging.rpc.dispatcher [-] 
Exception during message handling: Get error: RPCException: Cinder API error: 
NV-37CDE0F The host 228b02eb0e47e611e4b5060000c9f82a76 for boot volume 
991b9732-041d-4dab-8ef8-75006216920d  does not support extend volume.

Shared Volume Controller FlashCopy operations can only occur serially per image. If it is doing a copy or extending operation on a boot volume, you cannot start a new operation to make changes to the volume. You can check the Shared Volume Controller UI to see whether a flashcopy is in progress to the target volume or the current target volume is being extended:

2014-11-11 01:24:58.006 2095 TRACE nova.openstack.common.loopingcall
ResizeError: Get error: PVCExpendvdiskFCMapException: Flashcopy is in 
progress for boot volume, volume size didn't change. Please try again later. 
(HTTP 409) (Request-ID: req-29350427-32b3-4721-baca-504c5216b041) 
during resizing the instance in the PowerVC

Changing the PowerVC user name and password on the OpenStack Controller if the PowerVC user name or password changes

1. On the OpenStack Controller, edit the following file:

   /etc/powervc/powervc.conf

   Note: Take a backup of the file before making any changes.
2. Navigate to [powervc] and locate admin_user and admin_password.
3. Change the user name and password by using admin_user and admin_password.
   Note: The password must be encrypted so use openstack-obfuscate <password> to generate.
4. Click Save.
5. Restart the following services:

   service openstack-glance-powervc restart
   service openstack-neutron-powervc restart
   service openstack-nova-powervc restart
   service openstack-cinder-powervc restart
   service openstack-cinder-volume restart

Unable to attach volumes to Instances on Shared Storage Pool backed environment

Volume attach on Shared Storage Pool backed environments require that the Resource Monitoring and Control (RMC) software is installed and running on deployed instances. If the particular instance that you are deploying never appears to have RMC running, in other words the health of the virtual machine is warning, it is necessary to reset it (relevant to all operating system types) or install it, if it was not already installed (relevant to Linux on Power). AIX must have RMC included by default.

To install the RMC for Linux on Power, follow the instructions at http://www.ibm.com/support/customercare/sas/f/lopdiags/home.html to install the service and productivity tools.

To reset RMC, perform the following commands on a running instance and then recapture by using the usual process:

  # /usr/sbin/rsct/bin/rmcctrl -z
  # /usr/sbin/rsct/bin/rmcctrl -A
  # /usr/sbin/rsct/bin/rmcctrl -p

After the previous commands are run, the status of the virtual machine in the PowerVC UI must change from warning to OK. Changing the status might take several minutes.

rstrip error in nova-powervc.log and some actions such as starting or stopping the virtual machine is not responsive

Stop and start the services in the OpenStack Controller by using SCOrchestrator.py, if the following error is displayed in the nova-powervc.log:

2014-12-17 02:57:10.348 26453 ERROR powervc.nova.driver.compute.manager [-] 
Exception: Exception: 'NoneType' object has no attribute 'rstrip'

OpenStack Controller does not support secure connection to PowerVC server with mixed-case or uppercase host name

This problem affects only secure connections; insecure connections are unaffected. The following error is displayed:

Host "hostname" does not match x509 certificate contents: CommonName "hostNAME", 
subjectAltName "DNS:hostNAME, DNS IP"

To fix this problem, complete the following steps:

1. Reconfigure the PowerVC server so that the host name is specified in lowercase letters. Include the fully qualified domain name.
2. Reconfigure the PowerVC application by using the powervc-config command.
3. Run the powervc-replace-cert command to generate a new powervc.crt file that is based on the new host name.
4. Restart the httpd service on the PowerVC server.
5. Replace the powervc.crt file on the OpenStack Controller with the new powervc.crt file generated in step 3.
6. Restart the PowerVC services on the OpenStack Controller.

Deploy single virtual server offering always produces a no valid host found error

PowerVC has a mechanism known as Storage Connectivity Groups, it is a way of grouping storage in PowerVC. Some images only support certain Storage Connectivity Groups depending on the storage types they were created from. The Deploy single virtual server offering has no knowledge of the Storage Connectivity Group and thus it always attempts to deploy an AIX or Linux on Power server to the first Storage Connectivity Group that is listed in the /etc/powervc.conf file on the PowerVC OpenStack Controller. If this Storage Connectivity Group is not supported by the image being deployed, the deployment fails with the no valid host found error message. See Heat template examples for an example of how to deploy an AIX or Linux on Power server using Heat that is able to leverage Storage Connectivity Groups.