Troubleshooting PowerVM NovaLink

Learn how to troubleshoot problems that might occur with PowerVM NovaLink in IBM® Power systems.

About this task

The following table lists possible errors and ways to recover.

Table 1. Known problems and solutions for PowerVM NovaLink in Power servers
Problem Solution
The ping test fails during PowerVM NovaLink installation. Check the following parameters:
  • The correct adapter is selected and the selected adapter is wired to the switch.
  • The IP address that you have specified (gateway, Netmask, and so on) is correct.
  • In the SMS menu of POWER9 or Power10 systems, you can list all devices and test each device for connectivity to check which adapter is connected.
You are unable to load the operating system (OS) image in RAM-based file systems (ramfs) when you install the OS from a network by using the PowerVM NovaLink installer. Perform one of the following actions:
  • For the Ubuntu BootP server, check whether the bootptab attribute has an entry.
  • For the Ubuntu operating system, check whether vmlinux, netboot_initrd.gz, and filesystem.squash attributes have an entry.
  • For the CentOS or Red Hat® Enterprise Linux® (RHEL), check whether vmlinuz and initrd.img attributes have an entry.
You receive an error when you create a VIOS profile in the managed system. You can specify the msp attributes only when the managed system supports active partition mobility. Enable active partition mobility and then create a VIOS profile.
Information missing in BOOT_IMAGE= parameter in the VIOS or PowerVM NovaLink repository folder.
  • Check whether you have added a comment in any line in the GNU GRand Unified Bootloader (GRUB) file where the backslash ("\") marks the break for continuity.
  • If the line contains any special characters in the GRUB file, copy the GRUB file and edit it by using a plain vi editor as a precautionary step.
You encounter a disk-specific issue when you install VIOS.
  • Check whether hdisk0 has more than 100 GB disk space.
  • Check whether unsupported disks such as Pure Storage are attached.
You are installing the PowerVM NovaLink by using the ISO file and receive the following log entry:

Creating logical volume novalinklv on VIOS id 2 Unable

 Check whether an unsupported storage is attached to a central electronics process (CEC). Remove the storage and reinstall PowerVM NovaLink from the beginning. For more information about how to install extra drivers on VIOS for storage (that is not supported by VIOS), refer to Installing PowerVM NovaLink.
You receive the following error when you install PowerVM NovaLink: http://localhost:12080/rest/api/uom/VirtualIOServer: ReadTimeout for GET http://localhost:12080/rest/api/uom/VirtualIOServer: HTTPConnectionPool(host='localhost', port=12080): Read timed out. (read timeout=1200).

This error stops the download process of the latest release files from Ubuntu mirror. 		The DNS IP is not reachable or not working.
A postinstallation failure occurs when you run the pvmctl command in RHEL. Check the /var/log/novalink-post.log file to view the installation step that caused the failure and correct the issue.

You receive the following error when you download metadata for the PowerVM NovaLink repository: 

Curl error (7): Couldn't connect to server for:
http://public.dhe.ibm.com/systems/virtualization/Novalink/rhel/86/
novalink_2.1.0/repodata/repomd.xml
 Check whether the user can access the URL from the browser and choose one of the following options:
  • If the website is accessible, check the protocol and change the URL from HTTPS to HTTP or vice versa.
  • If the website is not accessible, specify the right URL in the pvm.repo file.
You receive the following error in the pvmctl sys list:

Unable to setup logging. Please check permissions on log directory. Error: [Errno 13] Permission denied: u'/var/log/pvm/pvmctl.log'

 Check the permission of the pvm folder and whether it contains the following file paths:
  • ll /var/log/pvm/
  • ll /var/log/
If these file paths are available, restart the PowerVM NovaLink software.
Note: Always restart PowerVM NovaLink after you run a yum update or after you install the PowerVM NovaLink software.
You receive the following error:

mkvterm --id 6 Unable to set up SLIP connection to /var/run/5250_lp6.

SLIP_set_disc(1): Invalid argument Cleaning up... vterm closed

 Check whether the kernel-modules-extra package is installed. If it is not installed, install it. The kernel-modules-extra package must be same as the Kernel that is loaded. You can check it by using the uname-r command.
You receive the following error:
pvmctl --version
/usr/bin/env: 'python': No such file or directory
 Check whether the Python path is set. Check ls /usr/bin/python filepath or look for alternatives such as --config python. The following programs provide Python option:
Selection    Command
-----------------------------------------------
*+ 1           /usr/libexec/no-python
   2           /usr/bin/python3
   3           /usr/bin/python3.9
Enter + to keep the current selection or type the selection number.
The pvmctl --version command shows the following error:

ModuleNotFoundError: No module named 'monotonic'
  1. Check whether all relevant Python packages for PowerVM NovaLink are installed in the respective Python folder. For PowerVM NovaLink version 2.0.1 to 2.0.3.1, the following Python packages must be installed:

    • RHEL: /usr/lib/python3.6/site-packages

    • Ubuntu: /usr/lib/python3.6/dist-packages

    For PowerVM NovaLink version 2.1.0 and later, the following Python packages must be installed:

    • RHEL: /usr/lib/python3.9/site-packages

    • Ubuntu: /usr/lib/python3.9/dist-packages

  2. Check whether the pvm.repo file has priority=1 value to ensure that the packages are pulled from the PowerVM NovaLink repository, if available.
  3. Check whether it is an upgrade from PowerVM NovaLink version 2.0.3.1, or earlier, to version 2.1.0, or later.
    Note: If it is an upgrade, you must perform a cleanup of Python 36 packages and install new Python 36 packages. For the installation procedure, see Upgrading PowerVM NovaLink software on an RHEL partition.
After the PowerVM NovaLink is installed, the Secure Shell Protocol (SSH) access to the logical partition (LPAR) of PowerVM NovaLink is lost. Check the adapter names and the adapter for which the IP is configured. Before installing PowerVM NovaLink, the adapter names would be env3 and env6 and after installation it changes to ibmveth3 and ibmveth6. Accordingly, change the IP address configuration.
You receive the following error in the pvmctl vios list command output:

Unable to connect to the NovaLink services. Check to make sure the pvm-core and pvm-rest services are running (systemctl status), and verify that this partition has PowerVM management authority. If you are currently updating pvm-novalink, you may see this error. Wait for the update to finish, then try again.

 If you receive this error for a long duration, complete the following steps:
  1. Stop the firewalld status and disable it by using the following commands:
    • systemctl stop firewalld
    • systemctl disable firewalld
  2. Enable the sestatus attribute by completing the following steps:
    1. Edit /etc/selinux/ and set SELINUX=disabled or SELINUX=permissive.
    2. Reboot the PowerVM NovaLink LPAR.
The yum update command fails in PowerVM NovaLink version 2.0.3.1 that is attached to PowerVC version 2.0.3.1.
  • To reduce python3-stevedore, python3-pbr, python3-six issues, complete the following steps:
    1. Set priority=1 in /etc/yum.repos.d/pvmrepo.
    2. Set subscription-manager release --set=8.6
  • Before you run the yum update command, perform a version lock for python3-amqp and python3-kombu attributes by using the following commands:
    • yum install python3-dnf-plugin-versionlock
    • yum versionlock python3-amqp
    • yum versionlock python3-kombu
  • For all future updates of PowerVM NovaLink, remove the version locks by using the yum versionlock clear command.
The pvmctl vm migrate --mgmt-svr 10.200.0.49 -i id=9 –validate command fails with the SSH key exception. Check whether the SSH keys are shared between the two PowerVM NovaLink versions. Check the following keys in both PowerVM NovaLink versions:
  • pvmctl sys list-auth-keys
  • pvmctl sys list-public-key
To share the keys with two versions of PowerVM NovaLink, run the following command: pvmctl sys add-auth-key --key "$(ssh neo@<Other Novalink IP> pvmctl sys list-public-key)"
The Remote Management and Control (RMC) connection does not become active. To ensure that the RMC connection becomes active, complete the following steps:
  1. Check whether IPv6 is enabled for a private network (4094 VLAN) and the following commands are set in pvmutil –o getrncstatus :
    • pvmutil -o getrmcstatus
    • lpar_id=1,lpar_env=aixlinux,rmc_state=none,rmc_ipaddr=
    • lpar_id=2,lpar_env=vioserver,rmc_state=active,rmc_ipaddr=192.168.128.2
    • lpar_id=3,lpar_env=aixlinux,rmc_state=active,rmc_ipaddr=fe80::743f:9eff:feb8:d259
    Note: If an IP address is not assigned to a private network, the IPv6 MAC address is used in the following format: ping –I <Int-Name> <IPv6 MacAddress>. For example, ping -I ibmveth6 fe80::743f:9eff:feb8:d259.
  2. Check whether the ping test works from both ends.
  3. Restart RSCT.

    For information about troubleshooting the No RMC Connection Error, refer to Fixing the No RMC Connection Error.
You have an issue with the ISO-based installer. Create a customer ticket with the following information:
  • The PuTTY log file.
    To extract this log file, complete the following steps:
    1. Access the PuTTY Reconfiguration dialog and click Logging > Printable output.
    2. Enter a file name and location for the log.
    3. Click Apply.
  • CEC details along with information about the Flexible Service Processor (FSP) IP, PowerVM NovaLink IP (also known as neo), VIOS IP, and details about the connected adapters.
You have an issue with the PowerVM NovaLink Installer 2.0. Create a customer ticket and send the /var/log/novalinkAutoInstaller/<log_file> log location along with the PuTTY log and CEC details.
You have other issues with PowerVM NovaLink. Run the pedbg –c –q 4 command and provide the log details. Provide at least one log file from the /var/log/pvm folder.
Table 2. Known problems and solutions for PowerVM NovaLink in specific Power servers
IBM Power systems Problem Solution

POWER8 and POWER9

 The Intelligent Platform Management Interface (IPMI) prompts are failing.
Check whether IPMI passwords are set. To change the password, complete the following steps:
  1. Log in to the Advanced System Management Interface (ASMI) tool.
  2. Click Profile > Change password.
  3. Enter your user ID and the new password.
  4. Click Save.

POWER8, POWER9, and Power10

 You cannot view the IPMI prompts.
To fix this issue, complete the following steps:
  1. To check whether the Hardware Management Console ( HMC) is disconnected, go to ASMI > System Configuration > Hardware Management Consoles.
  2. Reset the server to a non-HMC managed configuration.
  3. Go to ASMI > System Configuration > Console Type and select IPMI as the console type.
  4. In the Power On/Off System field, select Boot to SMS Menu and click Save.
    • For POWER8 and POWER9 systems, select Running Auto Start Always.
    • For Power10 system, select Auto-start only

POWER8 and POWER9

 You cannot create a VIOS profile.

In the ASMI tool, click On Demand Utilities > VET Capability Settings > 2 RPA I/O hosting and set the value of 2 RPA I/O hosting to true.

POWER9 and Power10

 During the PowerVM NovaLink installation, the VIOS logical partition cannot be powered on.
Complete the following steps:
  1. To view the VIOS list, run the pvmctl vios list command. An example of the output follows:
    / # pvmctl vios list
Virtual I/O Servers
+-------+----+--------------+----------+--------------+-------+-----+-----+
|  Name | ID |    State     |   RMC    |   Ref Code   |  Mem  | CPU | Ent |
+-------+----+--------------+----------+--------------+-------+-----+-----+
| vios1 | 2  |    error     | inactive | B2001230 LP> | 10240 |  2  | 1.0 |
| vios2 | 3  | not activat> | inactive |   00000000   | 10240 |  2  | 1.0 |
+-------+----+--------------+----------+--------------+-------+-----+-----+
/ #

  2. Ensure that enough central processing units (CPUs) and memory is available that can be assigned to the VIOS.

  3. Ensure that the VIOS is not a part of an enterprise pool. For example, in the following output, VIOS is part of a metered pool (ep2.0):
    / # lssyscfg -r sys -F metered_pool_id2022/ #
    In this case, remove the system from the metered pool and activate both processor and memory before installing the PowerVM NovaLink software.

Power10

 You receive the following error in log files:
--- Creating client adapter for NovaLink
Error creating virtual ethernet adapter
--- Creating virtual ethernet for NovaLink

Check whether the pvm_vtpm attribute is disabled. For more information on how to disable this attribute, refer steps 7 and 8 in Installing the PowerVM NovaLink 2.0.3.1 software on scale-out and mid-range IBM Power Systems.

Power10

 The Host Console operation gets stuck at standby mode when installing the PowerVM NovaLink software. To fix this problem, complete the following steps:

  1. Access the Redfish API endpoint: https://<BMC IP>/redfish/v1/Systems/system/Bios

  2. Disable the following BIOS attributes:
    • pvm_hmc_managed
    • pvm_stop_at_standby

    For more information about how to disable the BIOS attributes, refer steps 7 and 8 in Installing the PowerVM NovaLink 2.0.3.1 software on scale-out and mid-range IBM Power Systems.

Power10

 You cannot create a VIOS profile.

Check whether the system is capable for the VIOS in the BMC tool by clicking Resource Management > Capacity on Demand > Virtual I/O server capable.