IBM Spectrum Protect Plus: Migrating vSnap server to 10.1.12

Steps

Note- Follow the procedure carefully. If not followed, this procedure can result in a loss of data.

To update a vSnap server, complete the following steps:

1. Collect the vSnap server hardware and storage pool information:

1. The virtual vSnap servers running on VMware or Hyper-V, note the following properties of the virtual machine by using the VMware vSphere UI or Hyper-V Management Console.
   • CPU configuration
   • Memory configuration
   • Network adapter configuration
   • Hard disk configuration
   Alternatively, you can use VMware PowerCLI to collect the virtual machine information. For example:

   PS /Users/spike> get-vm -Name "vSnap-VM" | fl  
   PS /Users/spike> get-vm -Name "vSnap-VM" | get-harddisk | fl
   PS /Users/spike> get-vm -Name "vSnap-VM" | Select-Object -Property Name,@{Name=’Cluster’;Expression={$_.VMHost.Parent}}
   PS /Users/spike> get-vm -name "vSnap-VM" | Get-NetworkAdapter
   PS /Users/spike> (get-VM "vSnap-VM").Guest.Nics | fl

   Note: You can skip this step for physical vSnap servers that are reinstalling RHEL 8 on the same hardware.
2. Using Secure Shell (SSH), log in to the vSnap server as the serveradmin user.
3. Note the server network hostname by using the following commands:

    # Short hostname 
   hostname -s

   # Fully qualified domain name
   hostname --fqdn

4. To check the vSnap server network configuration, run the following command:

   nmcli device show

   Alternatively, examine the contents of the following configuration files:

   /etc/sysconfig/network-scripts/ifcfg-*

5. To check the vSnap server version and operating system, run the following command:

   vsnap system info

6. To check the vSnap pool information and determine the list of disks that are part of storage pool, run the following commands:
   1. To collect the general pool information, run the following command:

      vsnap pool show

   2. To identify the disks that are part of storage pool, run the following command:

      vsnap disk show

      Note: The disks in the pool are labeled as vsnap_pool or crypto_LUKS (for encrypted pools). Compare the disk information collected by both commands to ensure that the information is correct. The storage pool comprises all data disks, optional log, and cache disks.
   3. To collect the information about disk partition and identify which disks are in use, such as LVM or multipathing, run the following command:

      vsnap disk lsblk

   4. Additionally, to determine the number of volumes and snapshots in the pool, run the following commands:

      vsnap volume show | wc -l
      vsnap snapshot show | wc -l

      Note: You can use this information to verify pool contents after the migration is complete.
7. If multipath disks are attached to the vSnap server, which multipathd service is enabled, make a manual backup copy of the configuration file by using the following command:

   /etc/multipath.conf

   Note: Transfer the backup copy to another system by using Secure Copy Protocol. You can rename the backup file to make it easier to identify. This renaming is useful when migrating multiple vSnap servers.

2. Pause all jobs in IBM Spectrum Protect Plus

1. Log on to the IBM Spectrum Protect Plus server.
2. Jobs must not be actively running or scheduled to run during the migration procedure. Pause the schedule for all jobs to ensure that they do not attempt to run while the migration is occurring. Click Jobs and Operations > Schedule and then click Pause All Jobs.
3. Verify that no jobs are running by clicking Jobs and Operations > Running Jobs. If one or more jobs are running, wait for them to complete.

3. Optional: Temporarily remove replication partnerships: 
If the hostname or IP address of the migrated vSnap server changes, follow those steps to remove the replication partnership during the migration period.

1. Click System Configuration > Storage > vSnap servers, select the vSnap server.
2. Click Manage and then open the Storage Partners tab. Note the name of each replication partner for later use.
3. Click Remove to remove the replication partner.
   Note: Document the IP address and hostname of each replication partner for the future reference. Removing the partnerships does not affect the replication data. The partnership can be re-created in after the migration is complete. If the vSnap server has multiple IP addresses, and even one of the IP addresses changes during the migration, then you must remove and re-create the partnerships.
   If you are using the same hostname and IP address on new appliance, then proceed step (4) without removing replication partnerships.

4. Optional: Temporarily remove VADP proxy on the vSnap server:
If the vSnap server is registered as a VADP proxy, follow those steps to unregister a VADP proxy on a physical or virtual vSnap server.

1. In the navigation panel, click System Configuration > VADP Proxy and select the proxy associated with the vSnap server that is being migrated.
2. Make a note of the site under which the proxy is registered.
3. Click Proxy Options and then click the VADP proxy that you want to uninstall.
4. Click Unregister to remove the proxy from the IBM Spectrum Protect Plus server.
   • Note:
     • 1. Ensure that you document all proxy settings before you unregister VADP proxy from vSnap server for the later use.
     • 2. If the vSnap server is not registered as a VADP proxy, then proceed step (5).

5. Stop vSnap services, export the storage pool, and back up the vSnap configuration.

1. Using Secure Shell (SSH), log in to the vSnap server as the serveradmin user.
2. Disable the vSnap services by using the following command to ensure that the services do not start automatically on boot.

   sudo systemctl disable vsnap

3. Reboot the vSnap server by using the following command. This reboot is required to ensure that the storage pool is unmounted and the rest of the system is in a clean state before the configuration is backed up.

   sudo reboot -n

4. Wait for the reboot to complete. After the system is back online, login as the serveradmin user and run the following command to view the status of all services. Verify that the status of all services is listed as inactive.

   vsnap_status

5. Upload the script vsnap_config_backup.sh to the vSnap server by using Secure Copy Protocol and place it under /home/serveradmin or another suitable directory like /tmp. Make the script executable file by going to the directory where it is located and running:

   chmod +x vsnap_config_backup.sh

6. Create a backup of the vSnap configuration. The -f argument to the script specifies the full path to the output file where the backup is created. In the following example, the output is created under /home/serveradmin. You can also specify any other suitable directory like /tmp. The output of the script is piped to the tee command to capture it a log file for troubleshooting.

   sudo ./vsnap_config_backup.sh -f /home/serveradmin/vsnap_config_backup.tar.gz | tee -a /home/serveradmin/backup.log

   Verify that the output of the script ends with the following lines, which indicate that the backup was successful:

   *** CONFIG BACKUP SUCCESSFUL ***
   Generated output file: /home/serveradmin/vsnap_config_backup.tar.gz

7. Transfer the backup copy to another system by using Secure Copy Protocol. You can rename the backup file to make it easier to identify. This renaming is useful when migrating multiple vSnap servers.
8. Shut down the vSnap server:

   sudo shutdown

6. Detach the storage pool disks from the vSnap server.
Identify the storage disks that comprise the storage pool (including all data disks, optional log, and cache disks) and detach them from the vSnap server based on the disk information gathered in step (1)(f). You can reattach them to the new vSnap server after the migration is complete.

Note: Disks associated with the vSnap cloud cache (/opt/vsnap-data partition) do not need to be migrated and the content does not need to be copied to the new vSnap. The cloud cache will be freshly initialized on the new vSnap after migration.

IMPORTANT: For a virtual vSnap server version 10.1.10 and earlier that are running on VMware OVA or Hyper-V image, a default 100 GB unused virtual disk is included in the OVA or Hyper-V image. In addition, you can attach one or more large SAN disks to expand the storage pool. For example, if the vSnap server contains a default 100 GB unused virtual disk and later if one or more physical disks are attached to expand the storage pool, then now the storage pool consist of 100 GB virtual disk plus the large physical disks. Check the information gathered in step (1)(f) to confirm whether the virtual disk is still included in the storage pool. All disks that are included in storage pool must be migrated.

The steps to detach disks from the vSnap server can vary depending on the type of the vSnap server, the backing storage, and the method in which they are attached. The most common scenarios are described here. Consult your storage administrator for more details or other scenarios not covered in this document.

1. Virtual disks or RDM disks attached a virtual vSnap server running in VMware.
   1. Log in to the VMware vSphere UI and go to the vSnap virtual machine.
   2. Right-click the vSnap virtual machine and select Snapshots > Manage Snapshots. Ensure that there are no snapshots present on the VM. If one or more snapshots are present, delete them, and wait for the disk consolidation to complete.
   3. Right-click the vSnap virtual machine and select Edit Settings. Before you remove the hard disk from virtual machine, make a note of the backing details (datastore name, file path for virtual disks and device identifier for RDM disks). These details are needed when you reattach the same disks to the new vSnap virtual machine. Select the row that contains hard disk details, and click the "X" icon to remove it from the VM.
   WARNING: When you remove a hard disk from the VM, DO NOT select the checkbox labeled "Delete files from datastore". Selecting this option permanently deletes all data, which cannot be recovered.
2. Virtual disks attached to a virtual vSnap server running in Hyper-V.
   1. Log in to the Hyper-V Manager console and go to the vSnap virtual machine.
   2. Select the vSnap virtual machine and go to the Snapshots window. Ensure that there are no snapshots present on the VM. If one or more snapshots are present, delete those snapshots, and wait for the disk consolidation to complete.
   3. Right-click the vSnap virtual machine and select Settings. Before you remove the hard disk from virtual machine, make a note of the backing details (location and full path of the VHD file). These details are needed when you reattach the same disks to the new vSnap virtual machine. Click Remove to remove the disk from the VM.
3. SAN disks attached to a vSnap server by using Fibre Channel or iSCSI:
   On the SAN array, detach the LUNs from the vSnap server host. Refer to the documentation for your specific SAN array for detailed instructions, or consult a storage administrator.
4. Local disks directly attached to a physical vSnap server:
   If possible, physically detach the disks from the system or disconnect them in the local RAID controller configuration.
   Note: When you reinstall RHEL 8 on the same physical server, the disks must be detached from the vSnap server. When you reinstall the operating system, ensure that the contents of the pool disks are preserved. The pool disks must not be formatted or overwritten in any way.

7. Deploy and configure new vSnap server running 10.1.12

1. When you are migrating a physical vSnap server, install a supported version of RHEL 8 on the server, and then install the vSnap software.
2. When you are migrating a virtual vSnap server, deploy a new vSnap server running version 10.1.12 using the VMware OVA or Hyper-V. Before you deploy a new appliance, POWER OFF and rename the old appliance to avoid confusion (for example, by adding the suffix "old" to the name of the older server). Modify the virtual machine CPU and memory configuration to match the values noted earlier in Step (1)(a).
   Note: The new virtual vSnap server running version 10.1.12 does not contain a default 100 GB unused disk for the storage pool.
3. Log in to the vSnap server by using SSH. For virtual deployments, the default username is serveradmin and the password is sppDP758-SysXyz. Change the password when prompted. If the serveradmin account is used to register the vSnap server into IBM Spectrum Protect Plus, ensure that the new password on the new server matches the old password.
4. Run the following commands to stop and disable vSnap services on the new vSnap server. This disabling ensures that the new vSnap is not accessible to IBM Spectrum Protect Plus server. The services are enabled after the migration is complete.

   sudo systemctl stop vsnap
   sudo systemctl disable vsnap

   Note: Check the status of all services by running the command vsnap_status and ensure that all services are in inactive state.
5. Customize the network configuration to specify hostnames and IP addresses. If you are using the same hostnames and IP addresses as the old vSnap, refer to the values noted in step (1)(d) earlier. For virtual vSnap servers deployed in VMware, you can customize the network configuration as part of the deployment wizard, or through the operating system after the server is powered on. For other vSnap servers (physical or Hyper-V deployments), you can customize the network configuration through the operating system.
6. Customize the configuration of the disk partitions to match the values previously noted on the old server in step (1)(f). This action applies to the root partition, there is /opt/vsnap-data  partition (that is, the cloud cache), and any other partitions you configured on the old server. The sizes of the partitions on the new server must match the sizes on the old server.

8. Restore vSnap configuration to the new vSnap server.

1. Go to the vSnap configuration backup file created in step (5)(g). Transfer the backup file to the new vSnap server by using Secure Copy Protocol, and place it under /home/serveradmin or another suitable directory like /tmp.
2. Upload the script vsnap_config_restore.sh to the new vSnap server by using Secure Copy Protocol and place it under /home/serveradmin or another suitable directory like /tmp. Go to the script, and run the following command:

   chmod +x vsnap_config_restore.sh

3. Initialize the vSnap software without creating a pool by using the command:

   vsnap system init --skip_pool

   Restore the vSnap configuration. The -f argument to the script specifies the full path to the file where the backup file is located. In the following example, the backup file is under /home/serveradmin. The output of the script is piped to the tee command to capture the log file for troubleshooting.

   sudo ./vsnap_config_restore.sh -f /home/serveradmin/vsnap_config_backup.tar.gz | tee -a /home/serveradmin/backup.log

   • As part of the configuration restore process, the script re-creates vSnap user accounts that existed on the old vSnap. For each account on the old vSnap, the script checks if the same account exists on the new vSnap. If the account exists (for example, serveradmin), then it is skipped. If the account does not exist, the script re-creates it, and prompts you to enter the password. It is recommended that you specify the same password that is used on the old vSnap. If you want to skip creating the user account and continue with the rest of the restore, press Ctrl-C at the password prompt. If you skip creating the user account, you can still manually create it using command vsnap user create after the restore is complete.

   • You can verify the output of the script, and check whether the script ends with the following message, which indicates that the restore process is successful:

     *** CONFIG RESTORE SUCCESSFUL ***
     
     It is safe to proceed to the next steps

   • The configuration restore can fail if the storage pool disks are attached to the system. If the restore fails, the script shows the following failure message:

     One or more pool disks (listed above) are currently attached.
     Restoring the configuration while pool disks are attached can
     result in unintended deletion of data if there is a mismatch
     between the configuration and the contents of the disks.
     
     Detach the existing pool disks and restart the system, then retry the restore operation.
     The disks should be reattached after the restore is complete.
     
     Verifying no pool disks are attached ... [FAILED]

   • It is strongly advised that you detach the pool disks before you attempt the restore. However, for physical vSnap servers with direct-attached storage where you are unable to easily detach the disks, you can specify an optional flag -d to ignore the attached disks and proceed with the restore anyway. If you ensure that you followed step (7)(d) to fully stop and disable vSnap services before you attempt the restore, then it is safe to use the -d flag.
   • For example, the following command shows the optional flag being set:

     sudo ./vsnap_config_restore.sh -d -f /home/serveradmin/vsnap_config_backup.tar.gz | tee -a /home/serveradmin/backup.log

   • In this case, the output of the script contains the following message:

     Ignoring existing disks because the -d flag is specified.

4. If the configuration restore is successful, run the following command to enable the vSnap service. This enabling ensures to start the service automatically after the next reboot.
   Note: Enable the service, but do not start it yet.

   sudo systemctl enable vsnap

5. Shut down the vSnap server:

   sudo shutdown

• Note: If the vSnap server is running as a virtual machine, make sure you POWER OFF the VM.

9. Reattach the storage pool disks to the new vSnap server:
Reattach the pool disks to the new vSnap server (including all data disks, optional log, and cache disks). The steps to attach disks to the vSnap server can vary depending on the type of the vSnap server, the backing storage, and the method in which they are attached. The most common scenarios are described here. Consult your storage administrator for more details or other scenarios not covered here.

Note: Refer to step(6) for instructions on how to detach pool disks from the old vSnap server.

1. Virtual disks or RDM disks attached a virtual vSnap server running in VMware.
   1. Log in to the VMware vSphere UI and go to the new vSnap virtual machine.
   2. Right-click the vSnap virtual machine and select Edit Settings.
   3. For each virtual disk to be added, select Add New Device > Existing Hard Disk. For each RDM disk, select Add New Device > RDM Disk. Specify the path to the existing virtual disk or RDM device as previously noted in step (6)(a).
   Note: For virtual disks, setting the disk type to 'Independent' is strongly advised. Refer to the IBM Spectrum Protect Plus Blueprint for details and best practices about virtual disk configuration.
2. Virtual disks attached to a virtual vSnap server running in Hyper-V.
   1. Log in to the Hyper-V Manager console and go to the new vSnap virtual machine.
   2. Right-click the vSnap virtual machine and select Settings.
   3. For each virtual disk to be added, select SCSI Controller > Hard Drive > Add, then select Browse and locate the existing VHD file as previously noted in step (6)(b).
3. SAN disks attached to a vSnap server by using Fibre Channel or iSCSI.​​​​​
   • On the SAN array, configure a host for the new vSnap server and attach the LUNs to it. Refer to the documentation for your specific SAN array for detailed instructions, or consult a storage administrator.
   • Configuring the host can require extra Fibre Channel or iSCSI configuration on the vSnap server. If it is necessary to power on the new vSnap server to configure the storage, ensure that the vSnap services are stopped by using sudo systemctl stop vsnap. After the storage is configured, shut down the server.
4. Local disks directly attached to a physical vSnap server. 
   Reattach the disks as described in previous step.
   • Note: Attach disks to the system physically or connect them in the local RAID controller configuration.

10. Start services and run health checks on the new vSnap server.

1. Using Secure Shell (SSH), log in to the vSnap server as the serveradmin user and power on the vSnap server.
2. To check the status of vSnap services, run the command: vsnap_status
   • If all storage pool disks are intact and discovered correctly, the pool is mounted when the vsnap-data service starts. Depending on the configuration of the storage pool, it can take an extra 15 minutes for the pool deduplication table (DDT) to be loaded into memory during which time the vsnap-data service remains in activating status.
   • Ensure that all services are in active state before you proceed further.
   • If one or more services fail to start, run the command sudo vsnap_journal to examine startup messages for troubleshooting.
3. Run the following commands to view information about the system and the storage pool. In particular, verify that the system ID, pool size, and status match the information collected earlier in step (1)(f). Ensure that the number of disks are accurate. It is normal for the names of the pool disks to differ before and after migration.

   vsnap system info
   vsnap pool show
   vsnap disk show
   

4. To check the volumes and snapshots in the pool, run the following commands.

   vsnap volume show
   vsnap snapshot show
   

   • Also, verify that the number of volumes and snapshots matches the information collected earlier in step (1)(f).

     vsnap volume show | wc -l
     vsnap snapshot show | wc -l

5. Log in to the IBM Spectrum Protect Plus server and go to the list of vSnap servers, and then go to the migrated vSnap server.
6. If the hostname, IP address, or credentials of the vSnap changes after migration, edit the vSnap registration and specify the new values.
   • WARNING: Do not remove and readd the vSnap server. The vSnap address and credentials can be updated, but the vSnap need to remain registered in IBM Spectrum Protect Plus always.
7. Verify that the version is listed as 10.1.12. If the vSnap is shown as offline or shows the older version, click Refresh in the menu for that specific vSnap server. It can take a few minutes to refresh the information.

11. Verify accuracy of TLS certificate on the new vSnap server:
The vSnap servers running version 10.1.12 have a unique self-signed TLS certificate that uses the auto-detected hostnames and IP addresses. After the vSnap server migration is complete to version 10.1.12, it generates the unique certificate during the initial startup of services. In some cases, the certificate must be regenerated to detect hostnames and IP addresses. Alternatively, you can use a custom CA-signed certificate.

1. Use the following command on the vSnap server to examine the current certificate.

   vsnap system cert show --text
   

2. Take note the following lines in the command output, which shows the list of Subject Alternative Names (SAN) embedded in the certificate. By default, the SAN list is generated by auto-detecting the short hostname, full hostname (that is, the fully qualified domain name), and all IP addresses.

   X509v3 extensions:
       X509v3 Subject Alternative Name:
           DNS:vsnapsrv1, DNS: vsnapsrv1.example.com, IP Address:10.11.37.1
   

3. Log in to the IBM Spectrum Protect Plus server, go to the list of vSnap servers, and locate the entry for the vSnap server.
4. Verify that the hostname or IP address registered in vSnap matches one of the SANs in the certificate. If they do not match, then connections to the vSnap server fail when the IBM Spectrum Protect Plus server is migrated to version 10.1.12. To prevent a failure, the certificate must be replaced.
5. If there is a mismatch, use the following command on the vSnap server to regenerate a certificate by specifying a custom list of one, or more hostnames and optionally one or more IP addresses are used as the SAN list. If the vSnap server is registered in IBM Spectrum Protect Plus by using an IP address, then hostnames and IP addresses must be specified when regenerating the certificate. If the vSnap is registered by using a hostname, then the hostnames must be specified but the IP address can be omitted.

   vsnap system cert regenerate --hostnames "<comma-separated-hostnames>" --ipaddrs "<optional-comma-separated-IPs>"
   

   • A few examples:

     vsnap system cert regnerate --hostnames "myvsnap.mydomain,myvsnap.altdomain"
     vsnap system cert regnerate --hostnames "vsnapsrv1,vsnapsrv1.example.com" --ipaddrs "10.11.37.1"
     

6. As an alternative to step (e), you can also use a custom CA-signed certificate instead of generating a self-signed certificate, obtain the certificate (.crt file) and private key (.key file) and copy them to the following paths on the vSnap server. Ensure that both files are plain text files in PEM format.

   /etc/vsnap/ssl/spp-vsnap.crt
   /etc/vsnap/ssl/spp-vsnap.key
   

7. If the certificate is regenerated or replaced, restart the vSnap API by using the following command.

   sudo systemctl restart vsnap-api
   

12. Optional: Re-create replication partnerships if they were removed earlier.

This step is optional. If you previously removed the replication partnerships as described in step (3), ensure that they must be re-created. You can skip this step if the replication partnerships are not removed in the earlier step (3), and proceed to step (13).

1. In IBM Spectrum Protect Plus, go to System Configuration > Storage > vSnap servers, select the specific migrated vSnap server, and then click Manage.
2. Open the Storage Partners tab.
3. Select the previously removed partner in step (3)(c), add the partner again by clicking Add.

13. Optional: Reinstall VADP proxy if it was removed earlier.

This step is optional. If the old vSnap server is used as a VADP proxy server and if you previously removed it in step (4), ensure that you register them back in IBM Spectrum Protect Plus. If not used the old vSnap server as a VADP proxy, skip this step and proceed to step (14).

1. In IBM Spectrum Protect Plus, go to System Configuration > VADP Proxy.
2. Add the vSnap server as a VADP proxy. When prompted to select a site, specify the same site that was previously noted in step (4)(b). When prompted to enter credentials, specify the serveradmin user or another user that has password-less sudo privileges.
3. After you add VADP proxy, select 'Proxy Options', and modify the values to ensure that they match the settings previously noted in step (4)(c).

14. Final steps to start by using the migrated vSnap server.

On the vSnap server, the operating system user IDs and group IDs changes after migration. Because the backup files in the vSnap storage pool are no longer owned by the same user ID, even if the username remains the same. This can cause permission errors in catalog backup jobs. To reclaim ownership of catalog backup volumes, run the following command on the migrated vSnap server.

Note: In the following command, replace <user> with the actual username that was specified while you register the vSnap server in IBM Spectrum Protect Plus (for example, "serveradmin").

sudo /opt/vsnap/bin/resetCatalogBackupOwner.sh -u <user> 

In IBM Spectrum Protect Plus, go to Jobs and Operations > Schedule and release job schedules that were previously paused. Allow jobs to run against the newly migrated vSnap server and verify that they complete in an expected manner.

On the migrated vSnap server, the background space reclamation activity is disabled after migration. This disabling is a safety mechanism to ensure data in the pool is not accidentally deleted if errors occurred during or after the migration. The space reclamation must be manually enabled after you verified that the migration was successful and jobs are resumed.

Run the following commands on the vSnap server to resume the reclamation activity:

vsnap system pref clear --name maintServiceCleanupDisabled
sudo systemctl restart vsnap-maint