IBM Support

TSAMP: Changing a node IP address in a running cluster

Question & Answer


Question

Changing your nodes base NIC IP address in a running/configured cluster can be very complicated, what are the steps to perform this successfully?

Answer

Due to RSCT harvesting the base NIC IP address (not to be confused with floating IP addresses, known as VIPs (virtual IP) or SIPs (service IP)), changing these addresses can be difficult. Here are the steps with explanations for each step on what should be performed.

1. Set your environment to export the cluster scope to a value of 2:

export CT_MANAGEMENT_SCOPE=2

This variable sets all cluster commands to apply to all nodes in the peer domain, without this variable set then the commands will only return information from the local node.

2. Stop all resources in the cluster in preparation for stopping the domain.

chrg -o offline -s 1=1

3. Change CritRsrcProtMethod to 5 to stop a undesired reboot from potentially occurring.

chrsrc -c IBM.PeerNode CritRsrcProtMethod=5

4. Stop the cluster/domain

stoprpdomain <domain_name>

If you are unsure of the domain name issue 'lsrpdomain' command.

Ensure the domain is stopped by issuing 'lsrpdomain' command.

5. Change the IP addresses using whatever method is appropriate for your OS.

You should always ensure that your /etc/hosts file is updated to reflect the new IP addresses for all servers/nodes on all servers/nodes.

Also change any application specific configuration that needs to use these IP addresses as well, this includes LDAP and especially DNS.

6. Update RSCT's ctrmc configuration, to allow communication across the clustered nodes using the new IP addresses.

This command must be issued on all nodes as the root user.

preprpnode <node_01> <node_02> ...

This will ensure that the file /var/ct/ctrmc.acls is updated with the new IP addresses.

7. Start the cluster/domain

startrpdomain <domain_name>

Monitor the OpState of the domain with the 'lsrpdomain' command, do not proceed to next step unless the domain reports it is online.

8. Verify that RSCT has harvested the new IP addresses for all nodes

lsrsrc -Ab IBM.NetworkInterface

Verify that the IP addresses are updated to the new values, that the OpState for each resource is showing online (1) and that the subnet mask matches on each server for the same subnet.

If the new addresses are not showing up in step #8

At this point if the output of the command issued in step 8 does not show the new IP addresses then you will likely need to force RSCT to sync the config across all nodes.

Issue the following command as the root user on a single node:

runact -c IBM.PeerDomain SyncConfig TargetNodes='{"<node_01>","<node_02>"}'

Replace <node_01> and <node_02> with your own hostnames.

Wait a few minutes before then rechecking step #8 to verify that the new IP addresses have been picked up.

Post procedural cleanup:

1. Change CritRsrcProtMethod back to 3

chrsrc -c IBM.PeerNode CritRsrcProtMethod=3

2. Start your resources/application

chrg -o online -s 1=1

-----------------------

If the above procedure does not result with a functioning cluster on the new IP Addresses then the recommendation is to delete the cluster and recreate it after the new IP Addresses have been assigned.

[{"Product":{"code":"SSRM2X","label":"Tivoli System Automation for Multiplatforms"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"--","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"}],"Version":"3.1;3.2;3.2.1;3.2.2;4.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
24 June 2019

UID

swg21670498