Rolling back the IBM Spectrum Conductor Deep Learning Impact upgrade

Consider rolling back the upgrade from 1.2.3 to your previous version only if you encounter serious issues, for example, if key functionality is broken or if your Spark instance group fails to work. If possible, try changing the Spark instance group to fix the issue rather than rolling back the upgrade.

During a rollback, the order of the rolling upgrade process is reversed. You roll back your cluster-level upgrade and then the host-level upgrade. Typically, you might not need to roll back the upgrade on your hosts as well. If you must do this, first roll back the cluster, then roll back the individual hosts in reverse order of the upgrade.

Note:

Follow these steps when your cluster is installed to a local file system:
1. Log on to your master host as root or sudo to root (a user that has been set up in the egosetsudoers configuration file) if you installed IBM Spectrum Conductor Deep Learning Impact as root. If you didn’t install as root, you must log on with the cluster administrator ID. Ensure that your cluster has started, and run the cluster rollback command:
```
egoupgrade rollback cluster [-f] [-u user_account] [-x password]
```
  For syntax usage and details, see egoupgrade.
2. Source the IBM Spectrum Conductor Deep Learning Impact environment:
  - (csh) source $EGO_TOP/cshrc.platform
  - (bash) . $EGO_TOP/profile.platform
3. Complete this step only to remove version 1.2.3 from your hosts. Typically, after you upgrade your hosts, you test that your host works properly before you upgrade the cluster-level configuration. Therefore, it is unlikely that you need to roll back the hosts to your previous version. However, if you must completely remove your upgraded version from each host, complete these steps:
  1. Log on to each of your compute hosts by using the same user account that you used to upgrade version 1.2.3, and run the host rollback command:
```
egoupgrade rollback host [-f]
```
    For syntax usage and details, see egoupgrade.
  2. Source the environment:
    - (csh) source $EGO_TOP/cshrc.platform
    - (bash) . $EGO_TOP/profile.platform
  3. Repeat steps 3a and 3b on your management hosts. Log on to the host by using the same user account that you used to upgrade version 1.2.3.
  4. Repeat steps 3a and 3b on your master candidate hosts. Log on to the host by using the same user account that you used to upgrade version 1.2.3.
  5. Repeat steps 3a and 3b on your master host. Log on to the host by using the same user account that you used to upgrade version 1.2.3.
4. Downgrade the conda environments by running this script:
```
conda activate dlipy3
conda remove --yes powerai-license
conda install --yes powerai=1.6.0
conda install --yes keras==2.1.6
conda install --yes --no-update-dependencies cython==0.29.4 ipython==5.3.0 python-lmdb==0.94 nose==1.3.7 requests==2.13.0
conda install --yes --no-update-dependencies redis-py==2.10.5 chardet==3.0.4 flask==1.0.2
conda install --yes --no-update-dependencies python-gflags==3.1.2 pandas==0.24.1 pyzmq==17.1.2

pip install easydict==1.9
pip install hanziconv==0.3.2 gensim==3.6.0
pip install ipaddress==1.0.22 configparser==3.5.0 easydict==1.9 glog==0.3.1
pip install asyncio==3.4.3
conda deactivate

conda activate dlipy2
conda remove --yes powerai-license
conda install --yes powerai=1.6.0
conda install --yes keras==2.1.6
conda install --yes --no-update-dependencies cython==0.29.4 ipython==5.3.0 python-lmdb==0.94 nose==1.3.7 requests==2.13.0
conda install --yes --no-update-dependencies redis-py==2.10.5 chardet==3.0.4 flask==1.0.2
conda install --yes --no-update-dependencies python-gflags==3.1.2 pandas==0.24.1 pyzmq==17.1.2
conda install --yes --no-update-dependencies trollius==2.2

pip install weave==0.16.0
pip install ipaddress==1.0.22 configparser==3.5.0 easydict==1.9 glog==0.3.1
pip install hanziconv==0.3.2 gensim==3.6.0
conda deactivate

yum install openblas-devel glog-devel gflags-devel hdf5-devel leveldb-devel libsodium-devel lmdb-devel
yum install gcc-c++ gcc-gfortran freetype-devel libpng-devel libffi-devel openssl-devel openssh-clients
```

Follow these steps when your cluster is installed to a shared file system:

Log on to your master host as root or sudo to root (a user that has been set up in the egosetsudoers configuration file) if you installed IBM Spectrum Conductor Deep Learning Impact as root. If you didn’t install as root, you must log on with the cluster administrator ID. Ensure that your cluster has started, and run the cluster rollback command:
```
egoupgrade rollback cluster [-f] [-u user_account] [-x password]
```
For syntax usage and details, see egoupgrade.
If you must completely remove the configuration for your upgraded version, log on to any compute host in your cluster and run the host rollback command:
```
egoupgrade rollback host [-f]
```
For syntax usage and details, see egoupgrade.
Source the environment:
- (csh) source $EGO_TOP/cshrc.platform
- (bash) . $EGO_TOP/profile.platform
Restart EGO on the local host:
```
egosh ego shutdown
egosh ego start
```
On all other compute hosts in your cluster, complete these steps:
1. Source the environment:
  - (csh) source $EGO_TOP/cshrc.platform.comp
  - (bash) . $EGO_TOP/profile.platform.comp
2. Restart EGO on the host:
```
egosh ego shutdown
egosh ego start
```
Repeat steps 3 and 4 on all management hosts in your cluster.
Repeat steps 3 and 4 on the master host in your cluster.

Downgrade the conda environments by running this script:

conda activate dlipy3
conda remove --yes powerai-license
conda install --yes powerai=1.6.0
conda install --yes keras==2.1.6
conda install --yes --no-update-dependencies cython==0.29.4 ipython==5.3.0 python-lmdb==0.94 nose==1.3.7 requests==2.13.0
conda install --yes --no-update-dependencies redis-py==2.10.5 chardet==3.0.4 flask==1.0.2
conda install --yes --no-update-dependencies python-gflags==3.1.2 pandas==0.24.1 pyzmq==17.1.2

pip install easydict==1.9
pip install hanziconv==0.3.2 gensim==3.6.0
pip install ipaddress==1.0.22 configparser==3.5.0 easydict==1.9 glog==0.3.1
pip install asyncio==3.4.3
conda deactivate

conda activate dlipy2
conda remove --yes powerai-license
conda install --yes powerai=1.6.0
conda install --yes keras==2.1.6
conda install --yes --no-update-dependencies cython==0.29.4 ipython==5.3.0 python-lmdb==0.94 nose==1.3.7 requests==2.13.0
conda install --yes --no-update-dependencies redis-py==2.10.5 chardet==3.0.4 flask==1.0.2
conda install --yes --no-update-dependencies python-gflags==3.1.2 pandas==0.24.1 pyzmq==17.1.2
conda install --yes --no-update-dependencies trollius==2.2

pip install weave==0.16.0
pip install ipaddress==1.0.22 configparser==3.5.0 easydict==1.9 glog==0.3.1
pip install hanziconv==0.3.2 gensim==3.6.0
conda deactivate

yum install openblas-devel glog-devel gflags-devel hdf5-devel leveldb-devel libsodium-devel lmdb-devel
yum install gcc-c++ gcc-gfortran freetype-devel libpng-devel libffi-devel openssl-devel openssh-clients

Your IBM Spectrum Conductor Deep Learning Impact version level is rolled back to 1.2.2.