Installing the MLDL frameworks

Find instructions for installing the machine learning and deep learning (MLDL) frameworks.

Setting up the software repository

The WML CE MLDL packages are distributed as conda packages in an online conda repository. conda must be configured to give priority to installing packages from this channel.

Add the WML CE channel to the conda configuration by running the following command:

conda config --prepend channels \
https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

Creating conda environments (recommended)

With conda, you can create environments that have different versions of Python or packages installed in them. Conda environments are optional but recommended. If not used, packages are installed in the default environment called base, which often has a higher risk of containing conflicting packages or dependencies. Switching between environments is called activating the environment.

The syntax to create and activate a conda environment is:

conda create --name <environment name> python=<python version>
conda activate <environment name>
Note: It is recommended that you specify the Python version when creating a new environment. If you do not specify the version, Python 3.7 is installed when any package that requires Python are installed.

The only valid Python versions with WML CE are Python 3.6 and 3.7.

For example, to create an environment named wmlce_env with Python 3.6:

conda create --name wmlce_env python=3.6
conda activate wmlce_env

For more information on what you can do with conda environment see https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html.

Note: WML CE should be run as a non-privileged user and not root. The WML CE components are designed to be usable by normal users, and the pre-installed docker images provide a non-root user by default. Some of the WML CE components will give warnings or will fail when run as root.

Installing all frameworks at the same time

All the MLDL frameworks except RAPIDS packages can be installed at the same time by using the powerai meta-package. All the RAPIDS packages can be installed using the powerai-rapids meta-package.

  • To install powerai GPU packages in the conda environment, run:
    conda install powerai
  • To install powerai CPU packages in the conda environment, run:
    conda install powerai-cpu
  • To install RAPIDS packages, run:
    conda install powerai-rapids
Note: The Python 3.6 or 3.7 version of the package installed is determined by the Python version of the active environment.

Installing frameworks individually

You can install the MLDL frameworks individually. The framework packages include the following versions.

Table 1. Framework packages
Package Description Version Available on ppc64le Available on x86_64
caffe IBM-optimized version of Berkeley Vision and Learning Center Caffe 1.0.0 X X
caffe-cpu IBM-optimized Caffe CPU-only package 1.0.0 X X
cudf RAPIDS cuDF 0.11.0 X  
cuml RAPIDS cuML 0.11.0 X  
ddl Distributed Deep Learning 1.5.1 X X
pai4sk WML CE Snap ML 1.6.0 X X
pytorch PyTorch 1.3.1 X X
pytorch-cpu PyTorch 1.3.1 X X
snapml-spark WML CE Snap ML Spark 1.6.0 X X
tensorflow TensorFlow CPU-only package 2.1.0 X X
tensorflow-gpu TensorFlow with GPU support 2.1.0 X X
tensorflow-serving TensorFlow Serving 2.1.0 X X
py-xgboost-gpu xgboost with GPU support 0.90 X X
py-xgboost-cpu xgboost CPU-only package 0.90 X X

With the conda environment activated, run the following command:

conda install <package name>

Accepting the WML CE license agreement

During the conda install, the packages are downloaded from the internet and after downloading, the license agreement is presented. Read the license agreement and accept the terms and conditions to complete the install. If you decline the license agreement the packages are not installed.

After you finish reading the license agreement, future installations can be automated to silently accept the license agreement by running the following command before running the conda install command:

export IBM_POWERAI_LICENSE_ACCEPT=yes

The license accept has to be done only once on a per user basis.

Helping the conda installer: strict channel priority and the powerai-release package

The conda installer uses a set of rules to determine which packages to install. Channel priorities and package versions are weighted heavily, but the installer also considers factors such as the number of packages that would need to be installed, whether any packages would need to be upgraded or removed, and so on.

The conda installer will sometimes come up with a surprising installation solution. It may prefer to install:
  • Packages from Anaconda channels over the WML CE channel in spite of channel priorities.
  • Packages from an older release of WML CE in spite of newer versions being available.

You can guide the conda installer to ensure that it chooses the desired WML CE package using the strict channel priority option and the powerai-release meta-package.

Strict channel priority

The strict channel priority option forces the conda installer to give additional weight to the priority of channels defined in the configuration. It is useful in cases where the conda installer is preferring packages from lower-priority channels. The simplest use is just to add --strict-channel-priority to the install command:

conda install --strict-channel-priority tensorflow
You can check the priority of the channels in the configuration by running the following:
(my-wmlce-env) $ conda config --show
...
channel_priority: flexible
channels:
  - https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
  - defaults
...

You could permanently change the channel priority setting to strict:

conda config --set channel_priority strict

powerai-release meta-package

The powerai-release meta-package can be used to specify the WML CE release you want to install from. It is useful when the installer prefers packages from an earlier release, or if you intentionally want to install packages from an older release.

Examples:
conda install pytorch powerai-release=1.7.0

conda install pytorch powerai-release=1.6.2

The --strict-channel-priority option can be used with powerai-release for greater control:

conda install --strict-channel-priority pytorch powerai-release=1.6.2
Note: For some of the packages like cuml, you need to use powerai-release=1.7.0 explicitly in the conda install command in order to pick up the latest versions of packages. For example: conda install cuml powerai-release=1.7.0.

For more information on the conda installer's rules, see Running the solver at: Understanding and Improving Conda’s performance.

Additional conda channels

The main WML CE conda channel is described above. That channel includes the formal, supported WML CE releases.

Additional conda channels are available to complement the main channel. Packages in these channels are not formally supported. Both of these channels are optional. WML CE will install and run fine without either. They can also be used independently of each other (Supplementary does not need Early Access or vice versa). Use them if you want the packages they provide and do not need formal support.

  • The WML CE Supplementary channel is available at: https://anaconda.org/powerai/.

    This channel includes packages that are not part of WML CE, but which may be useful to WML CE users. The packages are built from recipes in the WML CE GitHub repository: https://github.com/ibm/powerai.

    Problem reports and recipe contributions from the community are welcome. More information about the Supplementary channel can be found in the PowerAI Supplementary Channel README.

  • The WML CE Early Access channel is available at: https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/.

    This channel is updated occasionally with latest versions of various packages included in WML CE. The purpose of the channel is to make new versions of frameworks available in advance of formal WML CE releases. Packages published in the Early Access channel may not exactly match a later WML CE release. For example, package and prerequisite versions may differ.

    Note: Easy upgrade between packages in the main and Early Access channels is not guaranteed. Using a separate conda environment for Early Access packages is recommended.

Creating a local mirror of the WML CE conda channel

If desired, there are multiple ways to create a local mirror of the WML CE conda channel.

One method of mirroring is to use conda-mirror, which is a good choice for customers wanting an easy way to mirror, and who have plenty of space to store packages from all WML CE releases. It is easy to limit the mirroring by system architecture or by python version.

Another method is to use a scripted download, which is good choice for customers who are interested in mirroring only a single WML CE release and are comfortable scripting the download process, or for customers who want stronger validation of the downloaded package files.

Note: The following sections describe how to mirror just the WML CE conda channel, but many of the packages in that channel depend on others from the Anaconda distribution channels. If a fully offline or air gapped solution is necessary, those channels must also be mirrored.
Creating a mirror using 'conda-mirror'

conda-mirror is a tool for mirroring conda channels, and is available from conda forge. It provides a convenient way to mirror the entire WML CE conda channel for a specific platform architecture, that is, ppc64le or x86.

Example:

conda create -n mirror_env

conda activate mirror_env

conda install conda-mirror -c conda-forge

export CHANNEL_URL=https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

conda-mirror -vv --upstream-channel $CHANNEL_URL --target-directory $HOME/wmlce-mirror --platform linux-ppc64le
conda-mirror -vv --upstream-channel $CHANNEL_URL --target-directory $HOME/wmlce-mirror --platform noarch

conda index $HOME/wmlce-mirror
Important: The --platform flag is required. Valid values are linux-ppc64le, linux-64, and noarch.

The tool downloads only packages for the specified platform. WML CE includes some required cross-platform packages in noarch, so you always need to mirror noarch in addition to one or both of the others.

The tool supports other options that may be useful:
  • --proxy: Specify a network proxy if needed.
  • --temp-directory: Specify a different location for temporary files.
  • --config: Specify a configuration file to limit what gets downloaded. For example, to exclude Python 2.7.

More information about conda-mirror is available at https://github.com/regro/conda-mirror.

Creating a mirror using the WML CE package listings

A package listing is provided for each release of WML CE. The listings can be found on the web, one directory level above the WML CE conda channel at https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/.

The package listing files are named after the WML CE release: wmlce-$RELEASE-conda-sha256.txt. The files contain a list of SHA256 hashes and file paths for each package in that release. For example:

a42bd21014f424866b201ebd4c80b5eed119e38c4d2686a6aa60f91754e4b117 linux-64/dali-0.13-py37_d1e100e_1443.gbc37618.tar.bz2
dbb2eb2ddfe79a4139eca446ba1177d01e1e28e2b8eebe844b7bf4fcfe73936c linux-ppc64le/onnx-1.5.0-py36_614.gd049fd7.tar.bz2
5b1309837c7dc821ad5ff25545a0424d1eded9148171524cd8ae2be70a1670a4 linux-64/pytorch-1.2.0-20238.g1faf942.tar.bz2
085bdb73206f15892462cc3157bd8adf72264e9415a4eb1818637f7ebfcb2d9d linux-ppc64le/nccl-2.4.8-586.gdba67b7.tar.bz2
...

The file paths are relative to the base of the conda channel (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda)

All of the packages in a single release can be downloaded with a simple shell script, for example:

#!/bin/bash

CHANNEL_URL=https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda

RELEASE=1.6.2
MIRROR_DIR=wmlce-${RELEASE}-mirror
MANIFEST=wmlce-${RELEASE}-conda-sha256.txt

set -e

mkdir -p $MIRROR_DIR

curl -L -O `dirname $CHANNEL_URL`/$MANIFEST

while read sha_hash file_path
do
    curl -L -O $CHANNEL_URL/$file_path

    dir_name=$MIRROR_DIR/`dirname $file_path`
    file_name=`basename $file_path`
    mkdir -p "$dir_name"
    mv -i "$file_name" "$dir_name"

done < $MANIFEST

conda index $MIRROR_DIR

exit 0

The sha256sum tool (available from your Linux distribution) can be used to calculate the SHA256 hashes of the downloaded files for comparison to the hashes in the listing file:

cd $MIRROR_DIR

sha256sum */*bz2

Using a local mirror

Once the local mirror is created, it can be included in the local channel configuration or referenced in one-off conda install commands. The local channel is specified as file://<full_path_to_channel_dir>:

conda config --prepend channels file://$HOME/wmlce-mirror

or

conda install -c file://$HOME/wmlce-mirror powerai

Local channels are self-contained and location-independent. They tolerate being tarred or zipped, and can be moved to other machines or file system locations. They can be placed in a shared file system to allow access from multiple hosts.

Channel directories can also be served locally using a web server, and clients would then use an http:// or https:// channel reference rather than file://.

The conda documentation has more information on creating local conda channels: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/create-custom-channels.html.