Installing the MLDL frameworks
Find instructions for installing the machine learning and deep learning (MLDL) frameworks.
- Setting up the software repository
- Creating conda environments (recommended)
- Installing all frameworks at the same time
- Installing frameworks individually
- Accepting the WML CE license agreement
- Helping the conda installer: strict channel priority and the powerai-release package
- Additional conda channels
- Creating a local mirror of the WML CE conda channel
Setting up the software repository
The WML CE MLDL packages are distributed as conda packages in an online conda repository. conda must be configured to give priority to installing packages from this channel.
Add the WML CE channel to the conda configuration by running the following command:
conda config --prepend channels \
https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
Creating conda environments (recommended)
With conda, you can create environments that have different versions of Python or packages installed in them. Conda environments are optional but recommended. If not used, packages are installed in the default environment called base, which often has a higher risk of containing conflicting packages or dependencies. Switching between environments is called activating the environment.
The syntax to create and activate a conda environment is:
conda create --name <environment name> python=<python version>
conda activate <environment name>
The only valid Python versions with WML CE are Python 3.6 and 3.7.
For example, to create an environment named wmlce_env with Python 3.6:
conda create --name wmlce_env python=3.6
conda activate wmlce_env
For more information on what you can do with conda environment see https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html.
Installing all frameworks at the same time
All the MLDL frameworks except RAPIDS packages can be installed at the same time by
using the powerai
meta-package. All the RAPIDS packages can be installed using the
powerai-rapids
meta-package.
- To install
powerai
GPU packages in the conda environment, run:conda install powerai
- To install
powerai
CPU packages in the conda environment, run:conda install powerai-cpu
- To install RAPIDS packages, run:
conda install powerai-rapids
Installing frameworks individually
You can install the MLDL frameworks individually. The framework packages include the following versions.
Package | Description | Version | Available on ppc64le | Available on x86_64 |
---|---|---|---|---|
caffe |
IBM-optimized version of Berkeley Vision and Learning Center Caffe | 1.0.0 | X | X |
caffe-cpu |
IBM-optimized Caffe CPU-only package | 1.0.0 | X | X |
cudf |
RAPIDS cuDF | 0.11.0 | X | |
cuml |
RAPIDS cuML | 0.11.0 | X | |
ddl |
Distributed Deep Learning | 1.5.1 | X | X |
pai4sk
|
WML CE Snap ML | 1.6.0 | X | X |
pytorch |
PyTorch | 1.3.1 | X | X |
pytorch-cpu |
PyTorch | 1.3.1 | X | X |
snapml-spark |
WML CE Snap ML Spark | 1.6.0 | X | X |
tensorflow |
TensorFlow CPU-only package | 2.1.0 | X | X |
tensorflow-gpu |
TensorFlow with GPU support | 2.1.0 | X | X |
tensorflow-serving |
TensorFlow Serving | 2.1.0 | X | X |
py-xgboost-gpu |
xgboost with GPU support | 0.90 | X | X |
py-xgboost-cpu |
xgboost CPU-only package | 0.90 | X | X |
With the conda environment activated, run the following command:
conda install <package name>
Accepting the WML CE license agreement
During the conda install, the packages are downloaded from the internet and after downloading, the license agreement is presented. Read the license agreement and accept the terms and conditions to complete the install. If you decline the license agreement the packages are not installed.
After you finish reading the license agreement, future installations can be automated to silently accept the license agreement by running the following command before running the conda install command:
export IBM_POWERAI_LICENSE_ACCEPT=yes
The license accept has to be done only once on a per user basis.
Helping the conda installer: strict channel priority and the powerai-release package
The conda installer uses a set of rules to determine which packages to install. Channel priorities and package versions are weighted heavily, but the installer also considers factors such as the number of packages that would need to be installed, whether any packages would need to be upgraded or removed, and so on.
- Packages from Anaconda channels over the WML CE channel in spite of channel priorities.
- Packages from an older release of WML CE in spite of newer versions being available.
You can guide the conda installer to ensure that it chooses the desired WML CE package using the strict channel priority option and the powerai-release meta-package.
Strict channel priority
The strict channel priority option forces the conda installer to give additional weight to the
priority of channels defined in the configuration. It is useful in cases where the conda installer
is preferring packages from lower-priority channels. The simplest use is just to add
--strict-channel-priority
to the install command:
conda install --strict-channel-priority tensorflow
(my-wmlce-env) $ conda config --show
...
channel_priority: flexible
channels:
- https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
- defaults
...
You could permanently change the channel priority setting to strict:
conda config --set channel_priority strict
powerai-release meta-package
The powerai-release meta-package can be used to specify the WML CE release you want to install from. It is useful when the installer prefers packages from an earlier release, or if you intentionally want to install packages from an older release.
conda install pytorch powerai-release=1.7.0
conda install pytorch powerai-release=1.6.2
The --strict-channel-priority
option can be used with
powerai-release
for greater control:
conda install --strict-channel-priority pytorch powerai-release=1.6.2
cuml
, you need
to use powerai-release=1.7.0
explicitly in the conda install command in order to pick up the latest versions of
packages. For example: conda install cuml powerai-release=1.7.0
.For more information on the conda installer's rules, see Running the solver
at: Understanding and Improving Conda’s performance.
Additional conda channels
The main WML CE conda channel is described above. That channel includes the formal, supported WML CE releases.
Additional conda channels are available to complement the main channel. Packages in these channels are not formally supported. Both of these channels are optional. WML CE will install and run fine without either. They can also be used independently of each other (Supplementary does not need Early Access or vice versa). Use them if you want the packages they provide and do not need formal support.
- The WML CE Supplementary channel is
available at: https://anaconda.org/powerai/.
This channel includes packages that are not part of WML CE, but which may be useful to WML CE users. The packages are built from recipes in the WML CE GitHub repository: https://github.com/ibm/powerai.
Problem reports and recipe contributions from the community are welcome. More information about the Supplementary channel can be found in the PowerAI Supplementary Channel README.
- The WML CE Early Access channel is available at: https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/.
This channel is updated occasionally with latest versions of various packages included in WML CE. The purpose of the channel is to make new versions of frameworks available in advance of formal WML CE releases. Packages published in the Early Access channel may not exactly match a later WML CE release. For example, package and prerequisite versions may differ.
Note: Easy upgrade between packages in the main and Early Access channels is not guaranteed. Using a separate conda environment for Early Access packages is recommended.
Creating a local mirror of the WML CE conda channel
If desired, there are multiple ways to create a local mirror of the WML CE conda channel.
One method of mirroring is to use conda-mirror
, which is a good choice for
customers wanting an easy way to mirror, and who have plenty of space to store packages from all
WML CE releases. It is easy to limit the
mirroring by system architecture or by python version.
Another method is to use a scripted download, which is good choice for customers who are interested in mirroring only a single WML CE release and are comfortable scripting the download process, or for customers who want stronger validation of the downloaded package files.
conda-mirror
is a tool for mirroring conda channels, and is available from conda
forge. It provides a convenient way to mirror the entire WML CE conda channel for a specific platform
architecture, that is, ppc64le or x86.
Example:
conda create -n mirror_env
conda activate mirror_env
conda install conda-mirror -c conda-forge
export CHANNEL_URL=https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
conda-mirror -vv --upstream-channel $CHANNEL_URL --target-directory $HOME/wmlce-mirror --platform linux-ppc64le
conda-mirror -vv --upstream-channel $CHANNEL_URL --target-directory $HOME/wmlce-mirror --platform noarch
conda index $HOME/wmlce-mirror
--platform
flag is required. Valid values are
linux-ppc64le
, linux-64
, and noarch
. The tool
downloads only packages for the specified platform. WML CE includes some required cross-platform packages in
noarch
, so you always need to mirror noarch
in addition to one or
both of the others.
--proxy
: Specify a network proxy if needed.--temp-directory
: Specify a different location for temporary files.--config
: Specify a configuration file to limit what gets downloaded. For example, to exclude Python 2.7.
More information about conda-mirror
is available at https://github.com/regro/conda-mirror.
Creating a mirror using the WML CE package listings
A package listing is provided for each release of WML CE. The listings can be found on the web, one directory level above the WML CE conda channel at https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/.
The package listing files are named after the WML CE release:
wmlce-$RELEASE-conda-sha256.txt
. The files contain a list of SHA256 hashes and file
paths for each package in that release. For
example:
a42bd21014f424866b201ebd4c80b5eed119e38c4d2686a6aa60f91754e4b117 linux-64/dali-0.13-py37_d1e100e_1443.gbc37618.tar.bz2
dbb2eb2ddfe79a4139eca446ba1177d01e1e28e2b8eebe844b7bf4fcfe73936c linux-ppc64le/onnx-1.5.0-py36_614.gd049fd7.tar.bz2
5b1309837c7dc821ad5ff25545a0424d1eded9148171524cd8ae2be70a1670a4 linux-64/pytorch-1.2.0-20238.g1faf942.tar.bz2
085bdb73206f15892462cc3157bd8adf72264e9415a4eb1818637f7ebfcb2d9d linux-ppc64le/nccl-2.4.8-586.gdba67b7.tar.bz2
...
The file paths are relative to the base of the conda channel
(https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
)
All of the packages in a single release can be downloaded with a simple shell script, for example:
#!/bin/bash
CHANNEL_URL=https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda
RELEASE=1.6.2
MIRROR_DIR=wmlce-${RELEASE}-mirror
MANIFEST=wmlce-${RELEASE}-conda-sha256.txt
set -e
mkdir -p $MIRROR_DIR
curl -L -O `dirname $CHANNEL_URL`/$MANIFEST
while read sha_hash file_path
do
curl -L -O $CHANNEL_URL/$file_path
dir_name=$MIRROR_DIR/`dirname $file_path`
file_name=`basename $file_path`
mkdir -p "$dir_name"
mv -i "$file_name" "$dir_name"
done < $MANIFEST
conda index $MIRROR_DIR
exit 0
The sha256sum
tool (available from your Linux distribution) can be used to
calculate the SHA256 hashes of the downloaded files for comparison to the hashes in the listing
file:
cd $MIRROR_DIR
sha256sum */*bz2
Using a local mirror
Once the local mirror is created, it can be included in the local channel configuration or
referenced in one-off conda install
commands. The local channel is specified as
file://<full_path_to_channel_dir>
:
conda config --prepend channels file://$HOME/wmlce-mirror
or
conda install -c file://$HOME/wmlce-mirror powerai
Local channels are self-contained and location-independent. They tolerate being tarred or zipped, and can be moved to other machines or file system locations. They can be placed in a shared file system to allow access from multiple hosts.
Channel directories can also be served locally using a web server, and clients would then use an
http://
or https://
channel reference rather than
file://
.
The conda documentation has more information on creating local conda channels: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/create-custom-channels.html.