Using the IPython plug-in in parallel mode LSF

The IPython plug-in can be used by LSF for parallel interactive computing.

1. Source the LSF profile.
   For example:

   • For csh or tcsh: %source <LSF_TOP>/conf/cshrc.lsf
   • For sh, ksh, or bash: $ . <LSF_TOP>/conf/profile.lsf
2. Parallel mode (PARALLEL_SUPPORT=Y) is enabled by default in the hpa-ipython.sh script under $LSF_BINDIR.
3. Set the NUM_ENGINES parameter.

   NUM_ENGINES is set to 2 by default, meaning 2 IPython engines will be started in one node.

   The example that follows assumes the default of NUM_ENGINES=2.

4. Set a different value for the IPYTHON_PROFILE if you want to distinguish other users if the execution user changes.
5. Start IPython to span across 2 nodes in your cluster, with each node start 2 engines.

   For example: bsub -R "select[IpythonHost] span[ptile=1]" -n 2 -q interactive -Is hpa-ipython.sh

   The IPython interactive shell will start and can be used to check the number of engines started. The following example shows the expected 2*2:

   In [1]: import ipyparallel as ipp
   In [2]: rc = ipp.Client()
   In [3]: rc.ids
   Out[3]: [0, 1, 2, 3]
   In [4]: exit
   

6. To end the session, quit the shell and the corresponding LSF job will be terminated.