Tutorial: Creating and managing an instance group

In this tutorial, you use the tools in IBM Spectrum Conductor 2.4.1 to create and manage an instance group.

IBM Spectrum Conductor is a multitenant solution for Apache Spark and Anaconda/Python. Within IBM Spectrum Conductor, an instance group is an installation of Apache Spark that can run Spark core services (Spark master, shuffle, and history) and notebooks as configured. You can create and run multiple instance groups, associating each instance group with different Spark version packages as required.

Throughout this tutorial, you will learn various concepts, tips, and best practices that you can apply when you are creating your own resources and instance groups.
Note: This tutorial is a learning exercise that describes one way of creating an instance group. Depending on your company’s processes and needs, other techniques might be more appropriate.

Learning objectives

After you complete this tutorial, you will be able to do the following tasks:
  • Create a resource group, consumers, and a resource plan for the instance group
  • Create and deploy an Anaconda distribution with a conda environment
  • Create and deploy an instance group
  • Submit and monitor a Spark batch application
  • Schedule a Spark batch application
  • Open a notebook and create a new note
  • Monitor and modify an instance group
  • Configure the instance group to run within Docker containers

Each module in this tutorial contains lessons and builds on the previous module so that you can work with the tutorial from beginning to end.

Time required

This tutorial should take approximately 65 minutes to finish. If you explore other concepts that are related to this tutorial, it might take longer to complete.

System requirements

Prerequisites

Before you start this tutorial, you must complete several Tutorial prerequisites.