Tutorial: Creating and managing an instance group
In this tutorial, you use the tools in IBM Spectrum Conductor 2.4.1 to create and manage an instance group.
IBM Spectrum Conductor is a multitenant solution for Apache Spark and Anaconda/Python. Within IBM Spectrum Conductor, an instance group is an installation of Apache Spark that can run Spark core services (Spark master, shuffle, and history) and notebooks as configured. You can create and run multiple instance groups, associating each instance group with different Spark version packages as required.
Throughout this tutorial, you will learn various concepts, tips, and best practices that you can
apply when you are creating your own resources and instance groups.
Note: This tutorial is a
learning exercise that describes one way of creating an instance group. Depending on your company’s
processes and needs, other techniques might be more appropriate.
Learning objectives
After you complete this tutorial, you will be able to do the following tasks:
- Create a resource group, consumers, and a resource plan for the instance group
- Create and deploy an Anaconda distribution with a conda environment
- Create and deploy an instance group
- Submit and monitor a Spark batch application
- Schedule a Spark batch application
- Open a notebook and create a new note
- Monitor and modify an instance group
- Configure the instance group to run within Docker containers
Each module in this tutorial contains lessons and builds on the previous module so that you can work with the tutorial from beginning to end.
Time required
This tutorial should take approximately 65 minutes to finish. If you explore other concepts that are related to this tutorial, it might take longer to complete.System requirements
- OS: Linux or Linux on POWER®
- IBM Spectrum Conductor 2.4.1 - See System requirements and recommendations
Prerequisites
Before you start this tutorial, you must complete several Tutorial prerequisites.
- Tutorial prerequisites
Before you start this tutorial, you must complete several prerequisites. - Resource planning
In this module, you are going to create all the resources for the instance group. - Workload
In this module, you learn how to manage workload using IBM Spectrum Conductor. - Summary
This completes the Creating and managing an instance group tutorial.