Removing Spark instance groups

Remove a Spark instance group that is in the Registered, Register Error, Ready, or Deploy Error state.

For deep learning workloads, you must ensure that any related deep learning assets are deleted before the Spark instance group is removed. Before removing the Spark instance group, ensure the following:

All related training jobs are stopped.
All related models are deleted.
All related datasets are deleted.

You must be a cluster administrator, consumer administrator, or have the Spark Instance Groups View, Spark Instance Groups Control, Services View, and Services Control permissions.

If the Spark instance group to remove has running workload, wait for the workload to complete.

When you remove a Spark instance group, the system cleans up the Spark instance group environment as follows - in some cases, however, you must manually clean up the environment:

Deletes logs under the elk_logs directory, which is used by Elastic Stack to harvest information.
Deletes the deployment directory and the Spark working directory ($SPARK_HOME/work) for the Spark instance group.
If high availability is enabled for the Spark master in the Spark instance group, deletes files under the directory that is specified by the spark.deploy.recoveryDirectory parameter. However, if this directory is a shared NFS location, you must manually clean up data under this location after the Spark instance group is removed.
If any notebooks associated with the Spark instance group use a customized location for the notebook base data directory, you must manually clean up that directory. The notebook base data directory is by default the same as the notebook deployment directory (which is {DEPLOY_DIR_OF_SIG}/{NOTEBOOK_NAME}-{NOTEBOOK_VERSION}).

From the cluster management console, click Spark Instance Groups.
Select one or more of the Spark instance groups that you want to remove.
Click Remove.
Confirm your selection.

The Spark instance group moves to the Undeploying state. After all the associated packages are undeployed, it moves to the Unregistering state and is later removed.