Configure the Apache Spark integration

Deploy Apache Spark and prepare LSF for the integration package

Before you begin

The Apache Spark integration supports the following platforms:
  • x86_64
  • ppc64
  • ppc64le

Procedure

  1. Download and deploy Apache Spark.

    The latest version is available from the official Apache Spark site: http://spark.apache.org/downloads

  2. Navigate to the $LSF_BINDIR directory.
  3. Edit the lsf-spark-submit.sh and lsf-spark-shell.sh connector script files and set the SPARK_HOME and SPARK_MASTER_PORT environment variables.
  4. Edit the lsf-stop-spark.sh connector script file and set the SPARK_HOME environment variable.