Troubleshooting issues with Apache Spark
Use the following information to troubleshoot issues you might encounter with Apache Spark.
You can also use Apache Spark log files to help identify issues with your Spark processes. See Spark log files for more information about where to find these log files.
Spark scripts fail with FSUM6196 and EDC5129I messages
Symptom: Spark scripts fail with the
following error message:
env: FSUM6196 bash: not executable: EDC5129I No such file or directory
Cause: Apache Spark expects to find the bash shell in
the user's PATH environment variable, but bash cannot be found when the
spark-script attempts to invoke bash.
Response: Ensure that the
PATH environment variable includes the directory where the bash executable is
installed. For example, if bash is located at /usr/bin/bash-4.2/bin/bash,
ensure that PATH includes/usr/bin/bash-4.2/bin.
Spark scripts fail with FSUM7332 message
Symptom: Spark scripts fail with the
following error message:
failed to launch org.apache.spark.deploy.master.Master:
/usr/lpp/IBM/Spark/bin/spark-class 77: FSUM7332 syntax error: got (, expecting done
Cause: Apache Spark expects to find the
env command in /usr/bin, but it cannot be found. Either the
/usr/bin/env symbolic link is missing or it is not pointing to
/bin/env. It is possible that creation of this symbolic link was missed during
Spark setup or that the
symbolic link was lost after a system IPL.
Response: Ensure that
/usr/bin/env exists and is a symbolic link to /bin/env,
and that the symbolic link persists across system IPLs. For more information, see
Verifying the env command path.
An error occurs when starting the master, but the master starts correctly
Symptom: When starting the master, the following
error occurs:
bash-4.2$ $SPARK_HOME/sbin/start-master.sh
starting org.apache.spark.deploy.master.Master,
logging to /u/user/Billy/logs/spark--org.apache.spark.deploy.master.Master-1-ALPS.out
failed to launch org.apache.spark.deploy.master.Master:
full log in /u/user/Billy/logs/spark--org.apache.spark.deploy.master.Master-1-ALPS.out
Cause: Apache Spark polls for a number of seconds to
repeatedly check to see if the master started successfully. If your system is under heavy load, this
message might appear, but it generally means that the check finished polling for success before the
master startup completed.
Response: Check the master log, or
issue the ps command and look for the master process to definitively see whether
or not the master started successfully.
Only one Spark executor is started
Symptom: You specified --num-executors
2 on a spark-submit, but only one executor was started.
Cause: The --num-executors parameter is only valid in
YARN mode, which is not supported on z/OS®. Instead, the
number of executors is determined by your resource settings.
Response: For more information about resource settings, see Configuring memory and CPU options.
Shell script displays unreadable characters
Symptom: When running a shell script, it displays unreadable characters on the screen, such as:
./start-master.sh: line 1: syntax error near unexpected token `$'\101\123\124\105\122^''
Cause: Incorrect file encoding or downlevel bash shell.
Response: Ensure that the file encoding is in EBCDIC, not
ASCII, and is not tagged as ASCII. You can check the tagging of a file by issuing the
ls -T shell command. Also, ensure that your bash shell level is 4.2.53. You can check the bash level by
issuing the bash -version command.
Spark-shell fails with java.lang.ExceptionInInitializerError error message
Symptom: Spark-shell fails with the following error
message:
java.lang.ExceptionInInitializerError …. Scala signature
package has wrong version expected: 5.0 found: 45.0 in scala.package
Cause: Your JVM is likely running with the wrong default
encoding.
Response: Ensure that you have the
following environment variable set:
IBM_JAVA_OPTIONS=-Dfile.encoding=ISO8859-1
For more information about setting
environment variables, see Setting up a user ID for use with IBM z/OS Platform for Apache Spark.The Spark master fails with JVMJ9VM015W error
Symptom: The Spark master fails to start and
gives the following error:
JVMJ9VM015W Initialization error for library j9gc28(2):
Failed to instantiate compressed references metadata; 200M requested
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Cause: The master JVM could not obtain enough memory to start. Memory is
most likely constrained by your ASSIZEMAX setting.
Response: For
more information about setting the ASSIZEMAX parameter, see Configuring memory and CPU options.
A Spark application is not progressing and shows JVMJ9VM015W error in the log
Symptom: The Spark master and worker started
successfully; however, the Spark application is not making any
progress, and the following error appears in the executor log:
JVMJ9VM015W Initialization error for library j9gc28(2):
Failed to instantiate heap; 20G requested
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Cause: The executor JVM could not obtain enough memory to start. Memory is
most likely constrained by your MEMLIMIT setting or IEFUSI exit.
Response: For more information about setting the MEMLIMIT parameter or using the IEFUSI exit, see
Configuring memory and CPU options.