Enabling the Spark history service

Complete this task to enable the Apache Spark history service to view information about completed applications.

About this task

The information that the application web UI displays is, by default, only available when the application is active. To view the information after an application completes, you must enable event logging before starting the application. The Spark history server can then use the event logs to reconstruct the application’s web UI.

Procedure

Complete the following steps to enable event logging and point the history server to the event logs.

Add the following statements to your spark-defaults.conf file:
```
spark.eventLog.enabled         true
spark.eventLog.dir             event-log-directory
spark.history.fs.logDirectory  event-log-directory
```
where event-log-directory is the directory you configured in Event log directory and file permissions, where each Spark application stores its event log files, such as file:///var/spark/events.
Important: Use the event log directory only for event logging. Unexpected errors can occur if Spark finds non-event log files in the event log directory.
To start the history server, issue the following command from bash:
```
$SPARK_HOME/sbin/start-history-server.sh
```
where SPARK_HOME is an environment variable that contains the installation path for z/OS® Spark.

After the history server is started, you can use your browser to access the event logs and reconstructed application web UI for completed applications.

Point your web browser to the history server URL:
```
http://host_IP:history_server_port
```
where:

host_IP

The IP address for the z/OS system on which Spark runs.

history_server_port

The Spark history server UI port number. The default port number is 18080.
Figure 1 shows an example of the history server web UI.

Figure 1. The Spark history server web UI
Click an entry in the App ID column of the history server web UI to see the reconstructed application web UI for that application.
To stop the history server, issue the following command from bash:
```
$SPARK_HOME/sbin/stop-history-server.sh
```