Log files for errors related to YARN
Examine the log files when you receive an error that is related to YARN.
- YARN client logs
- Application Master logs
- Resource Manager and Node Manager logs.
YARN client logs
The YARN client starts Application Masters that run the jobs on your Hadoop cluster. Errors that occur when you are starting a YARN client are logged in /tmp/yarn_client.out. Errors that occur after the YARN client is started are logged in $APT_ORCHHOME/logs/yarn_logs/yarn_client.out.
Application Master logs
The Application Master is what requests resources from YARN and runs job processes on the allocated resources. Two kinds of log files are generated:
- Common log file
- A single Application Manager can run multiple jobs. The common log file contains all messages that are not specific to a specific job. The name of the file starts application_ and includes a string of numbers. The file extension is .stdout. For example, the file might be application_1437631989065_0009.stdout.
- Individual job log
- An Application Master runs multiple jobs, which use Orchestrate Shell (OSH) commands for parallel processing. This log file is specific to a single job. The file name begins with oshjob., followed by the ID of the Application Master in the cluster, followed by an underscore (_) and the number of the job in the order that the Application Manager runs the jobs. For example, the file name oshjob.0009_0 indicates that the Application Master ID is 0009 and the job is the first in the series.
- Connecting to YARN Application Master at node_name:port_number
- Application Master log location is path
If you enable log aggregation by setting the configuration parameter yarn.log-aggregation-enable to true, the log files are moved to HDFS after the Application Master completes. To find the log files on HDFS, retrieve the job-tracking URL from the job log in the Designer client.
Resource Manager and Node Manager log files
If an error occurs at the YARN level, you might have to examine the log files for the Resource Manager and node managers. These files are on the computers that host the Resource Manager and each Node Manager. You would usually consult the Resource Manager log first. From that log, you can determine which Node Manager logs to check, if necessary.
The log files have a default location that differs based on the distribution you use. Unless you know that the default location is not used, look in the default location first. The following table shows the default location for log files for each distribution and the location where the default location can be overridden.
Distribution | Default location for Resource Manager and Node Manager log files | Where the default location can be overridden |
---|---|---|
Hortonworks | /var/log/hadoop-yarn/yarn | In the Ambari user interface, find the YARN Log Dir Prefix property in the YARN configuration file. |
Cloudera | /var/log/hadoop-yarn | In the Cloudera Manager, find the hadoop.log.dir parameter in the Yarn configuration file. |
BigInsights | /var/log/hadoop-yarn | In the Ambari user interface, find the YARN Log Dir Prefix property in the YARN configuration file. |