IBM Performance Management

Configuring Hadoop nodes

You must configure settings on each Hadoop node so that the nodes can send data to the Hadoop agent.

About this task

To update the Hadoop node settings for communicating with the agent, complete these tasks for each Hadoop node:
  1. Copy the Plugin.jar file to the location in each Hadoop node where other Hadoop-related JAR files are installed.
    For example, for the Hadoop cluster that is managed by BigInsights, all the Hadoop-related JAR files are installed at /usr/iop/4.0.0.0/hadoop/lib.
    The Plugin.jar file is available at the following location where the Hadoop agent is installed:
    • Linuxinstall_dir\lx8266/h8/bin
    • AIXinstall_dir\aix526/h8/bin
    • Windowsinstall_dir\TMAITM6_x64
  2. Update the Hadoop metrics in the hadoop-metrics2.properties file.
    Tip: You can use the Admin console for updating this file. For information about using the Admin console, see Updating Hadoop metrics by using Cloudera Manager.

Procedure

To update the hadoop-metrics2.properties file by using the command line, complete the following steps:

  1. At a Hadoop node, run the following command to locate the hadoop-metrics2.properties file: find / -name hadoop-metrics2.properties*. For example, if the Hadoop cluster is managed by Apache Ambari, the hadoop-metrics2.properties file is available at the following locations:
    • /etc/hadoop/conf.empty/hadoop-metrics2.properties
    • /usr/iop/4.0.0.0/etc/hadoop/conf.empty/hadoop-metrics2.properties
    • ambari_server_installation_directory/ambari-server/resources/stacks/BigInsights/4.0/hooks/before-START/templates/hadoop-metrics2.properties.j2
    • ambari_server_installation_directory/ambari-server/resources/stacks/BigInsights/4.0/services/HBASE/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
    • ambari_server_installation_directory/ambari-server/resources/stacks/BigInsights/4.0/services/HBASE/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
    • ambari_server_installation_directory/ambari-agent/cache/stacks/BigInsights/4.0/hooks/before-START/templates/hadoop-metrics2.properties.j2
    • ambari_server_installation_directory/ambari-agent/cache/stacks/BigInsights/4.0/services/HBASE/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
    • ambari_server_installation_directory/ambari-agent/cache/stacks/BigInsights/4.0/services/HBASE/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
  2. In a text editor, open the hadoop-metrics2.properties file, and complete the following steps:
    Attention: If you are not sure about the exact hadoop-metrics2.properties file to be updated, open any of the files that is available at the specified locations.
    1. Add the following code in the file. 
      *.sink.ibm.class=com.ibm.hadoop.Plugin
*.period=30
*.sink.ibm.AgentLocation=1.0.0.0
*.sink.ibm.AgentPort=6789
*.sink.ibm.SendPeriod=30
*.sink.ibm.Debug=true
namenode.sink.ibm.test=true
datanode.sink.ibm.test=true
jobtracker.sink.ibm.test=true
tasktracker.sink.ibm.test=true
maptask.sink.ibm.test=true
mrappmaster.sink.ibm.test=true
nodemanager.sink.ibm.test=true
reducetask.sink.ibm.test=true
resourcemanager.sink.ibm.test=true
secondarynamenode.sink.ibm.test=true
jobhistoryserver.sink.ibm.test=true
    2. Update values of the following metrics, and save the changes:
      • sink.ibm.AgentLocation - Specify the IP address of the server where the agent is installed.
        Recommendation: Specify the static IP address of the server where the agent is installed.
      • sink.ibm.AgentPort - Specify the port number that the agent uses to listen to the data from the socket clients.
    3. Copy this updated code and add it to each hadoop-metrics2.properties file that you located in step 1.
  3. Repeat steps 1 and 2 for all nodes in the Hadoop cluster.
  4. Restart the Hadoop services.
    • If you are using Cloudera Manager, complete these steps:
      1. Log in to Cloudera Manager to access the NameNode. The web address with the default port is http://IP_address_NameNode: 7180/cmf/login.
      2. Restart the Hadoop cluster.
    • If you are using Apache Ambari, complete these steps:
      1. Log in to Ambari to access the NameNode. The web address with the default port is http://IP_address_NameNode:8080/#/login.
      2. Restart the Hadoop services.
    Remember: Repeat this step each time when you update the hadoop-metrics2.properties file.

Updating Hadoop metrics by using Cloudera Manager

You can also use the Admin console of Cloudera Manager to update the hadoop-metrics2.properties file apart from using the command line.

Procedure

  1. Log in to Cloudera Manager. The web address with the default port is http://IP_address_NameNode: 7180/cmf/login.
  2. On the Home tab, click HDFS.
  3. On the Configuration tab, type hadoop-metrics2 in the Search filter. The search results display all the instances of hadoop-metrics2 file.
  4. Click Edit Individual Values, and copy the following code in the text area of each hadoop-metrics2.properties file after you update values of the sink.ibm.AgentLocation and sink.ibm.AgentPort parameters. 
    *.sink.ibm.class=com.ibm.hadoop.Plugin
*.period=30
*.sink.ibm.AgentLocation=1.0.0.0
*.sink.ibm.AgentPort=6789
*.sink.ibm.SendPeriod=30
*.sink.ibm.Debug=true
namenode.sink.ibm.test=true
datanode.sink.ibm.test=true
jobtracker.sink.ibm.test=true
tasktracker.sink.ibm.test=true
maptask.sink.ibm.test=true
mrappmaster.sink.ibm.test=true
nodemanager.sink.ibm.test=true
reducetask.sink.ibm.test=true
resourcemanager.sink.ibm.test=true
secondarynamenode.sink.ibm.test=true
jobhistoryserver.sink.ibm.test=true
  5. Click Save Changes.
  6. Repeat steps 2 through 5 for YARN and MapReduce services.
  7. Click the Home tab.
  8. From the cluster name list, select Restart.
    Remember: Repeat this step whenever you update hadoop-metrics2.properties files.