Example: Using resource usage thresholds and PE restart recommendations to balance workloads

Scenario: You see that the workload is not balanced across resources. You want to move PEs from the resources that are overloaded to a resource that is underloaded.

1. View the current metrics for each resource in your system by running the streamtool capturestate command. Look for the values of these metrics under the <host>element:
   • cpuUtilization
   • memoryUtilization
   • networkUtilization

   Example command: streamtool capturestate --select hosts=metrics

   On one of the resources, you see that the CPU usage and memory usage are high:

   …
       <metric name="cpuUtilization" lastChangeObserved="1504302827" userDefined="false">
         <metricValue xsi:type="streams:longType" value="90"/>
       </metric>
   …
       <metric name="memoryUtilization" lastChangeObserved="1504302827" userDefined="false">
         <metricValue xsi:type="streams:longType" value="90"/>
       </metric>
      …
       <metric name="networkUtilization" lastChangeObserved="1504302827" userDefined="false">
         <metricValue xsi:type="streams:longType" value="15"/>
       </metric>
   …

2. Modify the upper thresholds for CPU usage and memory usage:

   streamtool setproperty resourceCpuUsageUpperThreshold=75 resourceMemoryUsageUpperThreshold=50

3. Run the streamtool lsrestartrecs command to view PE restart recommendations for the instance. Example output:

   Threshold: INSTANCE
     Restart Priority JobName                       PE Current Resource
     none             application::TradesAppCloud_0 0  streamsqse.localdomain
     preferred        application::TradesAppCloud_0 1  streamsqse.localdomain 
     high             application::TradesAppCloud_0 2  streamsqse.localdomain 

   Processing element ID 1 has a preferred recommendation, which means that it is the best candidate to be restarted. Processing element ID 2 has a high recommendation, which means the current resource running the PE is overloaded and there is at least one candidate resource that is underloaded.

4. Stop the processing elements ID 1 and ID 2:

   streamtool stoppe 1 2

5. Restart the processing elements ID 1 and ID 2:

   streamtool restartpe 1 2 

6. After a representative amount of the workload has been processed, repeat step 1 to confirm that moving the PEs from overloaded resources to underloaded resources provides the load balancing that you want. Then you can optionally modify the thresholds again and view and run the PE restart recommendations.