Steps

On the HMC, setup "Data Collection" for your Servers

Log in to your HMC and the "All Systems" view

Select a POWER8 and POWER9 server and switch on "Data Collection":

1. Click the tiny "pie chart" button.
2. 
3. If the upper right "Data Collection" is light green colored, then it is already on.
4. If not, click the "O" button.  Note: "O" means off and a "|" means on.
5. 
6. You get a panel stating "Enable Data Collection"
7. 
8. Warning: This "Enabling" does not happen immediately but takes 10 minutes as a background activity.  The user interface does not get immediately updated, so don't bother trying to enable it again.
9. Check that it worked about 20 minutes later for the green color and a "|".
10. 

Data Collection

Each time you run the nextract_plus.py program, it collects the Performance data

• For the previous two hours
• At a sample rate of every five-minutes
• In an aggregated format - the average over the five-minute interval.

If you run the program, once an hour then you get one ours worth of overlapping data.  This overlap is not a problem as the overlapped data is exactly that same numbers. This overlapping data can be reloaded into InfluxDB with no problems and no duplication of data. To reduce the compute time running the program, you could run the program once every two hours but then you have to wait up to two hours to see recent data. You have to decided what you want to do.  There is not point in running it faster than once every 5 minutes.

I suggest:

• Running hourly using your crontab
  • Once an hour, insures a good overlap that can help, if a virtual machine running the program or the machine running InfluxDB service is unavailable, being updated or rebooting.
• And if you want the latest data, right now, then run the program by hand.
• Make up your own mind. 
  • Capturing too often, does no harm but waits compute time.
  • Capturing not often enough, means gaps in your graphs.

$ influx
> show databases
> create database nextractplus
> show databases
> exit
$ 

• These settings are determined during the initial InfluxDB setup that uses the InfluxDB graphical user interface (GUI)
• Make a note of the organization, bucket, and token settings

The code and files:

1. hmc_pcm.py - unchanged from previous versions
2. nextract_plus.py - new code to extract all the statistics.
   The previous version did the basics at Server level and totals for the LPAR (VM) level.
3. nextractplus_v41_Performance-<number>.json The Grafana Dashboard definition, which matches this new code level.

The configuration parameters for the three nextractplus Python programs are stored in a JSON format config file.

To create a config file run:

$ ./nextract_plus.py >example_config.json

$ cat example_config.json
Usage:  ./nextract_plus.py version 41
./nextract_plus.py filename.json - use the JSON file to find the parameters
Example config.json
{
"hmc_hostname": "hmc15",
"hmc_user": "pcmadmin",
"hmc_password": "panda123sausages!",
"output_nchart": 0,
"output_json": 0",
"output_csv": 0,
"comment1": "Valid setting for output_influx = 0 (off), 1 or 2",
"output_influx": 1,
"comment2": "InfluxDB 1.x options",
"ihost": "myinflux",
"iport": "8086",
"idbname": "nextractplus",
"iuser": "fred",
"ipassword": "blogs",
"comment3": "InfluxDB 2.x options",
"iurl": "https://influx.acme.com:8086",
"ibucket": "default",
"iorg": "IBM",
"itoken": "123456780..abcdefg_etc.",
"comment4": "other parameters are ignored",
"comment5": "last name=value line has no ending comma"
}

Remove those top three lines from the config file so it looks like this:

{
"hmc_hostname": "hmc15",
"hmc_user": "pcmadmin",
"hmc_password": "panda123sausages!",
"output_nchart": 0,
"output_json": 0",
"output_csv": 0,
"comment1": "Valid setting for output_influx = 0 (off), 1 or 2",
"output_influx": 1,
"comment2": "InfluxDB 1.x options",
"ihost": "myinflux",
"iport": "8086",
"idbname": "nextractplus",
"iuser": "fred",
"ipassword": "blogs",
"comment3": "InfluxDB 2.x options",
"iurl": "https://influx.acme.com:8086",
"ibucket": "default",
"iorg": "IBM",
"itoken": "123456780..abcdefg_etc.",
"comment4": "other parameters are ignored",
"comment5": "last name=value line has no ending comma"
}

Notes on these parameters:

1. The program parameters are JSON format file. JSON files are easy to parse into a Python dictionary. Any mistakes in the JSON format stop Python evaluating the parameters. Double check you have matching double quotation marks for string values and the item before the final curly closing brace does NOT have a comma.
2. Set the hmc_hostname to the hostname of your Hardware Management Console (HMC). You need one config file for each.
3. Set the  hmc_username, and hmc_password to an account on the HMC with full hscroot access.
4. If you want a html webpage output, which uses Google charts for the graphs, set output_nchart = 1. 
5. If you want the stats in JSON format, set output_json = 1. 
6. If you want a comma-separated values file output, set output_csv = 1. 
7. If you want the data sent to InfluxDB version 1, set output_influx = 1.
8. If you want the data sent to InfluxDB version 2, set output_influx = 2.
    
9. If output_influx = 1 then you need to set the following details
   1. Set ihost  - To the InfluxDB servers hostname.
   2. Set iport - To the network port being used by Influx 1. The default is 8086.
   3. Set idatabase - To the database that you created in InfluxDB 1. I recommend "nextractplus".
   4. If you set a username and password for InfluxDB access, then set iuser and ipassword. If iuser and ipassword are not needed, then they are ignored by InfluxDB.

$ influx
> show databases
> create database nextractplus
> show databases
> exit
$ 

Have all the Python programs in the same directly.

Simply run the Python code: ./nextract_plus.py

The output starts like this (the sample includes a server called P9-S924-red):

$ ./nextract_plus.py
-> Logging on to hmc15 as hmc_username nag
-> Get Preferences
-> Parse Preferences
-> ALL servers:
-> Server name=P8-S822-lemon    agg=false longterm=false - remove 
-> Server name=P9-S924-red      agg=true  longterm=true  - OK 
-> Server name=P7-p730b-green   agg=false longterm=false - remove 
-> Server name=P7-p710b-cyan    agg=false longterm=false - remove 
-> . . . 
-> Servers with Perf Stats

--> Server=1 Getting filenames for P9-S924-red
---> Received 17 file(s) in 3.26 seconds
---> Server=P9-S924-red File=1 ManagedSystem_0870cd27-ae34-3421-89e3-4825132237ce_20210119T104500+0000_20210119T124400+0000_30.json
---> Server=P9-S924-red File=2 LogicalPartition_5F0A663B-4EB1-424F-8643-7F417A8C9C97
---> Server=P9-S924-red File=3 LogicalPartition_085A4C6C-2ADF-4016-A33B-6602FA85665F
---> Server=P9-S924-red File=4 LogicalPartition_20D78650-9D62-415E-A882-C903A3DBAA2F
--->  . . .


ManagedSystem
---> Save readable JSON File=1 bytes=5088640
----> ServerInfo name=P9-S924-red mtms=9009-42A*7804930 type=Processed frequency=30 seconds
----> ServerInfo----> ServerInfo Date=2021-01-19 start=10:45:00 end=12:44:00
----> Server=P9-S924-red Filenames XML File=2 bytes=1319
----> Server=P9-S924-red Filenames XML File=3 bytes=1319
----> Server=P9-S924-red Filenames XML File=4 bytes=1319
----> Server=P9-S924-red Filenames XML File=5 bytes=1319
---->  . . .

Influx added 25089 records Server=P9-S924-red

The output is then followed by the details for each server.

If you get errors like this

* ERROR Server:P8-E850-ruby LPAR vm96withSSDviaVIOS3-4: status=2 errId=4002 from=VirtualIOServer
* ERROR Description="=vios: 3 is not in running state on Managed System: 8408-E8E*21D494V"

* ERROR Server:P8-E850-ruby LPAR vm96withSSDviaVIOS3-4: status=2 errId=3004 from=ManagedSystem
* ERROR Description="=Failed to fetch SRIOV adapter physical port statistics due to a technical error."

Then, nextract_plus keeps going but you need to investigate.

The first ERROR line points out that I have a Virtual I/O Server that is shut down. In my example case, it is a temporary test VIOS so that is not unexpected.

The second ERROR needs to be reporting by a PMR to be investigated

Note on ERROR:

• The code eliminates the reporting of duplicates otherwise the same error can be reported for every LPAR = 100's of times.
• Apparently, the HMC reports issues that are not a hard error, more like warnings or observations.

Setting up the Grafana Dashboard:

First, add the "nextractplus" InfluxDB database as a data source:

Click "Add data source" button:

Select InfluxDB from the list of support data sources:

Add the data source detail - I used mainly the defaults:

Add the Grafana Dashboard

Next, add the working example of the nextract_plus Grafana Dashboard:

Click "Upload JSON file" button:

Select the file nextract_plus_v20-1611011229698.json from the downloaded the .zip compressed file. Check the data source (lowest field) has the "nextractplus" as the data source:

And click the "Import" button.

The graph dashboard is displayed.  Select a time-range suitable for the amount of data that you collected so for example, the last hour or 2 days, depending on how load you have regularly run nextract_plus.

The nextract_plus Dashboard

Here are the first few screens on my 27-inches screen:

Notes on the nextract_plus Dashboard:

1. Select a suitable time range.
2. There are many pages of graphs to scroll down too.
3. Grafana does "lazy displaying".  The term means it does not draw the graphs that are not visible on the screen to reduce spikes on your workstation CPU as the dashboard is first displayed.  If you scroll down and the graph area is visible, it takes a few seconds to draw them.
4. The three dashboard variables at the top named "Servername", "viosname", and "LPARname" are not interlinked. So the VIOS named might not be on the Server selected. The same goes for the "LPARname."  If you select a VIOS that is not on the selected Server, then you get "N/A" or "No Data" warnings.
5. The "Graphs including All LPARS" means the LPAR on the Selected Server. These graphs show the LPARs on the Server selected - which is useful. But on Servers with 100's of LPARs is going to become a mess due to too many names and graph lines - best to zoom in on the graph.  Hint: hover the cursor over the graph and type: v
6. The time per instruction often shows nothing unless the server is busy.
7. The Dashboard needs more work.

Reporting problems:

1. Make sure you have the usernames and password correct by trying them by hand.
2. Switch on debug output in the "nextract_plus.py" program "debug=1" 
3. Run the program and save all the output:

   ./nextract_plus.py config.json  1>plus.out   2>plus.err

4. Return the two files
5. Run the following "influx" commands.  These commands are interesting, even if you have no problems as it confirms you have data in InfluxDB.

6. $influx
   > show databases
   > use nextractplus
   > show measurements
   
   > select count(*) from server_details
   > select count(*) from vios_details
   > select count(*) from lpar_details
   
   > select LAST(*) from server_details
   > select LAST(*) from vios_details
   > select LAST(*) from lpar_details
   
   > exit

7. Return the output