IBM Support

Mustgather for IBM PureData System for Operational Analytics

Troubleshooting


Problem

What information should be obtained for IBM PureData System for Operational Analytics Environments?

Resolving The Problem




Introduction:

The IBM PureData System for Operational Analytics appliance is composed of many components. Depending on the symptoms encountered there are some details that can be quickly obtained by the customer to provide to support as a starting point. This document may be updated on a regular basis. Note that additional information may be requested that are not currently documented here.

Preparations:

Each time data is requested from this table it is advised to walk through the appliance section. This establishes several environment variables that aid in support gathering. Following this section each time an upload is required will re-establish a new timestamp to help differentiate information sent to IBM. Also, following this method will retain a local history in the /BCU_Share/support directory on the management host.

The steps here assume a healthy management node and dsh is operational. If the management node is not healthy or dsh is not working then these steps will not apply.


Important Information:

What is the PMR number that was opened? The PMR number will be used in the data collection. If data collection is done prior to opening a PMR, then use 00000,000,000 in the PMR variable. Before uploading to IBM that packaged file will need to be renamed with the actual PMR number.

What version of PDOA are you using and what is the current fixpack level? The data collection should gather these details as part of the Appliance mustgather section.

What were the initial symptoms and what was the source of the symptom ?
  • Examples source statements:
    • a) Database user complains of ...
      b) Noticed an alert was sent from PDOA to my alerting system?
      c) User was unable to do ...









    •  

When were the symptoms noticed? This helps to establish a timeline when referencing the logs.

What if any remediation attempts were made to address the symptom prior to calling support?

What if any secondary symptoms were encountered?





Links to Component MustGather  





What to gather:

Appliance Info and MustGather setup.
 
Component /
Symptoms
Notes/Commands
Appliance If the management node is available, obtain this information for all PMRs.

Run as root on management: Environment variables are per session.

export PMR=<PMRNUMBER>
export DDIR=/BCU_share/support/${PMR}/data
export STAMP="date +%Y%m%d_%H%M%S"
export TS=$(${STAMP})
export SDIR=${DDIR}/${TS}

echo ${SDIR}

mkdir -p ${SDIR}

appl_ls_cat > ${SDIR}/appl_ls_cat_$(${STAMP}).txt
hals > ${SDIR}/hals_$(${STAMP}).txt


Packaging and sending to IBM.




Console,pflayer, and fixpack issues.
Console, pflayer issues and fixpack issues. Collect the following: all run as root on management.

pflayer/mi logs:

tar -cvf - -C / $(cd /;find BCU_share/aixappl/ -type f | grep -v appl_odm.log) | gzip > /${SDIR}/${PMR}_BCU_share_aixappl_${TS}.tgz


Console application server logs.

tar -cvf - -C / var/log/applmgmt  | gzip > ${SDIR}/${PMR}_var_log_applmgmt_${TS}.tgz


Verify the logs were collected.

$ ls -lt ${SDIR}
total 68936
-rw-r--r--    1 root     system     31406663 Oct 03 18:36 12345,123,123_var_log_applmgmt_20171003_173118.tgz
-rw-r--r--    1 root     system      3849235 Oct 03 18:36 12345,123,123_BCU_share_aixappl_20171003_173118.tgz

If there is more information to collect, collect it or refer to the 'Package and send the files to IBM.' section to tar and send the files to ibm.
Packaging and sending to IBM.




HA Tools
HALS or issues with hatools. All commands run as root on management.

Grab the Appliance data first. Then grab the TSA data as well. If this is a new or second data collection be sure to go through the appliance section to setup a new data collection directory on the management host in /BCU_share/support/<PMR>/data.

Verify DDIR and SDIR are setup.

(0) root @ host01: 7.1.0.0: /BCU_share/support/12345,123,123/data
$ echo ${SDIR}
/BCU_share/support/12345,123,123/data/20171003_173118

(0) root @ host01: 7.1.0.0: /BCU_share/support/12345,123,123/data
$ echo ${DDIR}
/BCU_share/support/12345,123,123/data


Copy the hatools failover logs and hatools files to the management host.


$ dcp -P -p -R -n ${ALL} /tmp/halog ${SDIR}/
$ dcp -P -p -R -n ${ALL} /usr/IBM/analytics/ha_tools ${SDIR}/



Verify the halog and ha_tools files were copied to the management host.

$ ls -1 ${SDIR}
20171003_15061507057601
20171003_15061507057601._host02
20171003_15061507057601._host03
20171003_15061507057601._host04
20171003_15061507057601._host05
20171003_15061507057601._host06
appl_ls_cat_20171003_173153.txt
ha_tools
ha_tools._host02
ha_tools._host03
ha_tools._host04
ha_tools._host05
ha_tools._host06
halog
halog._host02
halog._host03
halog._host04
halog._host05
halog._host06
hals_20171003_173159.txt

Packaging and sending to IBM.




GPFS
GPFS [ added 2017-10-16 ]

Collect GPFS data when symptoms involve issues with storage or GPFS specifically.

It is also beneficial to include GPFS data collection when there are potential network related issues. GPFS logs can also reveal communication issues..
 
Collect the HA_TOOLs data above:

GPFS data is gathered using the gpfs.snap command.

PDOA has multiple GPFS clusters. In PDOA V1.0.0.3 and earlier the management nodes belong to the foundation.gpfs cluster. In V1.0.0.4 and V1.1.0.0 and higher the management nodes have their own cluster.

GPFS snaps are good if we have identified GPFS issues.

Note if GPFS snap indicates network issues that most likely cause is not the networking, but workload, so if the network is indicated, please also grab getsadata and any nmon, performance data for the time.

# Use the following to get a copying version of the TS=<stamp> string. Since we are using dsh with single quotes, it isn't possible to pass this variable from dsh.

$ echo "TS=$TS"
TS=20170929_162609

# Create the collecting directory on /tmp.
dsh -n ${ALL} "mkdir -p /tmp/${TS}"

# Run the following to collect mmfs.snap data for all clusters.Make sure to modify the 'TS=<date>;' entry at the beginning of the command.


dsh -n ${ALL} 'TS=20171017_024251;mhost=$(/usr/lpp/mmfs/bin/mmgetstate -aL | tail +4 | while read num node rest;do echo ${node};done | sort | head -1);if [ "${mhost}" == "$(hostname)" ];then echo "Collecting data on ${mhost}..";/usr/lpp/mmfs/bin/gpfs.snap -a -d /tmp/${TS};fi'


# The following command can be run to verify progress in a separate session. Remember to update the TS=<stamp> to ensure the correct directory is tracked. You should look to make sure that the latest files are being updated. This command may not work correctly if the data collection is run near midnight as it only sorts on the time.

dsh -n ${BCUDB2ALL} 'TS=20171017_024251;find /tmp/${TS} -ls | sort -k 10 | tail -5'


# Run the following to collect the data on the management host. You only need to append the TS=<stamp> if running in a separate session. Replae $SDIR and $TS as appropriate in the beginning of the command.

dsh -n ${ALL} 'SDIR=/BCU_share/support/44860,442,000/data/20171017_024251;TS=20171017_024251;find /tmp/${TS} -name "all.*.tar" | while read f;do scp -p ${f} 172.23.1.1:${SDIR}/gpfs/$(hostname)_$(basename ${f});done'


Packaging and sending to IBM.





TSA
TSA

Unplanned failovers.

Planned failovers:
hafailover

Failure to start.
hastartdb2
hastartapp
hastartdpm

Failure to stop.
hastopdb2
hastopapp
hastopdpm
Collect the HA_TOOLS data above.

For this we always want a getsadata. Getsadata provides one of the most comprehensive data collections to grab not only TSA and RSCT data, but also AIX data such as errpt and syslogs. It is important to perform the getsadata as soon as possible as log files and trace files will often wrap and important data may be lost.

The getsadata link from TSA is here.

The best strategy is as follows:

a) getsadata on the master.
b) getsadata on the primary node for the service that failed.
c) getsadata on the standby node for the service that failed.
d) getsadata on a good server (preferably in the same domain) for comparison.

If possible, having getsadata for all hosts in the domain provides the most comprehensive view.

In PDOA V1.0.0.3 and earlier, PDOA has two domains, management and core. Management has a maximum of two nodes, but the core may have a singificant number of nodes making complete getsadata collection costly. In PDOA V1.0.0.4 and V1.1.0.0 and higher PDOA moves the core to use one domain per rack. So the maximum number of hosts to collect data from will always be 5 or less in V1.1.0.0 and 4 or less in V1.0.0.4.

In addition to getsadata we should also collect the files in /tmp/halog on all hosts in the same domain. This provides a more comprehensive history of the environment.

Identify the service that is having the issue. Management services such as DPM (aka OPM) and Warehouse Tools (ISW) run on the management hosts in the management domains.

The core instance is running on the core nodes.

These instructions use the ${ALL} variable to access all hosts. This ALL variable can be replaced as follows if targeting management or core domains.

management="${BCUMGMT},${BCUMGMTSTDBY}"
core="${BCUDB2ALL}"


1. Create a timestamp. This will be used for this collection to make it easier to pull data from each of the nodes down to the management node. Modify the rest of the commands with this timestamp. The TS=<tsmp> can then be replaced in the following dsh commands to ensure a consistent timestamp is used throughout the command line. Note that due using dsh with single quotes means that you can't pass the ST variable from the root session, so this will require some minimal replacement at the beginning of the command line. The TS and SDIR directories were established in the Appliance section.

$ echo "TS=$TS"
TS=20170929_162609


2. Create the directory on all hosts.

dsh -n ${ALL} "mkdir -p /tmp/${TS}"

3. Edit the commands below to change the -outdir to use the specfied directory.

4. The following command will generate getsadata on all master nodes in the environment. Remember to replace the TS=<stmp> in the beginning with the proper timestamp. This is required as the dsh commands are sent to each host in single quotes.

dsh -n ${ALL} 'TS=20170929_162609;master=$(lssrc -ls IBM.RecoveryRM | grep -i master | cut -d: -f 2 | while read h rest;do echo ${h};done);if [ "${master}" == "$(hostname)" ];then echo collecting data;getsadata -all -outdir /tmp/${TS} ;fi'

5. The following command will generate getsadata on all non-master nodes in the environment.

dsh -n ${ALL} 'TS=20170929_162609;master=$(lssrc -ls IBM.RecoveryRM | grep -i master | cut -d: -f 2 | while read h rest;do echo ${h};done);if [  ! "${master}" == "$(hostname)" ];then echo collecting data;getsadata -all -outdir /tmp/${TS};fi'


6. Verify the output is available on the available nodes.

dsh -n ${ALL} 'TS=20170929_162609;ls /tmp/${TS}" 2>&1 | dshbak -c

Example:
=========
$ dsh -n ${ALL} 'TS=20170929_162609;ls /tmp/${TS}/*.Z' | sort
host01: /tmp/20170929_162609/100317_150820-host01-2-m-sa_data.tar.Z
host02: /tmp/20170929_162609/100317_150821-host02-2-m-sa_data.tar.Z
host03: /tmp/20170929_162609/100317_161328-host03-1-sa_data.tar.Z
host04: /tmp/20170929_162609/100317_161328-host04-1-sa_data.tar.Z
host05: /tmp/20170929_162609/100317_150821-host05-1-m-sa_data.tar.Z
host06: /tmp/20170929_162609/100317_161328-host06-2-sa_data.tar.Z


7. Copy the files to the management host. Again, replacing the TS=<STMP> with the one used in this. ${SDIR} was defined in the Appliance step. The ${SDIR} timestamp may be the same or different.

TS=20170929_162609;dcp -P -p -R -n ${ALL} "/tmp/${TS}" ${SDIR}/

8. Verify the contents of the ${SDIR} directory. The dcp will copy the directory from each of the servers. There may be other files from other mustgather steps. dcp will add the _<hostname> to all of the hosts that are not the management host if using ${ALL} or the management hosts in the dsh command.

$ ls -t ${SDIR} | sort
20170929_162609
20170929_162609._host02
20170929_162609._host03
20170929_162609._host04
20170929_162609._host05
20170929_162609._host06
appl_ls_cat_20171003_15051507057553.txt
hals_20171003_15051507057559.txt


$ find ${SDIR} -name "*.Z"
/BCU_share/support/12345,123,123/data/20170929_162609/100317_150820-host01-2-m-sa_data.tar.Z
/BCU_share/support/12345,123,123/data/20170929_162609/20170929_162609._host02/100317_150821-host02-2-m-sa_data.tar.Z
/BCU_share/support/12345,123,123/data/20170929_162609/20170929_162609._host03/100317_161328-host03-1-sa_data.tar.Z
/BCU_share/support/12345,123,123/data/20170929_162609/20170929_162609._host04/100317_161328-host04-1-sa_data.tar.Z
/BCU_share/support/12345,123,123/data/20170929_162609/20170929_162609._host05/100317_150821-host05-1-m-sa_data.tar.Z
/BCU_share/support/12345,123,123/data/20170929_162609/20170929_162609._host06/100317_161328-host06-2-sa_data.tar.Z


9. Remove the TSA support files from each hosts TMP directory. Be careful with this command and verify that the /tmp/<TS> stamp is correct.

dsh -n ${ALL} 'rm -rf /tmp/20170929_162609'



10. Proceed to the Package and send the files to IBM section below.

Packaging and sending to IBM.





Performance Data [ Added 2017-11-06 ]
Gathering some quick performance data on PDOA environments.

PDOA is shipped with topasrec collecting some performance data every 5 minutes in either /etc/perf/daily (V1.0) or /var/perf/daily.

The collection is 5 to 7 days until it is rotated out.

This can be confirmed by running the following command:

dsh -n ${ALL} 'grep topasrec /etc/inittab'

Collect this information when there is indications of performance issues or symptoms pointing to network issues like network overload or congestion.
1. Verify you have setup the collection environment.

echo ${SDIR}
ls -lad ${SDIR}

2. Create a perf directory to collect the daily directory from each host.

mkdir ${SDIR}/perf

3. Convert all performance files into 'nmon' capable csv files.

PDOA V1.0:
===========
dsh -n ${ALL} 'find /etc/perf/daily -name "$(hostname)*topas" | while read f;do topasout -a ${f};done'

PDOA V1.1:
===========
dsh -n ${ALL} 'find /var/perf/daily -name "$(hostname)*topas" | while read f;do topasout -a ${f};done'

3. Collect the data onto /BCU_share.

PDOA V1.0:
===========
dcp -P -p -R -n ${ALL} "/etc/perf/daily" "${SDIR}/perf"

PDOA V1.1:
============
dcp -P -p -R -n ${ALL} "/var/perf/daily" "${SDIR}/perf"


4. Verify that the .csv files were copied.

find ${SDIR} -type f -name "*.csv"
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01_171031.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01_171101.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01_171102.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01_171103.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01_171104.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01_171105.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01_171106.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01mgt_160507.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01mgt_160508.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01mgt_160509.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01mgt_160510.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01mgt_160511.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01mgt_160512.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily/hostname01mgt_160513.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname02/hostname02_171031.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname02/hostname02_171101.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname02/hostname02_171102.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname02/hostname02_171103.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname02/hostname02_171104.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname02/hostname02_171105.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname02/hostname02_171106.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname04/hostname04_171031.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname04/hostname04_171101.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname04/hostname04_171102.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname04/hostname04_171103.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname04/hostname04_171104.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname04/hostname04_171105.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname04/hostname04_171106.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname05/hostname05_171031.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname05/hostname05_171101.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname05/hostname05_171102.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname05/hostname05_171103.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname05/hostname05_171104.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname05/hostname05_171105.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname05/hostname05_171106.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname06/hostname06_171031.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname06/hostname06_171101.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname06/hostname06_171102.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname06/hostname06_171103.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname06/hostname06_171104.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname06/hostname06_171105.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname06/hostname06_171106.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname07/hostname07_171031.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname07/hostname07_171101.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname07/hostname07_171102.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname07/hostname07_171103.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname07/hostname07_171104.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname07/hostname07_171105.topas.csv
/BCU_share/support/12345,123,123/data/20171106_182145/perf/daily._hostname07/hostname07_171106.topas.csv


5. Remove the generated csv files.

PDOA V1.0:
===========
dsh -n ${ALL} 'find /etc/perf/daily -name "$(hostname)*topas.csv" | xargs rm'

PDOA V1.1:
===========
dsh -n ${ALL} 'find /var/perf/daily -name "$(hostname)*topas.csv" | xargs rm'


6. Proceed to the Package and send the files to IBM section below.

Packaging and sending to IBM.

# Back to top link




Storage Enclosure issues (Flash900, V7000 Gen1 and V7000 Gen2)
Collecting snaps from V7000 and Flash900 Storage Enclosures.
This describes how to collect snap information from the Storage Enclosures in a PDOA environment.
The following assumes the that management node is online and the storage enclosures are listening to the management ip addresses.
The enclosures are setup to call home and should have sent the appropriate data so support as part of that process.
However, if it did not, having the snaps ready and available for support and also a birds eye view of all of the storage in the environment will not hurt.
Snap times can vary from a few minutes to 20 or 30 minutes depending on the enclosure.
This method can be used to provide data to storage support even when they ask for specific snaps within the web based UI.
1. Verify you have setup the collection environment.

echo ${SDIR}
ls -lad ${SDIR}
2. Identify the list of storage enclosures that have issues. This will list the enclosures with the MTM and enclosure serial numbers and a list of unfixed events. This is a birds eye view of all of the storage in the system.

$ appl_ls_hw -r storage -A M_IP_address | sort | sed 's|"||g' | while read ip;do echo "**** ${ip} ****"; ssh -n superuser@${ip} 'lsenclosure;lseventlog -alert yes -message no';done 2>&1 | tee ${SDIR}/storage_$(${STAMP}).txt
**** 172.23.1.204 ****
id status type      managed IO_group_id IO_group_name product_MTM serial_number total_canisters online_canisters total_PSUs online_PSUs drive_slots total_fan_modules online_fan_modules
1  online expansion yes     0           io_grp0       2076-24F    7820C0A       2               2                2          2           24          0                 0
2  online control   yes     0           io_grp0       2076-524    78209PX       2               2                2          2           24          2                 2
sequence_number last_timestamp object_type object_id object_name copy_id status fixed event_id error_code description
9000010         190912213130   cluster               V7_00               alert  no    079501   2700       Failed to synchronize cluster time to NTP server
**** 172.23.1.205 ****
id status type    product_MTM serial_number total_canisters online_canisters online_PSUs drive_slots
1  online control 9840-AE2    1351337       2               2                2           12
**** 172.23.1.206 ****
id status type      managed IO_group_id IO_group_name product_MTM serial_number total_canisters online_canisters total_PSUs online_PSUs drive_slots total_fan_modules online_fan_modules
1  online expansion yes     0           io_grp0       2076-24F    7820C0B       2               2                2          2           24          0                 0
2  online control   yes     0           io_grp0       2076-524    782093Y       2               2                2          2           24          2                 2
**** 172.23.1.207 ****
id status type    product_MTM serial_number total_canisters online_canisters online_PSUs drive_slots
1  online control 9840-AE2    1351347       2               2                2           12
**** 172.23.1.208 ****
id status type      managed IO_group_id IO_group_name product_MTM serial_number total_canisters online_canisters total_PSUs online_PSUs drive_slots total_fan_modules online_fan_modules
1  online expansion yes     0           io_grp0       2076-24F    7820C0K       2               2                2          2           24          0                 0
2  online control   yes     0           io_grp0       2076-524    78209RC       2               2                2          2           24          2                 2
**** 172.23.1.209 ****
id status type    product_MTM serial_number total_canisters online_canisters online_PSUs drive_slots
1  online control 9840-AE2    1351355       2               2                2           12

$ find ${SDIR}/
/BCU_share/support/12345/data/20190912_203136/
/BCU_share/support/12345/data/20190912_203136/storage_20190912_203216.txt

3. If you know the storage enclosure with the issue, login to the enclosure and run the following. For example, if the enclosure with an issue is 172.23.1.204 we would run the following as root on the management host.

$ time ssh -n superuser@172.23.1.204 'svc_snap clean;svc_snap a'
Deleted all snap*.tgz files in /dumps.
Collecting data
Packaging files
Snap data collected in /dumps/snap.78209PX-2.190912.214114.tgz

real    3m0.46s
user    0m0.03s
sys     0m0.01s

4. Copy the snap listed above into the collection directory.

$ scp superuser@172.23.1.204:/dumps/snap.78209PX-2.190912.214114.tgz ${SDIR}/
snap.78209PX-2.190912.214114.tgz                                                                                                                                                                                                       99%  153MB  76.4MB/s   00:02

$ find ${SDIR}/ -ls
172119    1 drwxr-xr-x  2 root      system         256 Sep 12 20:34 /BCU_share/support/12345/data/20190912_203136/
172121 156483 -rw-r--r--  1 root      system    160238423 Sep 12 20:34 /BCU_share/support/12345/data/20190912_203136/snap.78209PX-2.190912.214114.tgz
172120    3 -rw-r--r--  1 root      system        2699 Sep 12 20:32 /BCU_share/support/12345/data/20190912_203136/storage_20190912_203216.txt

5. The SNAP filenames includes the serial number of the enclosure and the date of the snap. You may be asked to upload this snap directly to the PMR by Storage Support or you can include the snaps as part of this mustgather packaging.





Packaging and sending to IBM
Package and send the files to IBM.

2017-10-16:
Updated tar command to use ${TS} instead of ${SDIR}. We don't need the full path with "/BCU_share/support/<PMR>/data" in the tar file, just the directories in the ${TS} directory. This reduces the long paths when unpacked for support.
1. Tar up the contents of the directory. Your contents will be different depending on the components collected.

cd ${DDIR}

(0) root @ host01: 7.1.0.0: /BCU_share/support/12345,123,123/data
$ cd ${DDIR}

(0) root @ host01: 7.1.0.0: /BCU_share/support/12345,123,123/data
$ ls -l
total 8
drwxr-xr-x    8 root     system         4096 Oct 03 17:44 20170929_162609

$ tar -cvf ${DDIR}/${PMR}_data_${TS}_pdoa_support.tar -C ${DDIR} ${TS}
a 20171003_15061507057601
a 20171003_15061507057601/100317_150820-host01-2-m-sa_data.tar.Z 167742 blocks.
a 20171003_15061507057601/vital_info.out 3 blocks.
a 20171003_15061507057601._host02
a 20171003_15061507057601._host02/100317_150821-host02-2-m-sa_data.tar.Z 140419 blocks.
a 20170929_162609/20171003_15061507057601._host02/vital_info.out 3 blocks.
a 20171003_15061507057601._host03
a 20171003_15061507057601._host03/100317_161328-host03-1-sa_data.tar.Z 153158 blocks.
a 20171003_15061507057601._host03/vital_info.out 3 blocks.
a 20171003_15061507057601._host04
a 20171003_15061507057601._host04/100317_161328-host04-1-sa_data.tar.Z 133687 blocks.
a 20171003_15061507057601._host04/vital_info.out 3 blocks.
a 20171003_15061507057601._host05
a 20171003_15061507057601._host05/100317_150821-host05-1-m-sa_data.tar.Z 141850 blocks.
a 20171003_15061507057601._host05/vital_info.out 3 blocks.
a 20171003_15061507057601._host06
a 20171003_15061507057601._host06/100317_161328-host06-2-sa_data.tar.Z 120883 blocks.
a 20171003_15061507057601._host06/vital_info.out 3 blocks.
a appl_ls_cat_20171003_173153.txt 1 blocks.
a hals_20171003_173159.txt 4 blocks.


2. Verify the directory is there. There may be multiple directories and tar files over time for other support issues.

$ ls -l ${DDIR}
total 857808
-rw-r--r--    1 root     system    439193600 Oct 03 17:54 12345,123,123_data_20170929_162609_pdoa_support.tar
drwxr-xr-x    8 root     system         4096 Oct 03 17:44 20170929_162609


3. Send the recently packaged tar files to IBM using the following link: https://www.ecurep.ibm.com/app/upload

4. To preserve space, remove the directory pointed to by ${SDIR} after the tar file is created.

[{"Product":{"code":"SSH2TE","label":"PureData System for Operational Analytics A1801"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"--","Platform":[{"code":"PF002","label":"AIX"}],"Version":"1.0;1.1","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
17 October 2019

UID

swg22009170