Troubleshooting
Problem
ATTENTION: This MustGather has been updated to utilize the new python based apic-mustgather tool. This new tool provides significant improvements:
- Reduction in the amount of time it takes the tool to run to completion
- Simplification to no longer require passing multiple parameters. By default, it will automatically collect:
- data from all namespaces relevant to API Connect
- diagnostic data for each subsystem present in the cluster
- data specific to OVA form factor
Resolving The Problem
Each section in this document contains the instructions to collect the MustGather data for IBM API Connect on the associated subsystem. This data is required by IBM Support to effectively diagnose and resolve issues.
- Installation or Upgrade Issue (all subsystems): VMware deployment
- Installation or Upgrade Issue (all subsystems): OpenShift, IBM Cloud Pak for Integration, or Kubernetes deployment
- Management subsystem
- Developer Portal subsystem
- Analytics subsystem
- Gateway subsystem: OpenShift, IBM Cloud Pak for Integration, or Kubernetes deployment
- Gateway subsystem: VMware deployment or physical appliance
Installation or Upgrade Issue (all subsystems): VMware deployment
- Download the apic-mustgather tool with the following command:
curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
- Run the
apic-mustgather
tool, and note the location where the output file is saved:- Via SSH, connect to the target appliance to which the
apic-mustgather
tool was downloaded and change to root user:sudo -i
- Change to the directory where the
apic-mustgather
tool was downloaded to the appliance and add execute permission:chmod +x apic-mustgather
- Execute the following command:
./apic-mustgather
- Via SSH, connect to the target appliance to which the
- As root user, gather the status of apic:
apic status > apic_status.out
- As root user, gather the version of apic:
apic version > apic_version.out
- Upload the following to the case:
- Any error messages received from running the
apicup subsys install
command - /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
- apic_status.out
- apic_version.out
- Archive file of the apicup project directory
- Any error messages received from running the
Installation or Upgrade Issue (all subsystems): OpenShift, IBM Cloud Pak for Integration, or Kubernetes deployment
-
apic-mustgather tool with the following command:Download the
curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
- Run the
apic-mustgather
tool, and note the location where the output file is saved:- Change to the directory where the
apic-mustgather
tool was downloaded and add execute permission:chmod +x apic-mustgather
- Execute the following command:
./apic-mustgather
- Change to the directory where the
- Upload the following to the case:
- /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
Management subsystem
- Download the apic-mustgather tool with the following command:
curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
- Reproduce the problem
- Run the
apic-mustgather
tool, and note the location where the output file is saved:- OVA deployment:
apic-mustgather
tool was downloaded and change to root user:sudo -i
Via SSH, connect to the target appliance to which the - Change to the directory containing the
apic-mustgather
tool and add execute permission:chmod +x apic-mustgather
- Execute the following command:
./apic-mustgather
- OpenShift, IBM Cloud Pak for Integration deployment or Native Kubernetes deployment:
apic-mustgather
tool was downloaded and add execute permission:chmod +x apic-mustgather
Change to the directory where the - Execute the following command:
./apic-mustgather
- OVA deployment:
- Upload the following to the case:
- /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
- Steps to reproduce the problem
- Time that the error occurred or start/stop time of reproducing the error
- Screen capture of error (if applicable)
Portal subsystem
- Download the apic-mustgather tool with the following command:
curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
- Reproduce the problem
- Run the
apic-mustgather
tool, and note the location where the apic-mustgather output file is saved:- OVA deployment:
apic-mustgather
tool was downloaded and change to root user:sudo -i
Via SSH, connect to the target appliance to which the - Change to the directory containing the
apic-mustgather
tool and add execute permission:chmod +x apic-mustgather
- Execute the following command:
./apic-mustgather
- OpenShift, IBM Cloud Pak for Integration deployment or Native Kubernetes deployment:
apic-mustgather
tool was downloaded and add execute permission:chmod +x apic-mustgather
Change to the directory where the - Execute the following command:
./apic-mustgather
- OVA deployment:
- Upload the following to the case:
- /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
- Steps to reproduce the problem
- Time that the error occurred or start/stop time of reproducing the error
- Screen capture of error (if applicable)
Analytics subsystem
- Download the apic-mustgather tool with the following command:
curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
- Reproduce the problem
- Run the
apic-mustgather
tool, and note the location where the apic-mustgather output file is saved:- OVA deployment:
apic-mustgather
tool was downloaded and change to root user:sudo -i
Via SSH, connect to the target appliance to which the - Change to the directory containing the
apic-mustgather
tool and add execute permission:chmod +x apic-mustgather
- Execute the following command:
./apic-mustgather
- OpenShift, IBM Cloud Pak for Integration deployment or Native Kubernetes deployment:
apic-mustgather
tool was downloaded and add execute permission:chmod +x apic-mustgather
Change to the directory where the - Execute the following command:
./apic-mustgather
- OVA deployment:
- Upload the following to the case:
- /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
- Steps to reproduce the problem
- Time that the error occurred or start/stop time of reproducing the error
- Screen capture of error (if applicable)
Gateway subsystem: OpenShift, IBM Cloud Pak for Integration, or Kubernetes deployment
-
apic-mustgather tool with the following command:Download the
curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
- Download the latest apicops command-line interface
- Ensure that you follow the requirements section so that the tool will work correctly in your environment
- Run the
apic-mustgather
andapicops
tools, and note the location where the apic-mustgather output file is saved:- Change to the directory where the
apic-mustgather
tool was downloaded and add execute permission:chmod +x apic-mustgather
- Execute the following command:
./apic-mustgather
- Execute the following command:
NOTE: If the command returns an error, review the steps documented in the requirements section./apicops debug:info
- Change to the directory where the
- Upload the following to the case:
- /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
- Output from apicops command in step 3.3
Gateway subsystem: VMware deployment or physical appliance
- Via SSH, connect to the DataPower server
- Collect API Connect gateway service log data by configuring the following log target in the API Connect application domain using the CLI.
- Repeat this step for each gateway server in the cluster
sw <apiconnect domain> configure terminal logging target gwd-log type file format text timestamp zulu size 50000 local-file logtemp:///gwd-log.log event apic-gw-service debug exit apic-gw-service;admin-state disabled;exit apic-gw-service;admin-state enabled;exit write mem exit
- Repeat this step for each gateway server in the cluster
- OPTIONAL: Enable gateway-peering debug logs via the DataPower CLI.
- Repeat this step for each gateway server in the cluster and replace GW_PEERING_OBJECT_NAME with the correct name of the peering object:
NOTE: To determine the configured peering objects, issue the following command within the apiconnect domain: show gateway-peering-statussw <apiconnect domain> diagnostics gateway-peering-debug GW_PEERING_OBJECT_NAME exit
- Repeat this step for each gateway server in the cluster and replace GW_PEERING_OBJECT_NAME with the correct name of the peering object:
- On the management subsystem, download and install:
- The apic-mustgather tool with the following command:
curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
- The latest apicops command-line interface
- Ensure that you follow the requirements section so that the tool will work correctly in your environment
- The apic-mustgather tool with the following command:
- Reproduce the problem.
- On the management subsystem, run the
apic-mustgather
andapicops
tools, and note the location where the apic-mustgather output file is saved:apic-mustgather
tool was downloaded and change to root user:sudo -i
Via SSH, connect to the target appliance to which the - Change to the directory containing the
apic-mustgather
tool and add execute permission:chmod +x apic-mustgather
- Execute the following command:
./apic-mustgather
- Execute the following command:
NOTE: If the command returns an error, review the steps documented in the requirements section./apicops debug:info
- Generate an error-report via the DataPower CLI.
- Repeat this step for each gateway server in the cluster
sw default conf; save error-report
- Repeat this step for each gateway server in the cluster
- OPTIONAL (only perform this step if you performed step 3): Dump the gateway-peering debug logs via the DataPower CLI and then disable gateway-peering debug.
- Repeat this step for each gateway server in the cluster and replace GW_PEERING_OBJECT_NAME with the correct name of the peering object:
NOTE: To determine the configured peering objects, issue the following command within the apiconnect domain: show gateway-peering-statussw <apiconnect domain> diagnostics gateway-peering-dump GW_PEERING_OBJECT_NAME no gateway-peering-debug GW_PEERING_OBJECT_NAME exit
- Repeat this step for each gateway server in the cluster and replace GW_PEERING_OBJECT_NAME with the correct name of the peering object:
- Upload the following to the case:
- /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
- Output from apicops command in step 6.4
- For each gateway server in the cluster:
- The gateway service log written to logtemp://gwd-log.log in the apiconnect domain
- <error report filename>.txt.gz (error report)
- gateway-peering logs (gatewaypeering.log and gatewaypeeringmonitor.log) in temporary:///<name of gateway peering object in API Connect application domain>
- Output of the following command issued from DataPower command-line interface: `show gateway-peering-status`
- Time that the error occurred or start/stop time of reproducing the error
- For a specific API that is failing - Optional and in addition to steps 9.1 - 9.3
- For the API Gateway:
- Files under temporary://
- Related yaml files
- DataPower configuration for application domain
- Probe of the failing transaction, see Configuring the API probe
- For a v5-compatible gateway:
- Related yaml files
- Probe of the failing transaction
- Export of the document cache for webapi and webapi-internal
- For the API Gateway:
How to submit diagnostic data to IBM Support |
---|
After you have collected the preceding information, and the case is opened, please see: For more details see submit diagnostic data to IBM (ECuRep) and Enhanced Customer Data Repository (ECuRep) secure upload |
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSMNED","label":"IBM API Connect"},"ARM Category":[{"code":"a8m50000000L0rvAAC","label":"API Connect"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"10.0.1;10.0.5;10.0.6;10.0.7;and future releases"}]
Was this topic helpful?
Document Information
Modified date:
05 June 2024
UID
ibm16242030