IBM Support

MustGather: API Connect v10 (all subsystems)

Troubleshooting


Problem

ATTENTION: This MustGather has been updated to utilize the new python based apic-mustgather tool. This new tool provides significant improvements:
  • Reduction in the amount of time it takes the tool to run to completion
  • Simplification to no longer require passing multiple parameters. By default, it will automatically collect:
    • data from all namespaces relevant to API Connect
    • diagnostic data for each subsystem present in the cluster
    • data specific to OVA form factor

Resolving The Problem

Each section in this document contains the instructions to collect the MustGather data for IBM API Connect on the associated subsystem. This data is required by IBM Support to effectively diagnose and resolve issues.
 
 
 
 
Installation or Upgrade Issue (all subsystems): VMware deployment
  1. Download the apic-mustgather tool with the following command: 
    curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
  2. Run the apic-mustgather tool, and note the location where the output file is saved:
    1. Via SSH, connect to the target appliance to which the apic-mustgather tool was downloaded and change to root user: 
      sudo -i
    2. Change to the directory where the apic-mustgather tool was downloaded to the appliance and add execute permission:  
      chmod +x apic-mustgather
    3. Execute the following command: 
      ./apic-mustgather
  3. As root user, gather the status of apic: 
    apic status > apic_status.out
  4. As root user, gather the version of apic: 
    apic version > apic_version.out
  5. Upload the following to the case:
    1. Any error messages received from running the  apicup subsys install command
    2. /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
    3. apic_status.out
    4. apic_version.out
    5. Archive file of the apicup project directory
 

Back to top



 
Installation or Upgrade Issue (all subsystems): OpenShift, IBM Cloud Pak for Integration, or Kubernetes deployment
  1. Download the apic-mustgather tool with the following command: 
    curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
  2. Run the apic-mustgather tool, and note the location where the output file is saved:
    1. Change to the directory where the apic-mustgather tool was downloaded and add execute permission:  
      chmod +x apic-mustgather
    2. Execute the following command: 
      ./apic-mustgather
  3. Upload the following to the case:
    1. /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
 

Back to top



 
Management subsystem
  1. Download the apic-mustgather tool with the following command: 
    curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
  2. Reproduce the problem
  3. Run the apic-mustgather tool, and note the location where the output file is saved:
    • OVA deployment:
      1. Via SSH, connect to the target appliance to which the apic-mustgather tool was downloaded and change to root user: 
        sudo -i
      2. Change to the directory containing the apic-mustgather tool and add execute permission:  
        chmod +x apic-mustgather
      3. Execute the following command: 
        ./apic-mustgather
    • OpenShift, IBM Cloud Pak for Integration deployment or Native Kubernetes deployment:
      1. Change to the directory where the apic-mustgather tool was downloaded and add execute permission:  
        chmod +x apic-mustgather
      2. Execute the following command: 
        ./apic-mustgather
  4. Upload the following to the case:
    1. /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
    2. Steps to reproduce the problem
    3. Time that the error occurred or start/stop time of reproducing the error
    4. Screen capture of error (if applicable)
 

Back to top



 
Portal subsystem
  1. Download the apic-mustgather tool with the following command: 
    curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
  2. Reproduce the problem
  3. Run the apic-mustgather tool, and note the location where the apic-mustgather output file is saved:
    • OVA deployment:
      1. Via SSH, connect to the target appliance to which the apic-mustgather tool was downloaded and change to root user: 
        sudo -i
      2. Change to the directory containing the apic-mustgather tool and add execute permission:  
        chmod +x apic-mustgather
      3. Execute the following command: 
        ./apic-mustgather
    • OpenShift, IBM Cloud Pak for Integration deployment or Native Kubernetes deployment:
      1. Change to the directory where the apic-mustgather tool was downloaded and add execute permission:  
        chmod +x apic-mustgather
      2. Execute the following command: 
        ./apic-mustgather
  4. Upload the following to the case:
    1. /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
    2. Steps to reproduce the problem
    3. Time that the error occurred or start/stop time of reproducing the error
    4. Screen capture of error (if applicable)
 

Back to top



 
Analytics subsystem
  1. Download the apic-mustgather tool with the following command: 
    curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
  2. Reproduce the problem
  3. Run the apic-mustgather tool, and note the location where the apic-mustgather output file is saved:
    • OVA deployment:
      1. Via SSH, connect to the target appliance to which the apic-mustgather tool was downloaded and change to root user: 
        sudo -i
      2. Change to the directory containing the apic-mustgather tool and add execute permission:  
        chmod +x apic-mustgather
      3. Execute the following command: 
        ./apic-mustgather
    • OpenShift, IBM Cloud Pak for Integration deployment or Native Kubernetes deployment:
      1. Change to the directory where the apic-mustgather tool was downloaded and add execute permission:  
        chmod +x apic-mustgather
      2. Execute the following command: 
        ./apic-mustgather
  4. Upload the following to the case:
    1. /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
    2. Steps to reproduce the problem
    3. Time that the error occurred or start/stop time of reproducing the error
    4. Screen capture of error (if applicable)
 

Back to top



 
Gateway subsystem: OpenShift, IBM Cloud Pak for Integration, or Kubernetes deployment
  1. Download the apic-mustgather tool with the following command: 
    curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
  2. Download the latest apicops command-line interface
    • Ensure that you follow the requirements section so that the tool will work correctly in your environment
  3. Run the apic-mustgather and apicops tools, and note the location where the apic-mustgather output file is saved:
    1. Change to the directory where the apic-mustgather tool was downloaded and add execute permission:  
      chmod +x apic-mustgather
    2. Execute the following command: 
      ./apic-mustgather
    3. Execute the following command:
      NOTE: If the command returns an error, review the steps documented in the requirements section 
      ./apicops debug:info
  4. Upload the following to the case:
    1. /tmp/apic-mustgather-*.tar.gz file that was generated from the apic-mustgather command
    2. Output from apicops command in step 3.3
   

Back to top



 
Gateway subsystem: VMware deployment or physical appliance
  1. Via SSH, connect to the DataPower server
  2. Collect API Connect gateway service log data by configuring the following log target in the API Connect application domain using the CLI.
    • Repeat this step for each gateway server in the cluster 
      sw <apiconnect domain>
configure terminal
logging target gwd-log
type file
format text
timestamp zulu
size 50000
local-file logtemp:///gwd-log.log
event apic-gw-service debug
exit
apic-gw-service;admin-state disabled;exit
apic-gw-service;admin-state enabled;exit
write mem
exit
  3. OPTIONAL: Enable gateway-peering debug logs via the DataPower CLI.
    • Repeat this step for each gateway server in the cluster and replace GW_PEERING_OBJECT_NAME with the correct name of the peering object:
      NOTE: To determine the configured peering objects, issue the following command within the apiconnect domain: show gateway-peering-status 
      sw <apiconnect domain>
diagnostics
gateway-peering-debug GW_PEERING_OBJECT_NAME
exit
  4. On the management subsystem, download and install:
    • The apic-mustgather tool with the following command: 
      curl -L -O https://github.com/ibm-apiconnect/v10-postmortem/releases/latest/download/apic-mustgather
    • The latest apicops command-line interface
      • Ensure that you follow the requirements section so that the tool will work correctly in your environment
  5. Reproduce the problem.
  6. On the management subsystem, run the apic-mustgather and apicops tools, and note the location where the apic-mustgather output file is saved:
    1. Via SSH, connect to the target appliance to which the apic-mustgather tool was downloaded and change to root user: 
      sudo -i
    2. Change to the directory containing the apic-mustgather tool and add execute permission:  
      chmod +x apic-mustgather
    3. Execute the following command: 
      ./apic-mustgather
    4. Execute the following command:
      NOTE: If the command returns an error, review the steps documented in the requirements section 
      ./apicops debug:info
  7. Generate an error-report via the DataPower CLI.
    • Repeat this step for each gateway server in the cluster 
      sw default
conf; save error-report
  8. OPTIONAL (only perform this step if you performed step 3): Dump the gateway-peering debug logs via the DataPower CLI and then disable gateway-peering debug.
    • Repeat this step for each gateway server in the cluster and replace GW_PEERING_OBJECT_NAME with the correct name of the peering object:
      NOTE: To determine the configured peering objects, issue the following command within the apiconnect domain: show gateway-peering-status 
      sw <apiconnect domain>
diagnostics
gateway-peering-dump GW_PEERING_OBJECT_NAME
no gateway-peering-debug GW_PEERING_OBJECT_NAME
exit
  9. Upload the following to the case:
    1. /tmp/apic-mustgather-*.tar.gz  file that was generated from the apic-mustgather command
    2. Output from apicops command in step 6.4
    3. For each gateway server in the cluster:
      • The gateway service log written to logtemp://gwd-log.log in the apiconnect domain
      • <error report filename>.txt.gz (error report) 
      • gateway-peering logs (gatewaypeering.log and gatewaypeeringmonitor.log) in temporary:///<name of gateway peering object in API Connect application domain>
      • Output of the following command issued from DataPower command-line interface: `show gateway-peering-status`
    4. Time that the error occurred or start/stop time of reproducing the error
    5. For a specific API that is failing - Optional and in addition to steps 9.1 - 9.3
      • For the API Gateway: 
        • Files under temporary://
        • Related yaml files
        • DataPower configuration for application domain
        • Probe of the failing transaction, see Configuring the API probe
      • For a v5-compatible gateway: 
        • Related yaml files
        • Probe of the failing transaction 
        • Export of the document cache for webapi and webapi-internal
  How to submit diagnostic data to IBM Support 

  After you have collected the preceding information, and the case is opened, please see:  
  Exchanging information with IBM Technical Support.

  For more details see submit diagnostic data to IBM (ECuRep) and Enhanced Customer Data Repository (ECuRep) secure upload

Back to top

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSMNED","label":"IBM API Connect"},"ARM Category":[{"code":"a8m50000000L0rvAAC","label":"API Connect"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"10.0.1;10.0.5;10.0.6;10.0.7;and future releases"}]

Document Information

Modified date:
05 June 2024

UID

ibm16242030

Page Feedback