Diagnosing The Problem

• Deprecated Instructions: Do Not Use
  • Deployed via OVA or physical appliance
  • Deployed via Kubernetes

  For an API Connect gateway service deployed via OVA or physical appliance

  1. Collect API Connect gateway service log data by configuring the following log target in the API Connect application domain using the CLI.
      

     configure terminal logging target gwd-log type file format text timestamp zulu size 50000 local-file logtemp:///gwd-log.log event apic-gw-service debug exit exit

  2. Enable and collect gateway-peering logs.
      

     diagnostics gateway-peering-debug <name of gateway peering object in API Connect application domain> gateway-peering-dump <name of gateway peering object in API Connect application domain> exit

  
  
  3. Open the API Manager deployment for edit to enable API Manager debug.
      

     kubectl get deployments --all-namespaces kubectl edit deploy <idOfAPIMDeployment> (such as r908c0d2347-apim-v2)

  
  
  4. Enter the following name value pair under containers -env.
      

     name: DEBUG value: audit,apim:routes:*,apim:routesc:*,apim:server,bhendi:probe,bhendi:flags,apim:webhookPayload

  
  
  5. Save file and Kubernetes should auto redeploy.
      
  6. Reproduce problem.
      
  7. Generate an error-report via the DataPower CLI.
      

     sw <apiconnect domain> conf; save error-report

  
  
  8. Collect API Connect logs.
      

     apic logs

  
  
  9. Submit all logs for API Connect gateway service deployed via OVA or physical appliance.
     1. node-logs.zip (from `apic logs` command)
     2. appliance-logs.zip (from `apic logs` command)
     3. The gateway service log written to logtemp://gwd-log.log in the apiconnect domain
     4. <error report filename>.txt.gz (error report) 
     5. gateway-peering logs (gatewaypeering.log and gatewaypeeringmonitor.log). in temporary:///<name of gateway peering object in API Connect application domain>
     6. output of the following command issued from DataPower command line interface: `show gateway-peering-status`
     7. All syslogs from var/log directory on each management subsystem virtual machine (syslog, syslog.1, syslog.2.gz, etc)
     8. For a specific API that is failing - Optional and in addition to steps 9.1 - 9.7 above. 
        • For the API Gateway: 
          • files under local://
          • related yaml files
          • DataPower configuration for application domain
        • For a v5-compatible gateway: 
          • related yaml files
          • Probe of the failing transaction 
          • Export of the document cache for webapi and webapi-internal

  Back to top

  For an API Connect gateway service deployed via Kubernetes

  1. Open the API Manager deployment for edit to enable API Manager debug.
      

     kubectl get deployments --all-namespaces kubectl edit deploy <idOfAPIMDeployment> (such as r908c0d2347-apim)

  
  
  2. Enter the following name value pair under containers -env.
      

     name: DEBUG value: audit,apim:routes:*,apim:routesc:*,apim:server,bhendi:probe,bhendi:flags,apim:webhookPayload

  
  
  3. Save file and Kubernetes should auto redeploy.
  
  
  4. Reproduce problem.
  
  
  5. Copy API Connect gateway service logs.

     kubectl cp -n <namespace> "<gateway pod name>:drouter/temporary/log/apiconnect/gwd-log.log" gwd-log.<gateway pod name>.log

  
  
  6. Attach to the gateway pod.
      

     kubectl attach <gateway pod name>-c dynamic-gateway-service --stdin=true --tty=true -n <namespace>

  
  
  7. Login with username and password of admin
  
  
  8. At the idg# command prompt
     • Copy the output of the following command.
        

       sw apiconnect show gateway-peering-status

     
     
     • Generate an error report.
        

       conf;save error-report

  
  
  9. Copy error report
      

     kubectl cp -n <namespace> "<gateway pod name>:drouter/temporary/<error report filename>.txt.gz" <error report filename>.txt.gz

  
  
  10. For multiple gateway service pods, repeat steps 5 - 9 for each gateway service pod. 
  11. Collect log files for all pods in a namespace with the attached script ( get_pod_logs.zip ) and upload resulting apic-logs.tgz. It will need slight modification prior to execution. Alternatively, a script can be created from the content found below.

      ---

      # $APIC_NAMESPACE refers to the k8s namespace where API Connect components were deployed. # $LOG_DIR is a destination directory for the log files, any empty dir should do APIC_NAMESPACE=default LOG_DIR=/home/ibmadmin/apic2018/logs for __pod in $(kubectl get pods -n $APIC_NAMESPACE --show-all -o name | cut -d'/' -f2); do for __container in $(kubectl get pod -n $APIC_NAMESPACE $__pod -o jsonpath="{.spec.containers[*].name}"); do kubectl logs -n $APIC_NAMESPACE $__pod -c $__container &> $LOG_DIR/${__pod}_${__container}.log kubectl logs --previous -n $APIC_NAMESPACE $__pod -c $__container &> $LOG_DIR/${__pod}_${__container}__previous.log [ $? -eq 0 ] || rm -f $LOG_DIR/${__pod}_${__container}__previous.log done done tar -C $LOG_DIR -cz -f apic-logs.tgz .

      ---

  
  

  12. Submit all logs for API Connect gateway service deployed to Kubernetes:

      1. apic-logs.tgz (pod logs)
      2. The API Connect gateway service log written to logtemp://gwd-log.log in the apiconnect domain
      3. <error report filename>.txt.gz (error report) 
      4. gateway-peering logs (gatewaypeering.log and gatewaypeeringmonitor.log) found in temporary:///<name of gateway peering object in API Connect application domain>
      5. Kubectl describe nodes output.
          

         kubectl describe nodes

      
      
      6. output of from show gateway-peering-status command
      7. For a specific API that is failing - Optional and in addition to steps 12.1 - 12.6 above. 
         • For the API Gateway: 
           • files under local://
           • related yaml files
           • DataPower configuration for application domain
         • For a v5-compatible gateway: 
           • related yaml files
           • Probe of the failing transaction 
           • Export of the document cache for webapi and webapi-internal

  Back to top