MustGather: Request failures accessing WebSphere Application Server dynamic content through IBM HTTP Server

The mustgather below is primarily for problems related to serving enterprise application content through IBM HTTP Server with the WAS WebServer Plug-in.

Higher level info and other scenarios are covered here: MustGather: Read first for IBM HTTP Server.
 

1. Apply configuration changes for basic webserver logging/tracing:
   1. Edit the plugin-cfg.xml file and change the value of LogLevel to Trace. 
      • Note the path to http_plugin.log (such as /opt/IBM/HTTPServer_plugins/logs/webserver1/http_plugin.log)
      • If heavy load is needed to reproduce the issue,  LogLevel can instead be set to Debug.
   2. Append "LogLevel debug" to httpd.conf
      • In IHS 9.0 or later, append "LogLevel trace8" instead for additional detail
   3. Enable extended Access logs in IHS (be sure to collect the resulting log and include it in your upload)
      • 9.0 and later and 8.5.5.19 and later: Append the following line to httpd.conf 

        GlobalLog logs/was-debug.log "%h %l %u %t \"%r\" %>s %b %{WAS_UNIQUE_ID}e %{RH}e WAS=%{WAS}e %{Set-Cookie}o %{JSESSIONID}C %{remote}p %D"

      • 8.5.5 and earlier (or non-IHS Apache): Append the following line once to httpd.conf and append to any VirtualHost involved in the request that already has any CustomLog directive

        CustomLog  logs/was-debug.log "%h %l %u %t \"%r\" %>s %b %{WAS_UNIQUE_ID}e %{RH}e WAS=%{WAS}e %{Set-Cookie}o %{JSESSIONID}C %{remote}p %D"

2. Optional: If directed by support, enable mod_net_trace to record the HTTP data passing through the system. Note: n/a for non-IHS Apache
   1. Uncomment "#LoadModule net_trace_module ..." entry in httpd.conf
   2. Confirm /tmp has enough space to record all of the HTTP input and output or modify NetTraceFile in httpd.conf
   3. Full details:  http://publib.boulder.ibm.com/httpserv/manual24/mod/mod_net_trace.html
3. Optional: If directed by support, collect detailed timing information for requests by adding the following configuration:
   Note: n/a for non-IHS Apache
   • IHS 9.0.0.0 and later and 8.5.5.19 and later: Append to httpd.conf:

     TrackHooks allhooks
     TrackHooksOptions millis permodule
     GlobalLog logs/access-detailed-timing.log "%h %l %u %t \"%r\" %>s %b %{RH}e TRH=%{TRH}e TCA=%{TCA}e TCU=%{TCU}e TPR=%{TPR}e TAC=%{TAC}e TTN=%{TTN}e TPCN=%{TPCN}e TPSB=%{TPSB}e TPWR=%{TPWR}e TPRB=%{TPRB}e %D RH=%{RH}e %{WAS}e %{JSESSIONID}C %{remote}p HS=%{SSL_HANDSHAKE_TIME}e %D"
     

   • IHS 8.5.5 or earlier: 
     Step 1: Append to httpd.conf globally:

     TrackHooks allhooks
     TrackHooksOptions millis permodule
     CustomLog logs/access-detailed-timing.log "%h %l %u %t \"%r\" %>s %b %{RH}e TRH=%{TRH}e TCA=%{TCA}e TCU=%{TCU}e TPR=%{TPR}e TAC=%{TAC}e TTN=%{TTN}e TPCN=%{TPCN}e TPSB=%{TPSB}e TPWR=%{TPWR}e TPRB=%{TPRB}e %D RH=%{RH}e %{WAS}e %{JSESSIONID}C %{remote}p HS=%{SSL_HANDSHAKE_TIME}e %D"
     

     Step 2: Append to each virtual host with existing logging (CustomLog) directives:

     # Also add to each VirtualHost that already has separate "CustomLog"
     CustomLog logs/access-detailed-timing.log "%h %l %u %t \"%r\" %>s %b %{RH}e TRH=%{TRH}e TCA=%{TCA}e TCU=%{TCU}e TPR=%{TPR}e TAC=%{TAC}e TTN=%{TTN}e TPCN=%{TPCN}e TPSB=%{TPSB}e TPWR=%{TPWR}e TPRB=%{TPRB}e %D RH=%{RH}e %{WAS}e %{JSESSIONID}C %{remote}p HS=%{SSL_HANDSHAKE_TIME}e %D"

4. Optional: If there is an SSL/TLS symptom, enable SSL/TLS traces from the following Mustgather: MustGather: IBM HTTP Server SSL handshake and configuration problems
5. Optional: If directed by support, to diagnose an issue communicating with the backend application server, enable application server traces:
   • WebSphere Application Server traditional: 
     • Follow the directions here to configure WebSphere tracing the number of trace entries. Use the following trace specification: 
       *=info:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:TCPChannel=all:GenericBNF=all
   • WebSphere Liberty: Liberty Mustgathers
     <logging maxfiles="10" maxfilesize="100" tracefilename="trace.log" traceformat="BASIC" tracespecification="com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:GenericBNF=all:HTTPDispatcher=all:TCPChannel=all "/>
6. Optional: If directed by support, to diagnose connection or request timeouts on Linux, follow the following document to capture details on TCP retransmissions:  https://www.ibm.com/support/pages/node/6480315
7. Stop IBM HTTP Server (IHS) and clear or truncate current logs (/tmp/nettrace, /tmp/gsktrace*, error/access/http_plugin logs) if possible.
8. Start (or restart) IHS to pick up the diagnostic settings and recreate your problem
   • It is critical to recreate the problem while the additional logging is configured, keeping the amount of extra traffic as low as possible
   • Communicate to support the URL, IP address, or any other distinguishing characteristics of the failure
   • If requested, recreate with a web browser trace: https://www.ibm.com/support/pages/troubleshoot-enabling-browser-trace
9. Collect Logs and Configuration:
   1. Add any additional files you want to share about the problem to a directory such as /tmp/toibm (or c:/toibm)
      • For example, traces collected from client software or WebSphere or optional packet captures, key stores, or GSKit traces not written to the IHS logs/directory.
   2. Create a zip or tar-file of the following files and directories, adjusting for your installation path and platform, and adding any alternate configuration or log directories or files from the preceding step
      • $IHS_HOME/conf $IHS_HOME/logs $IHS_HOME/version.signature
        • If mod_net_trace was used above, /tmp/nettrace
        • This should include any IHS config and logs.  If you are aware of others, include them.
      • /path/to/http_plugin.log  /path/to/plugin-cfg.xml
      • If collected, packet captures or GSKit traces (gsktrace_log*)
      • If collected, /tmp/tcpretransmits.log
      • IHS SSL key stores (KeyFile directive) and respective stash files: key.kdb, key.sth
      • If collected, WebSphere Logs and traces: 
        • WebSphere traditional: SystemOut.log, SystemErr.log, trace.log*
        • WebSphere Liberty: messages.log, console.log, trace.log*
10. Make a note of the time, date, IHS version, and failing fully qualified URL and supply it alongside the documentation
    • The IHS version is reported by  $IHS_HOME/bin/apachectl -V
      • Windows:  $IHS_HOME/bin/apache.exe -V
11. Follow instructions to send diagnostic information to IBM support.
12. Revert any changes (httpd.conf, plugin-cfg.xml, WebSphere Application Server configuration)
13. Restart affected services

To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:

• Service Request (SR)
• E-mail
• FTP to the Enhanced Customer Data Repository (ECuRep)

Instructions for exchanging information with IBM Support

MustGather: Read first for WebSphere Application Server

MustGather: WebSphere Application Server HTTP plug-in problems MustGather: Web server configuration problems with IBM HTTP Server MustGather: IBM HTTP Server SSL handshake and configuration problems