IBM Support

MustGather: SPNEGO problems on WebSphere Liberty

Troubleshooting


Problem

This document describes the process for collecting data for problems with the IBM WebSphere® Application Server Liberty SPNEGO component. Gathering this MustGather information before you call IBM support can help you understand the problem and save time analyzing the data.

Resolving The Problem

 

Runtime:



This document is for collecting data for LIBERTY. If you want to collect data for WebSphere traditional, see MustGather: SPNEGO problems on WebSphere traditional or click on the WebSphere traditional tab above.

  • Read first and related MustGathers

    MustGather: Read first for WebSphere Application Server and Liberty

    EJB container problem Servlet engine and Web container problem Security problem

    For a listing of all technotes, downloads, and educational materials specific to the Security component, search the WebSphere Application Server support portal.

  • Exchange data with IBM Support

    To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files by using one of following methods to help speed problem diagnosis:

    • Service Request (SR)
    • Email
    • FTP to the Enhanced Customer Data Repository (ECuRep)

    Instructions for exchanging information with IBM Support

  • SPNEGO on Liberty trace specifications

    • Add the following string to the <logging> element in server.xml:
      com.ibm.ws.security.*=all:com.ibm.wsspi.security.*=all:com.ibm.ws.webcontainer.security.*=all
    • Insert the following generic JVM arguments in the jvm.options file:
      If using IBM JDK: 
      -Dcom.ibm.security.jgss.debug=all
      -Dcom.ibm.security.krb5.Krb5Debug=all
      If using openJDK:
      -Dsun.security.krb5.debug=true
      -Dsun.security.jgss.debug=true
      -Dsun.security.spnego.debug=true
      Avoid Trouble: There is one entry per line in this file. Make sure you do not have any extra white space in your jvm.options file.
  • Diagnostic questions

    Provide answers to the following diagnostic questions:

    1. Describe your system environment
      1. Liberty server version :
      2. Active Directory version :
      3. Client OS version :
    2. Provide the complete Java™ version used by Liberty:
      • For example, unless the Java version that Liberty uses is specified under the server.env cofiguration file, you can choose to run the following command:
        For Windows platforms,
        java -version
        For Unix platforms,
        ./java -version
    3. What is the full web request URL accessed by the client browser?
      • Is the application that you are trying to access is protected uri?
    4. Is the requested hostname a DNS alias (CNAME Record) or real hostname (A Record)?
    5. What Active Directory user ID is used to map to the wanted SPN's?
      • Provide a screen capture of the Active Directory User properties Account Tab for this user.
    6. Provide the commands issued to create the keytab and SPN-mappings on the AD server.
      • If possible, also provide the command output.
    7. Find all SPN-mapping occurrences mapped to AD user names:
      • On the Active Directory Server, run following command:
        C:\ldifde -f output.txt -r "(servicePrincipalName=HTTP/hostname.domain.com)"
        (Where hostname.domain.com is the same fully qualified hostname used in the web request by the client)
    8. Are there any collective, members, load balancers, firewalls, proxies, or web servers in the mix, or any devices/appliances between the client browser and Liberty?
      • If yes, provide basic login flow details with relevant topology involved.
    9. Is this a single Active Directory domain or do you have trusted domains/forests?
      • If yes, elaborate.
    10. Are you using Constrained Delegation for Outbound SPNEGO Token?
      • If yes, elaborate.
    11. What is the client (Java client or browser)? if it is browser then need screen capture of client browser SPNEGO settings.
    12. Also provide your krb5 config and keytab files.
  • Collect data for Liberty (step by step)

    This section is for collecting data for LIBERTY. If you want to collect data for WebSphere traditional click here or see the WebSphere traditional tab above.

    Before you collect data, be sure to answer the Diagnostic questions in the section above.

    You can choose to follow this step-by-step document or you can watch the video in the Collect data for Liberty (Video) section below.

    SPNEGO issues on Liberty might be difficult to troubleshoot. Make sure to collect all the information below.
    When all the information for your issue is ready, follow the instructions on Exchanging information with IBM Technical Support for problem determination to send the information and files that you collected.
     

    SET UP LIBERTY FOR SPNEGO TRACING

    1. Set up the JVM for SPNEGO tracing
      1. Locate your jvm.options file:
        • The jvm.options files can be found under the following path:
          <LIBERTY_HOME>/usr/servers/<server name>/jvm.options
        • If the jvm.options file does not exist, create it with a text editor.
      2. Insert the following generic JVM arguments to the jvm.options file:
        If using IBM JDK: 
        -Dcom.ibm.security.jgss.debug=all
        -Dcom.ibm.security.krb5.Krb5Debug=all
        If using openJDK:
        -Dsun.security.krb5.debug=true
        -Dsun.security.jgss.debug=true
        -Dsun.security.spnego.debug=true
        Avoid Trouble: There is one entry per line in this file. Make sure you do not have any extra white space in your jvm.options file.
      3. Save the changes to your jvm.options file.
        • Your changes are not be picked up by the JVM until the server is restarted.
    2. Set up the Liberty server for SPNEGO tracing
      • Follow the instructions in the Enabling Trace on Liberty section in Set up trace and get a full dump for WebSphere Liberty.
      • Use the following trace string:
        com.ibm.ws.security.*=all:com.ibm.ws.webcontainer.security.*=all:com.ibm.websphere.wim.*=all:com.ibm.wsspi.wim.*=all:com.ibm.ws.wim.*=all
      • Additional information can be found in the Liberty:Logging and Trace topic in the IBM Documentation.
    3. Verify that your tracing is working as intended
      1. Stop the Liberty Server
      2. Delete any existing logs files found under the logs directory:
        <LIBERTY_HOME>/usr/servers/<serverName>/logs
      3. Restart the Liberty Server and review the logs to confirm that they are recent.
      4. Verify that the new Liberty trace setting has been picked up by reviewing the upper part of the trace.log file.
     

    COLLECT LIBERTY SPNEGO TRACES


    Avoid trouble: It is important that SPNEGO traces be gathered from Liberty server startup.
     
    1. Stop the Liberty server
    2. Restart the Liberty server
    3. Reproduce the problem, making note of the following information:
      Time when the problem occurs
      The client user ID which logged into the Microsoft Domain
      The Microsoft Domain name itself
      The exact URL being invoked.
     

    GATHER LIBERTY DATA TO SEND TO IBM SUPPORT


    Use the "dump" command to generate a .zip file containing the logs and config files which can be sent to support.
    For Windows platforms, run:
    <LIBERTY_HOME>\bin\server.bat dump <serverName>

    For UNIX platforms, run:
    <LIBERTY_HOME>/bin/server dump <serverName>

    Collect the resulting dump .zip file with date & time. These files can be found under the following path:
    <LIBERTY_HOME>/usr/servers/<serverName>

    File name example:
    (myserver.dump-17.03.20_22.20.57.zip)

     
  • Collect data for Liberty (Video)

    This section is for collecting data for LIBERTY. If you want to collect data for WebSphere traditional click here or see the WebSphere traditional tab above.



    Before you collect data, be sure to answer the Diagnostic questions in the section above.

    You can choose to watch this video or follow the step-by-step instructions in the Collect data for Liberty (step by step) section above.

    SPNEGO issues on Liberty might be difficult to troubleshoot. Make sure to collect all the information described in the video. When all the information for your issue is ready, follow the instructions on Exchanging information with IBM Technical Support for problem determination to send the information and files that you collected.

     

    The following video goes over the necessary steps to collect data for a SPNEGO problem on Liberty.


Note:

This document uses the term WebSphere traditional to refer to WebSphere Application Server v9.0 traditional, WebSphere Application Server v8.5 full profile, WebSphere Application Server v8.0 and earlier, WebSphere classic, traditional WebSphere, traditional WAS, and tWAS.
 

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Security","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF012","label":"IBM i"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"},{"code":"PF035","label":"z\/OS"},{"code":"PF022","label":"OS X"}],"Version":"Not Applicable","Edition":"Liberty","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"}}]

Document Information

Modified date:
05 February 2024

UID

swg22002649

Page Feedback