Linux problem isolation procedure

Use this procedure when servicing a Linux® partition or a server that has Linux as its only operating system.

1. Is the server managed by an HMC that is running Service Focal Point (SFP)?

   No

   Go to step 3.

   Yes

   Go to step 2.

2. Servers with Service Focal Point

   Look at the service action event log in SFP for errors. Focus on those errors with a timestamp near the time at which the error occurred. Follow the steps indicated in the error log entry to resolve the problem. If the problem is not resolved, continue with step 3.

3. Look for and record all reference code information or software messages on the operator panel and in the service processor error log (which is accessible by viewing the ASMI menus).
4. Choose a Linux partition that is running correctly (preferably the partition with the problem).

   Is Linux usable in any partition with Linux installed?

   No

   Go to step 11.

   Yes

   Go to step 5.

5. Do the following steps:
   Attention: Items 5 through 16 must be performed on a server or partition running the Linux operating system.
   1. Determine if there is a file named platform under /var/log directory of the server or partition. Log into the server or partition as the root user and enter the following command:

      ls -l /var/log/platform

      Does the /var/log/platform file exist?

      No

      Continue with substep 5.b.

      Yes

      Go to substep 5.c.

   2. Record that, for this Linux partition, you performed substep 5.b of 5 for later steps. Examine the Linux system log by entering the following command:

      cat /var/log/messages |grep RTAS |more

      Linux run-time RTAS error messages are logged in the messages file under /var/log. The following is an example of the Linux system RTAS error log messages.

      Aug 27 18:13:41 rasler kernel: RTAS: -------- event-scan begin -------- Aug 27 18:13:41 rasler kernel: RTAS: Location Code: U0.1-P1-C1 Aug 27 18:13:41 rasler kernel: RTAS: WARNING: (FULLY RECOVERED) type: INTERN_DEV_FAIL Aug 27 18:13:41 rasler kernel: RTAS: initiator: UNKNOWN target: UNKNOWN Aug 27 18:13:41 rasler kernel: RTAS: Status: predictive new Aug 27 18:13:41 rasler kernel: RTAS: Date/Time: 20020827 18134000 Aug 27 18:13:41 rasler kernel: RTAS: CPU Failure Aug 27 18:13:41 rasler kernel: RTAS: CPU id: 0 Aug 27 18:13:41 rasler kernel: RTAS: Failing element: 0x0000 Aug 27 18:13:41 rasler kernel: RTAS: A reboot of the system may correct the problem Aug 27 18:13:41 rasler kernel: RTAS: -------- event-scan end ----------

      Did you find any RTAS error log messages that are similar to the above messages?

      No

      Go to Using AIX® online and standalone diagnostics and run the eServer™ standalone diagnostics on the server or partition. If you receive a reference code go to the reference code list. If you cannot determine the problem using the diagnostic programs, contact your next level of support.

      Yes

      Go to step 6.

   3. Record that for this Linux partition, you performed substep 5.c of 5 for later steps.
      Use the following command to list diagela messages recorded in the Linux system log:

      cat /var/log/platform |grep diagela |more

      Linux run-time diagela error messages are logged in the platform file under /var/log.

      The following is an example of the Linux system error log diagela messages.

      Aug 13 09:38:45 larry diagela: 08/13/2003 09:38:44 Aug 13 09:38:45 larry diagela: Automatic Error Log Analysis has detected a problem. Aug 13 09:38:45 larry diagela: Aug 13 09:38:45 larry diagela: The Service Request Number(s)/Probable Cause(s) Aug 13 09:38:45 larry diagela: (causes are listed in descending order of probability): Aug 13 09:38:45 larry diagela: Aug 13 09:38:45 larry diagela: 651-880: The CEC or SPCN reported an error. Report the SRN and the following reference and physical location codes to your service provider. Aug 13 09:38:45 larry diagela: Location: n/a FRU: n/a Ref-Code: B1004699 Aug 13 09:38:45 larry diagela: Aug 13 09:38:45 larry diagela: Analysis of Error log sequence number: 3 Aug 29 07:13:04 larry diagela: 08/29/2003 07:13:04 Aug 29 07:13:04 larry diagela: Automatic Error Log Analysis has detected a problem. Aug 29 07:13:04 larry diagela: Aug 29 07:13:04 larry diagela: The Service Request Number(s)/Probable Cause(s) Aug 29 07:13:04 larry diagela: (causes are listed in descending order of probability): Aug 29 07:13:04 larry diagela: Aug 29 07:13:04 larry diagela: 651-880: The CEC or SPCN reported an error. Report the SRN and the following reference and physical location codes to your service provider. Aug 29 07:13:04 larry diagela: Location: U0.1-F4 FRU: 09P5866 Ref-Code: 10117661 Aug 29 07:13:04 larry diagela: Aug 29 07:13:04 larry diagela: Analysis of /var/log/platform sequence number: 24 Sep 4 06:00:55 larry diagela: 09/04/2003 06:00:55 Sep 4 06:00:55 larry diagela: Automatic Error Log Analysis reports the following: Sep 4 06:00:55 larry diagela: Sep 4 06:00:55 larry diagela: 651204 ANALYZING SYSTEM ERROR LOG Sep 4 06:00:55 larry diagela: A loss of redundancy on input power was detected. Sep 4 06:00:55 larry diagela: Sep 4 06:00:55 larry diagela: Check for the following: Sep 4 06:00:55 larry diagela: 1. Loose or disconnected power source connections. Sep 4 06:00:55 larry diagela: 2. Loss of the power source. Sep 4 06:00:55 larry diagela: 3. For multiple enclosure systems, loose or Sep 4 06:00:55 larry diagela: disconnected power and/or signal connections Sep 4 06:00:55 larry diagela: between enclosures. Sep 4 06:00:55 larry diagela: Sep 4 06:00:55 larry diagela: Supporting data: Sep 4 06:00:55 larry diagela: Ref. Code: 10111520 Sep 4 06:00:55 larry diagela: Location Codes: P1 P2 Sep 4 06:00:55 larry diagela: Sep 4 06:00:55 larry diagela: Analysis of /var/log/platform sequence number: 13

   4. Also use the following command to list RTAS messages recorded in the Linux system log:

      cat /var/log/platform |grep RTAS |more

      Linux RTAS error messages are logged in the platform file under /var/log. The following is an example of RTAS messages in the Linux system error log.

      Aug 27 12:16:33 larry kernel: RTAS: 15 -------- RTAS event begin -------- Aug 27 12:16:33 larry kernel: RTAS 0: 04440040 000003f8 96008508 19155800 Aug 27 12:16:33 larry kernel: RTAS 1: 20030827 00000001 20000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 2: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 3: 49424d00 55302e31 2d463400 00503034 Aug 27 12:16:33 larry kernel: RTAS 4: 10117661 04a0005d 10110000 00000000 Aug 27 12:16:33 larry kernel: RTAS 5: 00007701 000000e0 00000003 000000e3 Aug 27 12:16:33 larry kernel: RTAS 6: 00000000 01000000 00000000 31303131 Aug 27 12:16:33 larry kernel: RTAS 7: 37363631 20202020 20202020 55302e31 Aug 27 12:16:33 larry kernel: RTAS 8: 2d463420 20202020 20202020 03705a39 Aug 27 12:16:33 larry kernel: RTAS 9: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 10: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 11: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 12: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 13: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 14: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 15: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 16: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 17: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 18: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 19: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 20: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 21: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 22: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 23: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 24: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 25: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 26: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 27: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 28: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 29: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 30: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 31: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 32: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 33: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 34: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 35: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 36: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 37: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 38: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 39: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 40: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 41: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 42: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 43: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 44: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 45: 00000000 00000000 00000000 00000000

      Aug 27 12:16:33 larry kernel: RTAS 46: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 47: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 48: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 49: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 50: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 51: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 52: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 53: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 54: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 55: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 56: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 57: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 58: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 59: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 60: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 61: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 62: 00000000 00000000 00000000 00000000 Aug 27 12:16:33 larry kernel: RTAS 63: 00000000 00000000 00000000 00020000 Aug 27 12:16:33 larry kernel: RTAS: 15 -------- RTAS event end ----------

      Reference codes and location codes may appear as RTAS messages. The extended data is also provided in the form of an RTAS message. The extended data contains other reference code words that help in isolating the correct FRUs. The start of the extended data is marked, for example, by the line:

      Aug 27 12:16:33 larry kernel: RTAS: 15 -------- RTAS event begin --------
      The number after the colon is a sequence number that correlates this data with any diagela data having the same sequence number. The end of the extended data is marked by the line:

      Aug 27 12:16:33 larry kernel: RTAS: 15 -------- RTAS event end ---------- 

      with the same sequence number. Word 13 and word 19 are found in the RTAS messages. For example, to find word 13, first find the reference code in the left column of words of the extended data, 10117661. In this example, we find the reference code to the right of "RTAS 4:". This is also word 11. To get word 13, 10110000, simply count the words left-to-right, beginning at word 11.

6. If you performed substep 5.b of step 5, then record any RTAS messages found in the Linux system log in step 5. Or, if you performed substep 5.c of step 5, then record any RTAS and diagela messages found in the Linux system log in step 5, and also record any extended data found in the RTAS messages, especially word 13 and word 19. Ignore all other messages in the Linux system log.

   If the system is configured with more than one logical partition with Linux installed, repeat step 5 and step 6 for all logical partitions that have Linux installed.

7. If you performed substep 5.c of step 5 for the current Linux partition, go to step 8, and when asked in step 8, do not record any RTAS messages from step 7 for the current Linux partition.

   Examine the Linux boot (IPL) log by logging in to the system as the root user and entering the following command:

   cat /var/log/boot.msg |grep RTAS |more

   Linux boot (IPL) error messages are logged into the boot.msg file under /var/log. The following is an example of the Linux boot error log.

   RTAS daemon started RTAS: -------- event-scan begin -------- RTAS: Location Code: U0.1-F3 RTAS: WARNING: (FULLY RECOVERED) type: SENSOR RTAS: initiator: UNKNOWN target: UNKNOWN RTAS: Status: bypassed new RTAS: Date/Time: 20020830 14404000 RTAS: Environment and Power Warning RTAS: EPOW Sensor Value: 0x00000001 RTAS: EPOW caused by fan failure RTAS: -------- event-scan end ----------

8. Record any RTAS messages found in the Linux boot (IPL) log in step 7. Ignore all other messages in the Linux boot (IPL) log. If the system is configured with more than one logical partition with Linux installed, repeat step 7 and step 8 for all logical partitions that have Linux installed.
9. If you performed substep 5.c of 5 for the current Linux partition, go to step 10, and when asked in step 10, do not record any additional extended data from step 7 for the current Linux partition.

   Examine the extended data in both logs.

   The following is an example of the Linux extended data.

   <3>RTAS daemon started <3>RTAS: -------- event-scan begin -------- <3>RTAS: Location Code: U0.1-P1-C2 <4>RTAS: Log Debug: 04 4b2726fb04a00011702c0014000000000000000000000000f1800001001801d3ffffffff010000000000000042343138 20202020383030343236464238454134303030303 030303030303030 <4>RTAS: Log Debug: D2 5046413405020d0a000001000271400100000033434d502044415441000001000000000000010000f180000153595 320444154410000000000000000200216271501050920021627150105092002063715010509535243204441544170 2c001400000000000000020018820201d382000000000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000280048ea400000000000000000000 000000000000000000004350542044415441702cff08000000001c000000702cf0080000000080000000702cf10070 2cf200702c000400000800702c01040bf2002e702c02040c1fffbf702c0300702c1000702c11040bf2002e702c12040 c1fffbf702c1300702ca000702ca108000000000000a03c702ca208000000000000effc702cb000702cb10800000000 0000a03c702cb208000000000000effc702cc000702cc108000000000000a03c702cc208000000000000effc702c3 000702c31080000000000000003702c3208000000000000007b702c8000702c81080000000020e27a39702c820800 000000fffeffff702cd000702cd1080000000010004010702cd208000000007777f3ffffffffffffffffffffff ffffffffffffffffffffffffffffffffffffffffffff <3>RTAS: WARNING: (FULLY RECOVERED) type: INTERN_DEV_FAIL <3>RTAS: initiator: UNKNOWN target: UNKNOWN <3>RTAS: Status: unrecoverable new <3>RTAS: Date/Time: 20020905 15372200 <3>RTAS: CPU Failure <3>RTAS: Internal error (not cache) <3>RTAS: CPU id: 0 <3>RTAS: Failing element: 0x0000 <3>RTAS: -------- event-scan end ---------

10. Record any extended data found in the Linux system log in Step 5 or the Linux boot (IPL) log in step 7. Be sure to record word 13.
    Note: The line(s) in the Linux extended data that begin with "<4>RTAS: Log Debug: 04" contain the reference code listed in the next 8 hex characters. In the previous example, "4b27 26fb" is a reference code. The reference code is also known as word 11. Each 4 bytes after the reference code in the Linux extended data is another word (for example, "04a0 0011" is word 12, and "702c 0014" is word 13, and so on).

    If the system is configured with more than one logical partition with Linux installed, repeat step 9 and step 10 for all logical partitions that have Linux installed.

11. Were any reference codes or checkpoints recorded in steps 3, 6, 8, or 10?

    No

    Go to step 12.

    Yes

    Go to the Linux fast-path problem isolation with each reference code that was recorded. Perform the indicated actions one at a time for each reference code until the problem has been corrected. If all recorded reference codes have been processed and the problem has not been corrected, go to step 12.

12. If no additional error information is available and the problem has not been corrected, do the following:
    1. Shut down the system.
    2. If an HMC is not attached, see Accessing the Advanced System Management Interface (ASMI) for instructions to access the ASMI.
       Note: The ASMI functions can also be accessed by using a personal computer connected to system port 1.

       You need a personal computer (and cable, part number 62H4857) capable of connecting to system port 1 on the system unit. (The Linux login prompt cannot be seen on a personal computer connected to system port 1.) If the ASMI functions are is not otherwise available, use the following procedure:

       1. Attach the personal computer and cable to system port 1 on the system unit.
       2. With 01 displayed in the operator panel, press a key on the virtual terminal on the personal computer. The service ASMI menus are available on the attached personal computer.
       3. If the service processor menus are not available on the personal computer, perform the following steps:
          1. Examine and correct all connections to the service processor.
          2. Replace the service processor.
             Note: The service processor might be contained on a separate card or board; in some systems, the service processor is built into the system backplane. Contact your next level of support for help before replacing a system backplane.
    3. Examine the service processor error log. Record all reference codes and messages written to the service processor error log. Go to step 13.
13. Were any reference codes recorded in step 12?

    No

    Go to step 21.

    Yes

    Go to the Linux fast-path problem isolation with each reference code or symptom you have recorded. Perform the indicated actions, one at a time, until the problem has been corrected. If all recorded reference codes have been processed and the problem has not been corrected, go to 21.

14. Reboot the system and bring all partitions to the login prompt. If Linux is not usable in all partitions, go to step 18.
15. Use the lscfg command to list all resources assigned to all partitions. Record the adapter and the partition for each resource.
16. To determine if any device(s) or adapter(s) are missing, compare the list of partition assignments, and resources found, to the customer's known configuration. Record the location of any missing devices. Also record any differences in the descriptions or the locations of devices.

    You may also compare this list of resources that were found to a prior version of the device tree as follows:

    Note: Every time that the partition is booted, the update-device-tree command is run and the device tree is stored in the /var/lib/lsvpd/ directory in a file with the file name device-tree-YYYY-MM-DD-HH:MM:SS, where YYYY is the year, MM is the month, DD is the day, and HH, MM, and SS are the hour, minute and second, respectively, of the date of creation.
    • At the command line, type the following:

      cd /var/lib/lsvpd/

    • At the command line, type the following:

      lscfg -vpd db-2003-03-31-12:26:31.

      This displays the device tree created on 03/31/2003 at 12:26:31.

    The diff command offers a way to compare the output from a current lscfg command to the output from an older lscfg command. If the files names for the current and old device trees are current.out and old.out, respectively, type: diff old.out current.out. Any lines that exist in the old, but not in the current will be listed and preceded by a less-than symbol (<). Any lines that exist in the current, but not in the old will be listed and preceded by a greater-than symbol (>). Lines that are the same in both files are not listed; for example, files that are identical will produce no output from the diff command. If the location or description changes, lines preceded by both < and > will be output.

    If the system is configured with more than one logical partition with Linux installed, repeat 15 and 16 for all logical partitions that have Linux installed.

17. Was the location of one and only one device recorded in 16?

    No

    If you previously answered Yes to step 17, return the system to its original configuration. This ends the procedure. Go to MAP 0410: Repair checkout.

    If you did not previously answer Yes to step 17, go to step 18.

    Yes

    Perform the following steps one at a time. Power off the system before performing each step. After performing each step, power on the system and go to step 14.

    1. Check all connections from the system to the device.
    2. Replace the device (for example, tape or DASD)
    3. If applicable, replace the device backplane.
    4. Replace the device cable.
    5. Replace the adapter.
    6. If the adapter resides in an I/O drawer, replace the I/O backplane.
    7. If the device adapter resides in the CEC, replace the I/O riser card, or the CEC backplane in which the adapter is plugged.
    8. Call service support. Do not go to step 14.

18. Does the system appear to stop or hang before reaching the login prompt or did you record any problems with resources in step 16?
    Note: If the system console or VTERM window is always blank, choose NO. If you are sure the console or VTERM is operational and connected correctly, answer the question for this step.

    No

    Go to step 19.

    Yes

    There may be a problem with an I/O device. Go to PFW1542: I/O problem isolation procedure. When instructed to boot the system, boot a full system partition.

19. Boot the eServer standalone diagnostics, refer to Using AIX online and standalone diagnostics. Run diagnostics in problem determination mode on all resources. Be sure to boot a full system partition. Ensure that AIX diagnostics were run on all known resources. You may need to select each resource individually and run diagnostics on each resource one at a time.
    Did standalone diagnostics find a problem?

    No

    Go to step 23.

    Yes

    Go to the Reference codes and perform the actions for each reference code you have recorded. For each reference code not already processed in step 17, repeat this action until the problem has been corrected. Perform the indicated actions, one at a time. If all recorded reference codes have been processed and the problem has not been corrected, go to step 23.

20. Does the system have Linux installed on one or more partitions?

    No

    Return to the Start-of-call isolation procedure.

    Yes

    Go to step 3.

21. Were any location codes recorded in steps 3, 6, 8, 10, 11, or 12?

    No

    Go to step 14.

    Yes

    Replace, one at a time, all parts whose location code was recorded in steps 3, 6, 8, 10, 11, or 12 that have not been replaced. Power off the system before replacing a part. After replacing the part, power on the system to check if the problem has been corrected. Go to step 22 when the problem has been corrected, or all parts in the location codes list have been replaced.

22. Was the problem corrected in step 21?

    No

    Go to step 14.

    Yes

    Return the system to its original configuration. This ends the procedure. Go to MAP 0410: Repair checkout.

23. Were any other symptoms recorded in step 3?

    No

    Call support.

    Yes

    Go to the Start-of-call isolation procedure with each symptom you have recorded. Perform the indicated actions for all recorded symptoms, one at a time, until the problem has been corrected. If all recorded symptoms have been processed and the problem has not been corrected, call your next level of support.