Troubleshooting
Problem
Error "shell request failed on channel 0 " received when SSH login is attempted.
Symptom
System is inaccessible via SSH.
Run sshd in debug mode:
-- Stop sshd
# stopsrc -s sshd
# stopsrc -s sshd
-- Start up a script session
# script /tmp/sshd.debug
# script /tmp/sshd.debug
-- Start up sshd in debug mode
# /usr/sbin/sshd -d -d -d -d
# /usr/sbin/sshd -d -d -d -d
-- Recreate ssh issue by trying to establish a new SSH session
-- Stop script session
# exit
# exit
-- Restart sshd
# startsrc -s sshd
# startsrc -s sshd
/tmp/sshd.debug file will contain an entry similar to:
do_exec_pty: sem_init: no space lest device
Cause
It is possible the system is running out of semaphores to open new SSH sessions.
Environment
All versions of AIX
Diagnosing The Problem
Verify there aren't too many semaphores held by certain process or user.
# ipcs -a
Output should be similar to this:
T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS O TIME CTIME
m 1048576 0xffffffff D-rw------- root system root system 1 268435456 2818362 2818362 16:14:51 no-entry 16:14:51
m 1048577 0xffffffff D-rw------- root system root system 1 536870912 2818362 2818362 16:14:51 no-entry 16:14:51
m 9437192 0xffffffff D-rw------- user1 dstage user1 dstage 1 268435456 3801374 3801374 16:15:10 no-entry 16:15:10
m 9437193 0xffffffff D-rw------- user1 dstage user1 dstage 1 537329664 3801374 3801374 16:15:10 no-entry 16:15:10
m 1048586 0xadecb500 --rw-rw-rw- root system root system 19 27447484 6553752 19988504 12:12:00 12:12:03 16:14:54
m 5242891 0xffffffff D-rw------- user1 dstage user1 dstage 1 268435456 3801374 3801374 16:15:10 no-entry 16:15:10
m 12 0xadeeb500 --rw-rw-rw- root system root system 19 4851604 5701664 19988504 12:12:00 12:12:03 16:14:54
m 1048591 0xffffffff D-rw------- user1 dstage user1 dstage 1 65536 7798922 7798922 16:15:11 no-entry 16:15:11
m 1048593 0xffffffff D-rw------- user1 dstage user1 dstage 1 65536 7798922 7798922 16:15:11 no-entry 16:15:11
m 18 0xffffffff D-rw------- user1 dstage user1 dstage 1 268435456 7798922 6815806 16:15:10 16:15:11 16:15:10
In the above example user1 is holding a number of semaphores. Usually the output is longer than this, indicating top contributor for semaphore consumption.
Resolving The Problem
There are 2 ways to address this:
1- Use ipcrm command to free up some semaphores:
To remove the shared memory segment associated with SharedMemoryID 1048593.
# ipcrm -m 1048593
2- Reboot system.
Document Location
Worldwide
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8m0z0000001fMuAAI","label":"AIX General Support"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]
Was this topic helpful?
Document Information
Modified date:
09 August 2020
UID
ibm16254674