Action 2: investigate closed client connection

Edit online

Use this recovery procedure when the file /var/ct/IW/log/mc/default contains a message indicating that the client connection was closed due to an incorrect message.

Symptom:

The file /var/ct/IW/log/mc/default contains the following message:

2610-204 The client connection is closed due to incorrect message

(incorrect message code)

Diagnosis:

The 2610-204 message is logged by the RMC subsystem, and the client session is closed, under the following circumstances:

The client message policy has been violated (incorrect message code is 65536)
An incorrectly formatted message has been received (incorrect message code is less than 65536)
A time limit has been exceeded (incorrect message code is 131072)

Recovery procedure:

If the client message policy has been violated, see the description of the rmcctrl command's -m flag in Technical Reference: RSCT for AIX® or Technical Reference: RSCT for Multiplatforms guides. See also the description of RMC network port usage, data flows and security in Administering RSCT guide.

If an incorrectly-formatted message has been received, then a program not using the RMC Application Programming Interface (RMC API) has connected to the RMC subsystem.

If the incorrect message code is greater than 32768, then the program connected via the 657/tcp port. In this case, verify that connections to the RMC daemon are issued from trusted or permitted hosts.
If the incorrect message code is less than or equal to 32768, the program connected via the local UNIX Domain Socket /var/ct/IW/soc/mc/clsrv. In this case, review usage of the local system with respect to the RMC subsystem.

If a time limit has been exceeded, verify that legitimate RMC commands or client applications connecting to the local RMC subsystem have failed. If so, then execute the following command:

lssrc -ls ctrmc

In the section of command output labeled Internal Daemon Counters, examine the value of the counters.

1st msg timeouts =          0  Message timeouts =          0 

Start timeouts    =          0  Command timeouts =         0

In the command output:

The 1st msg timeouts counter indicates the number of times the RMC subsystem closed client connections that failed to send the first message of the start session protocol within the client message time-out limit.
The Message timeouts counter indicates the number of times the RMC subsystem closed client connections that failed to send a complete message within the client message time-out limit. Use the -t option of the rmcctrl command to change the client message time-out limit.
The Start timeouts counter indicates the number of times the RMC subsystem closed client connections that failed to complete the start session protocol within the start session time-out limit. Use the rmcctrl command with its -u option to change the start session time-out limit.
The Command timeouts counter indicates the number of times the RMC subsystem closed client connections that failed to send the first command, subsequent to completion of start session processing, within the first command time-out limit. Use the rmcctrl command with its -v option to change the first command time-out limit.

If legitimate RMC command and client applications are failing and these counters are incrementing, the most likely cause is notwork congestion. The time-out limits may be increased. If RMC commands and client applications are failing as a result of reaching the limit on number of clients sessions, then the first command threshold may be decreased (using the rmcctrl command with its -w option), and first command time-outs for non-root authenticated client sessions can be enabled (using the rmcctrl command with its -x option). The result of these actions is to increase the number of client sessions subject to the first command timer.

If legitimate RMC command and client applications are not failing, but these counters are incrementing, it is likely that unknown applications are connecting to the 657/tcp port or the local UNIX Domain Socket. In the former case, verify that connections to the RMC daemon are issued from trusted or permitted hosts. In the latter case, review usage of the local system with respect to the RMC subsystem.

To view the current configuration of these time-out limits, execute the following command:

lssrc -ls ctrmc

Included in the output are messages similar to the following:

Client message timeout: 10 

Client start session timeout: 60 

Client first command threshold: 150 

Client first command timeout: 10 

Client first command timeout applies to unauthenticated and non-root  

  authenticated users

If first command timers are not enabled (the threshold is 0), the last three messages are not present. The first command threshold can be modified using the rmcctrl command with its -w option. By default, first command timers do not apply to non-root authenticated users. To have non-root authenticated users subject to the first command time-out limits use the rmcctrl command with its -x option.