TCPC0005W: TCP Channel sonOutputTcpConnectorOutputChannel_0 could not obtain thread from thread pool sonOutThreadPool

Troubleshooting

Problem

Intelligent Management provides application server virtualization, resource management, and a host of advanced operational facilities, such as performance visualization, health management, and application edition management. Use Intelligent Management to enhance operational efficiency by deploying dynamic operations, servicing high-volume transactional workloads with linear scalability and high availability, and by managing large scale, continuously available application server environments.

Symptom

Some clients may see on rare instances the following warning messages being issued:

[5/26/23 7:34:06:101 CEST] 00000052 ThreadPool I WSVR0629I: The request buffer for thread pool "sonOutThreadPool" has reached its capacity.
[5/26/23 7:34:06:128 CEST] 00000052 WorkQueueMana W TCPC0005W: TCP Channel sonOutputTcpConnectorOutputChannel_0 could not obtain thread from thread pool sonOutThreadPool.
[5/26/23 7:46:02:879 CEST] 00000052 WorkQueueMana W TCPC0005W: TCP Channel sonOutputTcpConnectorOutputChannel_0 could not obtain thread from thread pool sonOutThreadPool.
[5/26/23 8:07:20:888 CEST] 00000052 WorkQueueMana W TCPC0005W: TCP Channel sonOutputTcpConnectorOutputChannel_0 could not obtain thread from thread pool sonOutThreadPool.
[5/26/23 9:46:24:531 CEST] 00000052 WorkQueueMana W TCPC0005W: TCP Channel sonOutputTcpConnectorOutputChannel_0 could not obtain thread from thread pool sonOutThreadPool.

Clients may see these messages from time to time. Concerns around possible tuning of the resources may arise from customers.

Cause

The warning messages do not indicate that there is an issue, its simply a warning message. The following cell custom properties can be adjusted
son.tcpInThreadPoolMin,
son.tcpInThreadPoolMax,
son.tcpOutThreadPoolMin,
son.tcpOutThreadPoolMax.

The "son" here is revolving around P2P/SON. P2P/SON is the Intelligent Management (IM) communication infrastructure. P2P/SON is used primarily by ODC and BBSON. ODC is a collection of live topology information and metrics delivered to every JVM in the cell and every JVM contributes to this shared information. BBSON is a bulletin board implementation used solely by IM that sits on-top of P2P/SON.

In order for Intelligent Management (IM) to be completely dynamic as IM features are enabled, some internal IM functions are always active and P2P/SON will be carrying IM data.

Diagnosing The Problem

They are basic references in the performance cookbook
https://publib.boulder.ibm.com/httpserv/cookbook/WebSphere_Application_Server-WAS_traditional-Intelligent_Management.html#WebSphere_Application_Server_traditional-Intelligent_Management-Bulletin_Board_over_the_Structured_Overlay_Network_BBSON

Resolving The Problem

Modifications to the default values may provide relief from the messages. It would be best practice to start with the recommended updates and then change incrementally while testing the changes. Making large changes that deviate from the recommended values below may negatively impact the overlay and cause it to become unstable or unreliable.

The following four values control the P2P/SON thread-pool. There really isn't a good place to make note of them in IBM Doc's.

The current defaults listed below:

son.tcpInThreadPoolMin = 5

son.tcpInThreadPoolMax = 10

son.tcpOutThreadPoolMin = 5

son.tcpOutThreadPoolMax = 10

If seeing frequent warnings `WorkQueueMana W TCPC0005W: TCP Channel sonOutputTcpConnectorOutputChannel_0 could not obtain thread from thread pool sonOutThreadPool.` modify the cell custom properties to use the following values.

son.tcpInThreadPoolMin = 10

son.tcpInThreadPoolMax = 10

son.tcpOutThreadPoolMin = 20

son.tcpOutThreadPoolMax = 20

Save, sync, and restart all running jvms for the new values to be honored. The above values should never be changed such they are SMALLER than the defaults.

Related Information

Bulletin Board over the Structured Overlay Network (BBSON)

Document Location

Worldwide

Operating System

Cross Brand:All operating systems listed

[{"Type":"MASTER","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"ARM Category":[{"code":"a8m0z0000004CnTAAU","label":"WebSphere Application Server traditional-All Platforms-\u003EIntelligent Management"}],"ARM Case Number":"TS013152858","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Product Synonym

Intelligent Management;IM

Was this topic helpful?

Document Information

Modified date:
16 July 2024

UID

ibm17006883

Tips