IBM Streams 4.2.1

Unable to connect to the ZooKeeper ensemble

You might not be able to connect to the ZooKeeper ensemble if you exceeded the maximum number of connections, or there are network, host, or ZooKeeper problems.

Symptoms

If you exceeded the maximum number of allowed connections, you might see the following message in the ZooKeeper logs:

WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /xxx.xxx.x.xxx - max is 60

If you encountered an unknown host error, you might see the following exception in the ZooKeeper logs:

java.net.UnknownHostException: xyz: xyz
at java.net.InetAddress.getAllByName0(InetAddress.java:1270)
at java.net.InetAddress.getAllByName(InetAddress.java:1197)
at java.net.InetAddress.getAllByName(InetAddress.java:1119)
at org.apache.zookeeper.client.StaticHostProvider.<init>(ZooKeeper.java:445)
at org.apache.zookeeper.ZooKeeper<init>(ZooKeeper.java:380)

If IBM® Streams interfaces or services have problems connecting to ZooKeeper, you might see one or more of the following symptoms:
- ConnectionLoss error in the ZooKeeper logs.
- ConnectionLoss error at the command prompt after a streamtool command hangs and then fails.
- ConnectionLoss error in the IBM Streams trace file after a service fails to start.
- Automatic stop and restart of a service.

Causes

You exceeded the maximum number of allowed connections.
An unknown host error occurred.
ZooKeeper or the network might be down.

Resolving the problem

Ensure that the configuration setting for the maximum number of client connections is sufficient enough to avoid the loss of connections.
- External ZooKeeper: If necessary, update the value of the maxClientCnxns configuration parameter in the ZooKeeper-installation-directory/conf/zoo.cfg file.
- Embedded ZooKeeper: Check the value of the streams.zookeeper.property.maxClientCnxns bootstrap property by using the streamtool getbootproperty -a command. By default, the number of connections is unlimited (0). If necessary, update the value by using the streamtool setbootproperty command.
Ensure that the host can be resolved by your DNS.
Ensure that the ZooKeeper ensemble is running and serving by using the ZooKeeper srvr or stat command. For more information, see the ZooKeeper Administrator’s Guide.
Restart ZooKeeper:
- To restart external ZooKeeper, use the zkServer.sh script.
- To restart embedded ZooKeeper, use the streamtool embeddedzk --stop and streamtool embeddedzk --start commands.
If all hosts are up and running and you continue to see ConnectionLoss errors, ensure that there are no system issues with CPU services, memory, disk input/output, or other system resources.