ProblemDetermination: NodeAgent and Deployment Manager discovery problems

Resolving The Problem

Traces:

1. Did the discovery initialize properly?
   
   The following are examples of working initialization traces for each process:
   • dmgr Discovery init:
     [12/09/04 9:32:42:378 EDT] 3008e257 JMXConnectors > initDiscovery          .....                                          
     [12/09/04 9:32:42:608 EDT] 3008e257 JMXConnectors d processing endPoint:
     NODE_DISCOVERY_ADDRESS
     [12/09/04 9:32:42:608 EDT] 3008e257 JMXConnectors d node agent discovery endpoint found for node: myNode    
     ....
     [12/09/04 9:32:42:718 EDT] 3008e257 JMXConnectors > doCellMgrDiscovery  .....
     [12/09/04 9:32:43:650 EDT] 3008e257 JMXConnectors e The deployment manager is looking for the node agent:
     myNode
     xxx.yyy.ibm.com
     7272
     {cell=myNetwork, node=myNode, role=NodeAgent}
     .....
     [12/09/04 9:32:43:650 EDT] 3008e257 JMXConnectors < doCellMgrDiscovery  .....
     [12/09/04 9:32:43:650 EDT] 3008e257 JMXConnectors < initDiscover
   • NodeAgent Discovery init:
     [12/09/04 9:31:57:343 EDT] 303c6255 JMXConnectors > initDiscovery
     ....
     [12/09/04 9:31:57:543 EDT] 303c6255 JMXConnectors d processing endPoint:
     CELL_DISCOVERY_ADDRESS
     [12/09/04 9:31:57:543 EDT] 303c6255 JMXConnectors d cellmgr node name: myManager
     ....
     [12/09/04 9:31:58:234 EDT] 303c6255 JMXConnectors > doNodeAgentDiscovery
     ....
     [12/09/04 9:31:58:585 EDT] 303c6255 JMXConnectors e The node agent is looking for the cell manager:
     myManager
     xxx.yyy.ibm.com
     7277
     {cell=myNetwork, node=myManager, role=DeploymentManager}
     ....
     [12/09/04 9:31:58:615 EDT] 303c6255 JMXConnectors e looking for the managed process (multicast based):
     232.133.104.73
     5000
     {cell=myNetwork, node=myNode, role=ManagedProcess}
     ....
     [12/09/04 9:31:59:656 EDT] 303c6255 JMXConnectors < doNodeAgentDiscovery
     ....
     [12/09/04 9:31:59:656 EDT] 303c6255 JMXConnectors < initDiscovery
      
   • Application Server Discovery init:
     [12/09/04 9:34:18:466 EDT] 3069e27f JMXConnectors > initDiscovery
     .....
     [12/09/04 9:34:18:806 EDT] 3069e27f JMXConnectors d Initialize the managed process discovery service: multicast
     ......
     [12/09/04 9:34:18:947 EDT] 3069e27f JMXConnectors e The managed process is looking for its node agent:
      myNode
      xxx.yyy.ibm.com
      7272
      {cell=myNetwork, node=myNode, role=NodeAgent}
     [12/09/04 9:34:18:957 EDT] 3069e27f JMXConnectors < doManagedProcessDiscovery
     .....
     [12/09/04 9:34:18:957 EDT] 3069e27f JMXConnectors < initDiscovery
      
2. Did the discovery broadcast a message and get a connection?
   
   The following are example trace entries to verify if the preceding worked. Verify the host and port entries in the serverindex.xml file.
   • Sender: Broadcast to Receiver Discovery Ports:
     [6/28/04 11:58:55:374 EDT] 604a9020 DiscoveryServ d message[src, dest]
     tcp://9.37.37.37:7272/DiscoveryService
     tcp://9.37.37.37:7277/DiscoveryService
     ...
     [6/28/04 11:58:55:374 EDT] 604a9020 TcpMessenger  > openSocket
     [6/28/04 11:58:55:374 EDT] 604a9020 TcpMessenger  d InetAddress, port: 9.37.37.37/9.37.37.37
     7277
     ...
     [6/28/04 11:58:55:384 EDT] 604a9020 TcpMessenger  < openSocket - good
     ...
     [6/28/04 11:58:55:384 EDT] 604a9020 TcpMessenger  < sendMessage - successful
     ...
     [6/28/04 11:58:55:384 EDT] 604a9020 TcpConnection d  End of inputsteam. Closing socket
      
3. Did the Receiver get a valid Discovery Query Message?
   
   The following is example trace output when the NodeAgent is acting as the Receiver and the Deployment Manager initiates the discovery process.
   • Receiver: NodeAgent trace.log:
     [6/28/04 11:58:52:219 EDT] 604a9020 TcpConnection > run
     …
     [6/28/04 11:58:52:239 EDT] 604a9020 DiscoveryServ d the incoming discovery message is:
     <?xml version="1.0" encoding="UTF-8"?>
     <jxta:DiscoveryQuery><Type>0</Type><QueryId>1</QueryId><PeerAdv>
     <jxta:PeerAdvertisement><Name>dmgr</Name><Pid>4344</Pid><Version>5.0.2.2</Version>
     <Cell>ibmNetwork</Cell>
     <Node>ibmManager</Node>
     <Role>DeploymentManager</Role>
     <Endpoint><jxta:EndpointAdvertisement>
     <Name>tcp</Name><Address>tcp://ibm:7277</Address>
     <Transport><jxta:TCPTransportAdvertisement>
     <Protocol>tcp</Protocol>
     <Port>7277</Port>
     <PublicAddress>ibm</PublicAddress>
     <LocalAddress>ibm</LocalAddress>
     </jxta:TCPTransportAdvertisement>
     </Transport>
     </jxta:EndpointAdvertisement>
     </Endpoint>
     <Service><jxta:ServiceAdvertisement>
     <Name>AdminService</Name>
     <AccessMethod>endpoint:name=BOOTSTRAP_ADDRESS;host=ibm;port=9809
     </AccessMethod>
     <AccessMethod>connector:type=SOAP;host=ibm;port=8879;preferred=true;isInternal=true
     </AccessMethod>
     <AccessMethod>connector:type=RMI;host=ibm;port=9809;isInternal=true
     </AccessMethod>
     </jxta:ServiceAdvertisement>
     </Service>
     </jxta:PeerAdvertisement>
     </PeerAdv>
     <Cell>ibmNetwork</Cell>
     <Node>ibm</Node>
     <Role>NodeAgent</Role>
     </jxta:DiscoveryQuery>
     …
     [6/28/04 11:58:54:903 EDT]  b875025 DiscoveryMBea I ADMD0023I: Process discovered (name: dmgr, type: DeploymentManager, pid: 4344)
     …
     [6/28/04 11:58:55:384 EDT] 604a9020 TcpConnection d  End of inputsteam. Closing socket
4. Did the Sender get a valid Discovery Response Message?
   
   The following is example trace output when the Deployment Manager is acting as the Sender and gets the response from the NodeAgent.
   • Sender: Deployment Manager trace.log:
     [6/28/04 11:58:55:384 EDT] 7d9b9044 TcpConnection > run
     ...
     [6/28/04 11:58:55:384 EDT] 7d9b9044 DiscoveryServ d the incoming discovery message is:
     <?xml version="1.0" encoding="UTF-8"?>
     <jxta:DiscoveryResponse><Type>0</Type><QueryId>1</QueryId><Response>
     <jxta:PeerAdvertisement><Name>nodeagent</Name><Pid>3316</Pid>
     <Version>5.0.2.2</Version>
     <Cell>ibmNetwork</Cell>
     <Node>ibm</Node>
     <Role>NodeAgent</Role>
     <Endpoint><jxta:EndpointAdvertisement>
     <Name>tcp</Name>
     <Address>tcp://ibm:7272</Address>
     <Transport><jxta:TCPTransportAdvertisement><Protocol>tcp</Protocol>
     <Port>7272</Port>
     <PublicAddress>ibm</PublicAddress>
     <LocalAddress>ibm</LocalAddress>
     </jxta:TCPTransportAdvertisement></Transport></jxta:EndpointAdvertisement>
     </Endpoint>
     <Service><jxta:ServiceAdvertisement>
     <Name>AdminService</Name>
     <AccessMethod>endpoint:name=BOOTSTRAP_ADDRESS;host=ibm;port=2809
     </AccessMethod>
     <AccessMethod>connector:type=SOAP;host=ibm;port=8878;preferred=true;isInternal=true
     </AccessMethod>
     <AccessMethod>connector:type=RMI;host=ibm;port=2809;isInternal=true
     </AccessMethod>
     </jxta:ServiceAdvertisement>
     </Service>
     </jxta:PeerAdvertisement>
     </Response>
     <Cell>ibmNetwork</Cell>
     <Node>ibmManager</Node>
     <Role>DeploymentManager</Role>
     </jxta:DiscoveryResponse>
     ...
     [6/28/04 11:58:57:937 EDT] 1888d04c DiscoveryMBea I ADMD0023I: Process discovered (name: ibm, type: NodeAgent, pid: 3316)
     …
     [6/28/04 11:58:58:268 EDT] 7d9b9044 TcpConnection d  End of inputsteam. Closing socket