What should I do if a node does not respond?

Look for a possible absorption throttling

Open a shell or a JMX console and execute the listLimits command.

If one of the channels drops to zero, the node is throttling the absorption requests. It means that performance is temporarily reduced to cope with a burst.

If the queue size reaches a high value, there are a lot of absorption requests that are waiting to be processed. The performance of the node is degraded in this situation. You should dump the content of the absorption queue.

g! listLimits
Default channel: 16/16 available permits
Channel cube_computing: 16/16 available permits
Channel ws_data_integration: 16/16 available permits
Channel indicator_computing: 16/16 available permits
Channel data_integration: 16/16 available permits
Queue size: 0

Dump the absorption queue

Open a shell or a JMX console and execute the dumpAbsorptionQueue command (if the queue is really big you might want to execute the dumpAbsorptionQueueIntoFile command instead).

g! dumpAbsorptionQueue
AbsorptionTransactionImpl{transactionTime = 1441542908712845313, channel=behavior_computing, localPermit=Acquired, estimatedSize=1548, ...
AbsorptionTransactionImpl{transactionTime = 1441542908717041665, channel=behavior_computing, localPermit=Acquired, estimatedSize=4, ...
AbsorptionTransactionImpl{transactionTime = 1441542908718092289, channel=behavior_management, localPermit=Acquired, estimatedSize=1, ...

Look for blocked threads

Read How to generate a JVM thread dump? for the steps to follow. Look for threads waiting for a lock.

Report a bug

If a standalone node or a primary node seem stuck, you can report a bug. To help fix the bug, we encourage you to submit the following information along with the bug report:

Related Links