Troubleshoot startup failure with large numbers of trading pickups

Problem: When starting Activator servers when you have large number of trading pickups, the Activator system throttle is engaged and startup fails.

"Too may open files" and null pointer exceptions are thrown. The following is an example of the type of errors related to this failure:

2014-06-09 10:32:18,889 - WARN [SystemThrottle] (SystemThrottle.run:224) - Engaging the SystemThrottle, status is Error writing test file

/nfs3/user/b2b21_L3851_Common/common/data/backup/fileSystemHealth/linux51_te.txt - java.io.FileNotFoundException:

/nfs3/user/b2b21_L3851_Common/common/data/backup/fileSystemHealth/linux51_te.txt (Too many open files)

 

2014-06-03 17:02:57,135 - WARN [Cluster Thread 56] (Jetty.getHandlersFromJettyConfig:307) - Jetty configuration file for HTTP 0.0.0.0:10199 missing:

/home/user/B2B2.1/Interchange/bin/../webappsSpecial/jettyTe.xml

2014-06-03 17:02:57,136 - ERROR [Cluster Thread 56] (ManagedHttpServer.start:76) - Error starting HTTP 0.0.0.0:10199

com.axway.transport.http.server.HttpServerException: Error starting HTTP 0.0.0.0:10199; java.lang.NullPointerException

at com.axway.transport.http.server.Jetty.doStart(Jetty.java:86)

at com.axway.transport.http.server.HttpServletEngine.start(HttpServletEngine.java:77)

at com.cyclonecommerce.tradingengine.transport.http.server.ManagedHttpServer.start(ManagedHttpServer.java:72)

at com.cyclonecommerce.tradingengine.transport.server.ManagedServer.startServer(ManagedServer.java:292)

at com.cyclonecommerce.tradingengine.transport.server.ManagedServer.start(ManagedServer.java:100)

at com.cyclonecommerce.tradingengine.transport.server.ServerManager.start(ServerManager.java:108)

at com.cyclonecommerce.tradingengine.transport.system.distribution.StartStopMessage.execute(StartStopMessage.java:118)

at com.axway.cluster.messaging.MessageExecutionWrapper.execute(MessageExecutionWrapper.java:29)

at com.axway.cluster.bus.connection.Connection.handleMessage(Connection.java:249) at com.axway.cluster.bus.comm.MessageExecutor.run(MessageExecutor.java:48)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)

at com.axway.cluster.extensions.thread.EventedThread.primRun(EventedThread.java:102)

at com.axway.cluster.extensions.thread.EventedThread.run(EventedThread.java:80)

Caused by: java.lang.NullPointerException

at org.eclipse.jetty.server.handler.HandlerCollection.setHandlers(HandlerCollection.java:86)

at org.eclipse.jetty.server.handler.HandlerCollection.addHandler(HandlerCollection.java:155)

at com.axway.transport.http.server.Jetty.handlerAsCollection(Jetty.java:333)

at com.axway.transport.http.server.Jetty.getHandlersFromJettyConfig(Jetty.java:318)

at com.axway.transport.http.server.Jetty.createHandlersFromConfig(Jetty.java:182)

at com.axway.transport.http.server.Jetty.doStart(Jetty.java:71) ... 14 more

Solution: Working with your system administrator, modify the Activator user account on the operating system where Activator is installed. Set the ‘open files’ property to a high number. The optimum setting depends on the way your system is to be used. The greater the number of pickups, the larger the setting should be.

For example, for a community configured with 2000 Trading pickups, begin with an 'open files' setting of 102400, check the results and adjust accordingly.

Tips for estimating the optimum 'open files' setting on Linux

A Linux root user can run the following command to get a rough estimate of the open file count of a given user for any given moment:

[root@linux38 ~]# lsof | grep <user_name> | wc –l

Run the command in a loop while the system starts to determine if the existing setting for the "open files" property is acceptable. If the errors listed above appear in the logs, the setting is not high enough. If the system starts cleanly the setting is correct.

Example:

Use the lsof command to estimate an appropriate value for the ‘open files’ property:

Run the following loop while the nodes are restarting - (run as your user, not root).

[root@linux38 ~]# while true; do lsof | grep <my_user> | wc -l; sleep 5; done

This command reports the open file count for the specified user every 5 seconds. For example:

15387

15387

15387

15387

15387

15387

15387

15387

Working with the system administrator, use these results to set the 'open files' value. The values that are returned vary depending on the number of community pickups that are defined. The setting you use for the 'open files' property should be higher than the maximum 'open file count' value that is returned, allowing a buffer for additional resources.

Related Links