Elasticsearch problems on Seafile Pro

Fred · February 15, 2021, 8:51pm

Hi,
I’ve been having this issue for a long time and I thought it was due to the automatic script installation but I’ve spent some time recently manually installing seafile and the issue persists.
On seafile start there are 2 elasticsearch java processes created which causes elasticsearch to crash and restart - this will go on indefinitely with the increasing cpu load and RAM making the server unresponsive.
This issue has been referenced multiple times here in the forum:

I am hosting seafile (pro edition) on a Kimsufi server. I have installed it on a KVM VM at home and the issue is not present - so, I thought it was due to something OVH may have done to the kernel which led me to install ubuntu (tried 16.04, 18.04 and 20.04) from debootstrap but the issue remains with all editions.

It has nothing to do with lack of RAM (server has 4GB, there is only 1 user and the issue is present when I first start seafile right after installation, without any data, any libraries or even any seahub user created); I have also checked values at seafile-server-latest/pro/elasticsearch/config/jvm.options (both correctly configured with “Xms1g”, starting seafile with the option “LimitNOFILE=65536” and adding “vm.max_map_count=262144” to the sysctl.conf file - all to no avail. I have also confirmed that all permissions and file/folder ownership are set correctly. I have tried deleting all nodes under pro-data to no avail (again, this happens even before there are any nodes, at the very first start of seafile after running installation script)

The elasticsearch log file shows:

[2021-02-15T20:16:35,485][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/seafile/pro-data/search/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.13.jar:5.6.13]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/seafile/pro-data/search/data/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:265) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.13.jar:5.6.13]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.13.jar:5.6.13]

The controller log:

[02/15/21 20:13:08] seafile-controller.c(96): spawn_process: /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/bin/elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
[02/15/21 20:13:08] seafile-controller.c(130): spawned /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/bin/elasticsearch, pid 23916
[02/15/21 20:13:18] seafile-controller.c(585): pid file /opt/seafile/pids/elasticsearch.pid does not exist
[02/15/21 20:13:18] seafile-controller.c(633): elasticsearch need restart...
[02/15/21 20:13:18] seafile-controller.c(96): spawn_process: /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/bin/elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
[02/15/21 20:13:18] seafile-controller.c(130): spawned /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/bin/elasticsearch, pid 23979
[02/15/21 20:13:28] seafile-controller.c(585): pid file /opt/seafile/pids/elasticsearch.pid does not exist
[02/15/21 20:13:28] seafile-controller.c(633): elasticsearch need restart...
[02/15/21 20:13:28] seafile-controller.c(96): spawn_process: /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/bin/elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
[02/15/21 20:13:28] seafile-controller.c(130): spawned /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/bin/elasticsearch, pid 24042

The various java processes being created (note that one process remains locked, which forces elasticsearch to keep restarting and creating a new process):

$ ps aux | grep 'java'
seafile    23443 78.9 30.9 4565156 1243432 ?     Sl   20:11   1:21 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24042  140 28.3 3513748 1138964 ?     Sl   20:13   0:05 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24094  0.0  0.0   8160   668 pts/0    S+   20:13   0:00 grep java
$ ps aux | grep 'java'
seafile    23443 71.4 30.9 4565156 1243696 ?     Sl   20:11   1:21 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24107  138 28.4 3513748 1139888 ?     Sl   20:13   0:06 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24159  0.0  0.0   8160   732 pts/0    S+   20:13   0:00 grep java
$ ps aux | grep 'java'
seafile    23443 67.2 30.9 4565156 1243696 ?     Sl   20:11   1:21 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24172 82.5 26.5 1665352 1064328 ?     Sl   20:13   0:01 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24211  0.0  0.0   8160   736 pts/0    S+   20:13   0:00 grep java
$ ps aux | grep 'java'
seafile    23443 60.8 30.9 4565156 1243696 ?     Sl   20:11   1:21 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24237  157 28.4 3513748 1142408 ?     Sl   20:13   0:07 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24289  0.0  0.0   8160   736 pts/0    S+   20:14   0:00 grep java
$ ps aux | grep 'java'
seafile    23443 50.0 30.9 4565156 1243696 ?     Sl   20:11   1:21 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24427  143 28.3 3513748 1138680 ?     Sl   20:14   0:05 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24479  0.0  0.0   8160   736 pts/0    S+   20:14   0:00 grep java
$ ps aux | grep 'java'
seafile    23443 40.5 30.9 4565156 1243696 ?     Sl   20:11   1:21 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24681 63.0 27.9 3046356 1121668 ?     Sl   20:15   0:01 /bin/java -Xms1g -Xmx1g -XX:-AssumeMP -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch -cp /opt/seafile/seafile-pro-server-7.1.13/pro/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
seafile    24724  0.0  0.0   8160   664 pts/0    S+   20:15   0:00 grep java

The strangest thing for me is the fact that I am unable to reproduce this issue at home installing seafile on a proxmox VM but no matter what i do the issue is recurring on the kimsufi servers (KS-3). The issue has been present for well over a year, possibly since seafile 6.x

The only way is to either kill the stuck java process or restart seafile (sometimes several times as restarting seafile can cause the same issue again). Starting seafile from systemctl or using seafile user is exactly the same.

This is driving me crazy and I’m all out of ideas. Any help would be very much appreciated.

Weeehe · February 16, 2021, 5:45pm

did you ever stop/kill all ES processes and check for the existence of the pid-file? If it exists just remove it and start ES again.
When the controller restart all the time this is a common issue.

Fred · February 16, 2021, 6:38pm

When I kill all ES processes (I think you mean seafile as I think there is no way to start/stop ES independently of seafile) there isn’t any PID file left in the folder, they are all cleared.

upD8R · February 17, 2021, 8:34pm

I feel your pain.Same story here …
I set up a dedicated ES instance on the same server, which doesn’t show this behavior. Unfortunately, Seafile is not compatible with recent ES versions, so I’m stuck again.

upD8R · February 19, 2021, 6:37am

It’s a pity that noone is looking into this. As said, I had to disable Seafile autostart because of this but the server required a reboot today. So I started Seafile manually afterwards and here’s the result:

[02/19/21 07:26:33] seafile-controller.c(130): spawned /usr/bin/python3, pid 1984
[02/19/21 07:26:34] seafile-controller.c(585): pid file /home/seafile/pids/seafdav.pid does not exist
[02/19/21 07:26:34] seafile-controller.c(627): seafdav need restart...
[02/19/21 07:26:34] seafile-controller.c(96): spawn_process: /usr/bin/python3 -m wsgidav.server.server_cli --server gunicorn --root / --log-file /home/seafile/logs/seafdav.log --pid /home/seafile/pids/seafdav.pid --port 8080 --host 127.0.0.1
[02/19/21 07:26:34] seafile-controller.c(130): spawned /usr/bin/python3, pid 1985
[02/19/21 07:26:35] seafile-controller.c(585): pid file /home/seafile/pids/elasticsearch.pid does not exist
[02/19/21 07:26:35] seafile-controller.c(633): elasticsearch need restart...
[02/19/21 07:26:35] seafile-controller.c(96): spawn_process: /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch -Epath.logs=/home/seafile/logs -Epath.data=/home/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /home/seafile/pids/elasticsearch.pid
[02/19/21 07:26:35] seafile-controller.c(130): spawned /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch, pid 1987
[02/19/21 07:26:43] seafile-controller.c(585): pid file /home/seafile/pids/elasticsearch.pid does not exist
[02/19/21 07:26:43] seafile-controller.c(633): elasticsearch need restart...
[02/19/21 07:26:43] seafile-controller.c(96): spawn_process: /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch -Epath.logs=/home/seafile/logs -Epath.data=/home/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /home/seafile/pids/elasticsearch.pid
[02/19/21 07:26:43] seafile-controller.c(130): spawned /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch, pid 2060
[02/19/21 07:27:03] seafile-controller.c(585): pid file /home/seafile/pids/elasticsearch.pid does not exist
[02/19/21 07:27:03] seafile-controller.c(633): elasticsearch need restart...
[02/19/21 07:27:03] seafile-controller.c(96): spawn_process: /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch -Epath.logs=/home/seafile/logs -Epath.data=/home/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /home/seafile/pids/elasticsearch.pid
[02/19/21 07:27:03] seafile-controller.c(130): spawned /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch, pid 2158
[02/19/21 07:27:13] seafile-controller.c(585): pid file /home/seafile/pids/elasticsearch.pid does not exist
[02/19/21 07:27:13] seafile-controller.c(633): elasticsearch need restart...
[02/19/21 07:27:13] seafile-controller.c(96): spawn_process: /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch -Epath.logs=/home/seafile/logs -Epath.data=/home/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /home/seafile/pids/elasticsearch.pid
[02/19/21 07:27:13] seafile-controller.c(130): spawned /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch, pid 2214
[02/19/21 07:27:23] seafile-controller.c(585): pid file /home/seafile/pids/elasticsearch.pid does not exist
[02/19/21 07:27:23] seafile-controller.c(633): elasticsearch need restart...
[02/19/21 07:27:23] seafile-controller.c(96): spawn_process: /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch -Epath.logs=/home/seafile/logs -Epath.data=/home/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /home/seafile/pids/elasticsearch.pid
[02/19/21 07:27:23] seafile-controller.c(130): spawned /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch, pid 2300
[02/19/21 07:27:33] seafile-controller.c(585): pid file /home/seafile/pids/elasticsearch.pid does not exist
[02/19/21 07:27:33] seafile-controller.c(633): elasticsearch need restart...
[02/19/21 07:27:33] seafile-controller.c(96): spawn_process: /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch -Epath.logs=/home/seafile/logs -Epath.data=/home/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /home/seafile/pids/elasticsearch.pid
[02/19/21 07:27:33] seafile-controller.c(130): spawned /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch, pid 2356
[02/19/21 07:27:43] seafile-controller.c(585): pid file /home/seafile/pids/elasticsearch.pid does not exist
[02/19/21 07:27:43] seafile-controller.c(633): elasticsearch need restart...
[02/19/21 07:27:43] seafile-controller.c(96): spawn_process: /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch -Epath.logs=/home/seafile/logs -Epath.data=/home/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /home/seafile/pids/elasticsearch.pid
[02/19/21 07:27:43] seafile-controller.c(130): spawned /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch, pid 2425
[02/19/21 07:27:53] seafile-controller.c(585): pid file /home/seafile/pids/elasticsearch.pid does not exist
[02/19/21 07:27:53] seafile-controller.c(633): elasticsearch need restart...
[02/19/21 07:27:53] seafile-controller.c(96): spawn_process: /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch -Epath.logs=/home/seafile/logs -Epath.data=/home/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /home/seafile/pids/elasticsearch.pid
[02/19/21 07:27:53] seafile-controller.c(130): spawned /home/seafile/seafile-pro-server-7.1.11/pro/elasticsearch/bin/elasticsearch, pid 2502

This starting behaviour is crazy, I need to be quick here because multiple Java/ES processes will kill the machine in less than 2 minutes.
Before the process w/ pid 2502 is started I executed a “killall java”, which usually helps.

If this is a timing issue (ES not starting up fast enough, so the pid file creation is delayed?) - why cant we configure the controller to check only after 30s or even 1min if a process is down? This could be a config parameter, @daniel.pan ?

daniel.pan · February 22, 2021, 3:47am

We will check the problem.