Elastic Search Indexing Fails / Keeps restarting


#1

from index.log

10/15/2018 14:31:20 [ERROR] seafes:160 start_index_local: Index process init error: 
ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f731e8d6550>: Failed to establish a new connection: 
[Errno 111] Connection refused) caused by: 
NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f731e8d6550>: 
Failed to establish a new connection: [Errno 111] Connection refused).

I can run /pro/pro.py search --update perfectly fine, it indexes all files

I can also see the java process running on port 9200, I can also do a search in seahub, it’s just… its index is never updated automatically

Part of my controller.log:

[10/15/18 14:27:46] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 27981
[10/15/18 14:27:56] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:27:57] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:27:58] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:27:58] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 27983
[10/15/18 14:28:13] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:28:17] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:28:19] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:28:20] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 27994
[10/15/18 14:28:24] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:28:26] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:28:26] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:28:26] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 28000
[10/15/18 14:28:40] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:28:48] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:28:50] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:28:51] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 28008
[10/15/18 14:28:51] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:28:51] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:28:51] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:28:51] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 28009
[10/15/18 14:29:01] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:29:01] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:29:01] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:29:01] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 29679
[10/15/18 14:29:11] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:29:11] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:29:11] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:29:11] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 29938
[10/15/18 14:29:23] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:29:24] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:29:25] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:29:25] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 30027
[10/15/18 14:29:39] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:29:40] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:29:40] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:29:41] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 30038
[10/15/18 14:29:48] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:29:57] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:29:59] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:29:59] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 30042
[10/15/18 14:30:00] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:30:00] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:30:00] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:30:00] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 30046
[10/15/18 14:30:09] seafile-controller.c(590): pid file /seafile/pids/elasticsearch.pid does not exist
[10/15/18 14:30:09] seafile-controller.c(628): elasticsearch need restart...
[10/15/18 14:30:09] seafile-controller.c(87): spawn_process: /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/seafile/logs -Des.path.data=/seafile/pro-data/search/data -Des.network.host=0.0.0.0 -Des.insecure.allow.root=true -p /seafile/pids/elasticsearch.pid
[10/15/18 14:30:09] seafile-controller.c(102): spawned /seafile/seafile-pro-server-6.3.6/pro/elasticsearch/bin/elasticsearch, pid 31139

elastisearch.pid is in the folder as we speak. For some reason it keeps restarting, but why…?

Could it be that the the java process isn’t started quick enough so it thinks it is not running and restarts and restarts and restarts and restarts?

EDIT: it seems to be working now although I didn’t change anything. Is this a RAM issue or something similar? If so, maybe the controller should be a little less aggressive in restarting so the machine has some time to get things going…


#2

You can check the permissions, e.g. ensure all the files in /seafile/ is owned by the user you use to start seafile server.


#3

For some reason the elastic search process stopped last evening after the server experienced some high load. It never recovered. I rebooted the server this morning and it started working again. The recovery script (if there is one, but I think there is) doesn’t really does its job correctly it seems :slight_smile:

Is there a manual way to start the elastic search process for seafile without having to reboot?


#4

How many RAM does your server have?


#5

1 gig, which is probably not enough, but it is just a cheap test setup to see if it works at all before scaling up its capacity. It is however, a dedicated 3rd server just for the background tasks of seafile. And it does work, it returns search results and files are being indexed.

The question remains, is there a command to start/stop/restart the elasticsearch process seafile relies on? If so, it also allows me to setup monitoring on that service.


#6

1GB RAM is far from enough for a production java process like elasticsearch. You need at least 2GB.


#7

Hi @lins05,

same problem here and same messages in controller.log. 8 GB RAM. What else could I check?

Thanks and best regards,
Jochen


#8

I found somewhat of a workaround to at least keep things running:

As soon as port 9200 is no longer occupied I restart the seafile process which starts elastic search again. This way the setup at leasts recovers after problems. The controller script definitely has problems now, which should be fixed on seafile’s end as far as I can see.


#9

How do you check for port 9200?
[Edit]: Ah, you’re using an external elasticsearch server!? I’m using an internal one…[/Edit]
Restarting seafile doesn’t restart elasticsearch here…?
There is no java process… Should there be one?


#10

I’m using the pro version, which allows the search function.

Please check the seafile manual about enabling search.

You can check if a port is occupied by running netstat -tulpn | grep 9200 which will return nothing or a line of text with the process. You can check the exit code for 0 or 1

If this all doesn’t make sense you could dive into bash programming to familiarize yourself with exit codes.

Please note, I have monitoring software running which triggers the seafile process restart, you’ll need to create something that works for you.

Also, if the seafile startup doesn’t trigger elastic search and java I don’t know if any of this applies.

Also, this is a workaround and shouldn’t be necessary, at best we should do nothing and wait for a seafile update that fixes their elasticsearch restart script.


#11

Thanks for your answer.

I’m also running the pro version and search function had worked before. Can’t exactly tell, when it crashed.

There is no process on port 9200 but I#m wondering whether there should be any java process beacause

ps aux |grep java (or grep jdk) doesn’t return anything…!?


#12

Elasticsearch is java based, so yes, at least my server shows a java process. Mmaybe try restarting the server


#13

Have rebooted several times. No effect…


#14

I’m pushing this thread up because i had exactly the same problem as @MrMoronIV

My solution was that i had the wrong JRE installed. After installing openjdk-8-jre from the repository it works like a charm again.

Maybe this helps some people in the future .