Elasticsearch not running after upgrade to 7.0.8 (from 6.2.4)

Seafile Server
1

Hi,

elasticsearch is respawning constantly and stopping with the following log messages:

[3] bootstrap checks failed
[1]: initial heap size [536870912] not equal to maximum heap size [1073741824]; this can cause resize pauses and prevents mlockall from locking the entire heap
[2]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

any ideas how I can adjust those settings? or am I doing something wrong? at least I needed no special settings for seafile 6.2.4

best,
Hp

2

Hi hkunz,
I also struggled with elasticsearch problems after an upgrade. In the end these warnings of elasticsearch where not the real reason for the problems. I had some java processes that were not killed and made some problems.

Can you try to stop seafile and seahub. Kill all java processes and then restart seafile and seahub?
Best regards
Christoph

3

hi christoph,

yes, I think I read the thread mentioning old java processes. In my case this cannot be the case, the node was rebooted (just to be sure). the problem, unfortunately, persisted.

best,
hp

4

Hi hp,

can you please try the following:

  • stop seafile and seahub
  • kill all java and python processes
  • update all your python packages. Either by pip install [packagename] -U or with pip-review --interactive
  • remove all elasticsearch node locks. There are two paths: (paths can be different at your server)
    rm -r ./seafile/pro-data/search/data/nodes/0
    rm -r ./seafile/seafile-server-latest/pro/elasticsearch/data/nodes/0/
  • remove seahub cache in /tmp
  • restart seafile and seahub

Then please check your log files in seafile/logs.
I am sure we will fix this issue.
Best regards
Christoph

5

Hi christoph,

thanks, I performed almost all the steps you suggested. most of my python packages are installed by debian. so I did not upgrade those with pip (however I am sure it is not a problem of python, elasticsearch is java). I removed the whole elasticsearch search folder (although I tried that before). and cleared /tmp. and rebooted the machine.

The problem is still the same. elasticsearch refuses to start (bootstrap checks failed): here is the complete log of a elasticsearch startup:

[2019-09-03T13:56:53,109][INFO ][o.e.n.Node ] [] initializing …
[2019-09-03T13:56:53,371][INFO ][o.e.e.NodeEnvironment ] [oJ32maZ] using [1] data
paths, mounts [[/srv/seafile (/dev/mapper/melinda-seafile)]], net usable_space [20.
6gb], net total_space [22gb], spins? [possibly], types [btrfs]
[2019-09-03T13:56:53,372][INFO ][o.e.e.NodeEnvironment ] [oJ32maZ] heap size [100
7.3mb], compressed ordinary object pointers [true]
[2019-09-03T13:56:53,377][INFO ][o.e.n.Node ] node name [oJ32maZ] deri
ved from node ID [oJ32maZzSDKphL-09LG5Ug]; set [node.name] to override
[2019-09-03T13:56:53,378][INFO ][o.e.n.Node ] version[5.6.13], pid[374
1], build[4d5320b/2018-10-30T19:05:08.237Z], OS[Linux/3.16.0-10-amd64/amd64], JVM[Or
acle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b11]
[2019-09-03T13:56:53,379][INFO ][o.e.n.Node ] JVM arguments [-Xms512m,
-Xmx1g, -XX:-AssumeMP, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=
75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.head
less=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonical
Path=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.nett
y.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/srv/seafile/seafile-pro-server-7.0.8/pro/elasticsearch]
[2019-09-03T13:56:55,551][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [aggs-matrix-stats]
[2019-09-03T13:56:55,552][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [ingest-common]
[2019-09-03T13:56:55,553][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [lang-expression]
[2019-09-03T13:56:55,554][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [lang-groovy]
[2019-09-03T13:56:55,554][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [lang-mustache]
[2019-09-03T13:56:55,554][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [lang-painless]
[2019-09-03T13:56:55,555][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [parent-join]
[2019-09-03T13:56:55,555][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [percolator]
[2019-09-03T13:56:55,555][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [reindex]
[2019-09-03T13:56:55,556][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [transport-netty3]
[2019-09-03T13:56:55,556][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded module [transport-netty4]
[2019-09-03T13:56:55,557][INFO ][o.e.p.PluginsService ] [oJ32maZ] loaded plugin [analysis-ik]
[2019-09-03T13:57:00,219][INFO ][o.e.d.DiscoveryModule ] [oJ32maZ] using discovery type [zen]
[2019-09-03T13:57:01,581][INFO ][o.e.n.Node ] initialized
[2019-09-03T13:57:01,582][INFO ][o.e.n.Node ] [oJ32maZ] starting …
[2019-09-03T13:57:01,992][INFO ][o.e.t.TransportService ] [oJ32maZ] publish_address {130.60.155.220:9300}, bound_addresses {[::]:9300}
[2019-09-03T13:57:02,030][INFO ][o.e.b.BootstrapChecks ] [oJ32maZ] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-09-03T13:57:02,040][ERROR][o.e.b.Bootstrap ] [oJ32maZ] node validation exception
[3] bootstrap checks failed
[1]: initial heap size [536870912] not equal to maximum heap size [1073741824]; this can cause resize pauses and prevents mlockall from locking the entire heap
[2]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-09-03T13:57:02,057][INFO ][o.e.n.Node ] [oJ32maZ] stopping …
[2019-09-03T13:57:02,107][INFO ][o.e.n.Node ] [oJ32maZ] stopped
[2019-09-03T13:57:02,107][INFO ][o.e.n.Node ] [oJ32maZ] closing …
[2019-09-03T13:57:02,147][INFO ][o.e.n.Node ] [oJ32maZ] closed

according to the info I can find on the net (I have little knowledge of elasticsearch) the problem are the failed bootstrap checks.

so, the question here is, why is elasticsearch failing on my node. as far as I know, I did not touch the elasticsearch config. they are part of the seafile-server tar file, right?

Many thanks,
Hp

6

update: I was able to get rid of the first failed bootstrap check

[1]: initial heap size [536870912] not equal to maximum heap size [1073741824]; this can cause resize pauses and prevents mlockall from locking the entire heap

by setting

-Xms1g
-Xmx1g

seafile-server-latest/pro/elasticsearch/config/jvm.options

original setting was:

-Xms512m
-Xmx1g

7

update 2: second boostrap check faillure

[2]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

solved by adding

LimitNOFILE=65536

to the systemd services for seafile

8

update 3: finally, 3rd failed bootstrap check

fixed by setting

vm.max_map_count=262144

in /etc/sysctl.conf.

so, elasticsearch is now running.

the question that remains is, why I have to changes these settings in the first place (update of elasticsearch?) and why I did not find anything on this in the documentation.

best,
hp