File search not working, elasticsearch wont index after deploying Seafile 9.0.7 docker

Seafile Server
,
1

I have migrated from Seafile CE to Seafile Pro 9.0.7 (both as docker), and everything works as before, except the file search function in the web. When I try to search for a file, I get the spinning wheel and an error saying “internal server error”

On my server side, I can see via sudo docker-compose ps everything seems to be up and running: seafile, seafile-elaticsearch, seafile-memcached, and seafile-mysql.

When looking at the directory of elasticsearch, I don’t see any indexes, only a weird lock.node file. Here is the dir structure:

$ sudo tree /opt/seafile-elasticsearch/
/opt/seafile-elasticsearch/
├── data
│   └── nodes
│       └── 0
│           ├── node.lock
│           └── _state
│               ├── global-0.st
│               └── node-0.st

I tried installing the dependencies with pip3 as some ppl have suggested, but nothing seems to work. The search function is still not working.

My seafile docker is behind an apache with reverse proxy and a custom port, not sure if that could be the issue?

Other info
The last log line from docker-compose reads:
recovered [0] indices into cluster_state

For some reason, whenever I start seafile, the user and group of /opt/seafile-elasticsearch/data changed to systemd-resolve:systemd-journal instead of root:root

When I run pro.py search --update, I get the following error:

$ sudo docker exec -it seafile /opt/seafile/seafile-server-latest/pro/pro.py search --update
…
[ERROR] seafes:158 start_index_local: Index process init error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x… >: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f2ab2133c40>: Failed to establish a new connection: [Errno 111] Connection refused)

The output of the controller.logs keeps repeating these lines:

2022-09-07 21:44:25 seafile-controller.c(585): pid file /opt/seafile/pids/elasticsearch.pid does not exist
2022-09-07 21:44:25 seafile-controller.c(680): elasticsearch need restart…
2022-09-07 21:44:25 seafile-controller.c(97): spawn_process: /opt/seafile/seafile-pro-server-9.0.7/pro/elasticsearch/bin/elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
2022-09-07 21:44:25 seafile-controller.c(131): spawned /opt/seafile/seafile-pro-server-9.0.7/pro/elasticsearch/bin/elasticsearch, pid 2863
2022-09-07 21:44:25 seafile-controller.c(119): failed to execvp /opt/seafile/seafile-pro-server-9.0.7/pro/elasticsearch/bin/elasticsearch

2

here is the output of pro.py search --update

$ sudo docker exec -it seafile /opt/seafile/seafile-server-latest/pro/pro.py search --update

Updating search index, this may take a while…

/opt/seafile/seafile-server-latest/seahub/thirdpart/requests/init.py:102: RequestsDependencyWarning: urllib3 (1.26.11) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn’t match a supported version!
warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn’t match a supported "
09/07/2022 19:13:45 [INFO] seafes:208 main: storage: using filesystem storage backend
09/07/2022 19:13:45 [INFO] seafes:210 main: index office pdf: True
09/07/2022 19:13:45 [ERROR] seafes:158 start_index_local: Index process init error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa4714bec40>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa4714bec40>: Failed to establish a new connection: [Errno 111] Connection refused).

@daniel.pan when you get a chance, do you have any ideas of what is going on? I have been troubleshooting this for a while and I still cannot get it to work. It looks like many users are having similar issues with elasticsearch.

3

this is the output after firing sudo docker-compose up

seafile-elasticsearch | [2022-09-08T02:18:13,424][INFO ][o.e.n.Node ] initializing …
seafile-elasticsearch | [2022-09-08T02:18:13,551][INFO ][o.e.e.NodeEnvironment ] [a5H-7Ul] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdc1)]], net usable_space [2.2tb], net total_space [3.5tb], spins? [possibly], types [ext4]
seafile-elasticsearch | [2022-09-08T02:18:13,551][INFO ][o.e.e.NodeEnvironment ] [a5H-7Ul] heap size [989.8mb], compressed ordinary object pointers [true]
seafile-elasticsearch | [2022-09-08T02:18:13,554][INFO ][o.e.n.Node ] node name [a5H-7Ul] derived from node ID [a5H-7UlARty8ab2QIje-aA]; set [node.name] to override
seafile-elasticsearch | [2022-09-08T02:18:13,554][INFO ][o.e.n.Node ] version[5.6.16], pid[1], build[3a740d1/2019-03-13T15:33:36.565Z], OS[Linux/5.4.0-125-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_212/25.212-b01]
seafile-elasticsearch | [2022-09-08T02:18:13,554][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms1g, -Xmx1g, -Des.path.home=/usr/share/elasticsearch]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [aggs-matrix-stats]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [ingest-common]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [lang-expression]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [lang-groovy]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [lang-mustache]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [lang-painless]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [parent-join]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [percolator]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [reindex]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [transport-netty3]
seafile-elasticsearch | [2022-09-08T02:18:14,125][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded module [transport-netty4]
seafile-elasticsearch | [2022-09-08T02:18:14,126][INFO ][o.e.p.PluginsService ] [a5H-7Ul] loaded plugin [analysis-ik]
seafile-elasticsearch | [2022-09-08T02:18:14,963][INFO ][o.e.d.DiscoveryModule ] [a5H-7Ul] using discovery type [zen]
seafile-elasticsearch | [2022-09-08T02:18:15,335][INFO ][o.e.n.Node ] initialized
seafile-elasticsearch | [2022-09-08T02:18:15,335][INFO ][o.e.n.Node ] [a5H-7Ul] starting …
seafile-elasticsearch | [2022-09-08T02:18:15,426][INFO ][o.e.t.TransportService ] [a5H-7Ul] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
seafile-elasticsearch | [2022-09-08T02:18:18,467][INFO ][o.e.c.s.ClusterService ] [a5H-7Ul] new_master {a5H-7Ul}{a5H-7UlARty8ab2QIje-aA}{vftNu_Y3Q56mpH51DPnhYQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
seafile-elasticsearch | [2022-09-08T02:18:18,477][INFO ][o.e.h.n.Netty4HttpServerTransport] [a5H-7Ul] publish_address {172.20.0.4:9200}, bound_addresses {0.0.0.0:9200}
seafile-elasticsearch | [2022-09-08T02:18:18,477][INFO ][o.e.n.Node ] [a5H-7Ul] started
seafile-elasticsearch | [2022-09-08T02:18:18,546][INFO ][o.e.g.GatewayService ] [a5H-7Ul] recovered [0] indices into cluster_state

4

Hi, your elasticsearch docker image version is 5.6.16, it should be version 7.16.x, like 7.16.2
You should modify the version of elasticsearch in docker-compose.yml and redeploy.

6

You are absolutely right !!! I accidentally downloaded the yml file for a previous version. I changed the lines to:

elasticsearch:
image: elasticsearch:7.16.2
and it looks like elasticsearch is now indexing! Thank you.

7

I think I celebrated too soon. It looks like the elasticsearch was beginning to index, as the initial logs had lines with “…adding index template…” and “…adding index lifecycle…”. However the search fuction still does not work and running pro.py search --update results again in the connection error 111.

sudo docker exec -it seafile /opt/seafile/seafile-server-latest/pro/pro.py search --update Updating search index, this may take a while… /opt/seafile/seafile-server-latest/seahub/thirdpart/requests/init.py:102: RequestsDependencyWarning: urllib3 (1.26.11) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn’t match a supported version! warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn’t match a supported " 09/08/2022 10:59:32 [INFO] seafes:208 main: storage: using filesystem storage backend 09/08/2022 10:59:32 [INFO] seafes:210 main: index office pdf: True 09/08/2022 10:59:32 [ERROR] seafes:158 start_index_local: Index process init error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fd630917c40>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fd630917c40>: Failed to establish a new connection: [Errno 111] Connection refused).

I waited over an hour and the elasticsearch/data/nodes/0/indices folder size has not increased (still less than 40M). Could the warning that urllib3 is not matching a supported version be the issue?

8

also the controller.log keeps writing these lines:

2022-09-08 11:16:09 seafile-controller.c(585): pid file /opt/seafile/pids/elasticsearch.pid does not exist
2022-09-08 11:16:09 seafile-controller.c(680): elasticsearch need restart…
2022-09-08 11:16:09 seafile-controller.c(97): spawn_process: /opt/seafile/seafile-pro-server-9.0.7/pro/elasticsearch/bin/elasticsearch -Epath.logs=/opt/seafile/logs -Epath.data=/opt/seafile/pro-data/search/data -Enetwork.host=127.0.0.1 -p /opt/seafile/pids/elasticsearch.pid
2022-09-08 11:16:09 seafile-controller.c(131): spawned /opt/seafile/seafile-pro-server-9.0.7/pro/elasticsearch/bin/elasticsearch, pid 3135
2022-09-08 11:16:09 seafile-controller.c(119): failed to execvp /opt/seafile/seafile-pro-server-9.0.7/pro/elasticsearch/bin/elasticsearch

what is interesting is that the path in the log : /opt/seafile/seafile-pro-server-9.0.7/pro/elasticsearch/bin/elasticsearch does not exists in the docker. I wonder if that could be the source of the error. The elasticsearch folder in the docker that I was able to find has the following path:/opt/seafile/seafile-pro-server-9.0.7/pro/python/elasticsearch, as per the command below.

root@:/opt/seafile# find ./ -iname elasticsearch
./seafile-pro-server-9.0.7/pro/python/elasticsearch

9

After troubleshooting for several hours, I was able to fix this issue. I will open an issue on github for the devs to modify the manual.

Here is the solution. For a docker deployment of Seafile, the IP of the elasticsearch container needs to be specified in the seafevents.conf file, otherwise seafile won’t be able to communicate with elasticsearch.

If you are getting “Failed to establish a new connection” error in the controller.log, and the container seafile-elasticsearch seems to be up an running, try the following steps:

1. check the IP of your elasticsearch container:

docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' seafile-elasticsearch

2. Make sure you get a good response from elasticsearch container IP address and port 9200: (e.g. IP: 172.20.0.5). If you get could not resolve host, then its likely elasticsearch container is not running.

curl -X GET 172.20.0.5:9200

3. Modify the seafevents.conf to contains the following lines. Note that the external_es_server msut be set to true, although the container is running in the same server/machine as seafile. You can enter the IP in es_host, however, the container IP will change after a reboot. So instead, add the name of the container as below:

[INDEX FILES]
interval = 1m
external_es_server = true
es_host = seafile-elasticseach
es_port = 9200

4. After that, update the index :

docker exec -it seafile /opt/seafile/seafile-server-latest/pro/pro.py search --update

If in doubt, you can clear the index, and then update as follows:

docker exec -it seafile /opt/seafile/seafile-server-latest/pro/pro.py search --clear
docker exec -it seafile /opt/seafile/seafile-server-latest/pro/pro.py search --update

Now you should see the indexing running.

1 Like