Upgraded to Seafile Pro, now elasticsearch doesn't work

bhugh · May 19, 2024, 9:04am

Since upgrading from Seafile Community server to Seafile pro, my elasticsearch has failed to work since the upgrade.

It appears to be indexing (the elasticsearch directory is frequently updated with something and is about 1 gig in size). However when you type anything into the search bar, the result is inevitably “No results matching” or “No result”.

I can connect to the elasticsearch server using curl although all searches I have tried there are also blank (I don’t understand the syntax or search terms very well, though).

Part of the problem I’m having is I can’t figure out how to get any reasonable error message from the logs.

However: I thought maybe resetting the search index and starting indexing over would be a good idea.

I ran pro.py search --clear and you can see the results below - some really crazy error messages.

Does anyone have any ideas about how to address this?

 docker exec -it seafile /opt/seafile/seafile-server-latest/pro/pro.py search --clear
Delete seafile search index ([y]/n)? y

Delete search index, this may take a while...

05/19/2024 03:51:41 [INFO] seafes:210 main: storage: using filesystem storage backend
05/19/2024 03:51:41 [INFO] seafes:212 main: index office pdf: True
05/19/2024 03:51:41 [INFO] elastic_transport.transport:359 perform_request: HEAD http://127.0.0.1:9200/repo_head [status:N/A duration:0.001s]
05/19/2024 03:51:41 [WARNING] elastic_transport.node_pool:246 mark_dead: Node <Urllib3HttpNode(http://127.0.0.1:9200)> has failed for 1 times in a row, putting on 1 second timeout
05/19/2024 03:51:41 [WARNING] elastic_transport.transport:396 perform_request: Retrying request after failure (attempt 0 of 3)
Traceback (most recent call last):
  File "/opt/seafile/seafile-server-latest/pro/python/elastic_transport/_transport.py", line 329, in perform_request
    meta, raw_data = node.perform_request(
  File "/opt/seafile/seafile-server-latest/pro/python/elastic_transport/_node/_http_urllib3.py", line 199, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x782070ef0df0>: Failed to establish a new connection: [Errno 111] Connection refused)
05/19/2024 03:51:41 [INFO] elastic_transport.transport:359 perform_request: HEAD http://127.0.0.1:9200/repo_head [status:N/A duration:0.001s]
05/19/2024 03:51:41 [WARNING] elastic_transport.node_pool:246 mark_dead: Node <Urllib3HttpNode(http://127.0.0.1:9200)> has failed for 2 times in a row, putting on 2 second timeout
05/19/2024 03:51:41 [WARNING] elastic_transport.transport:396 perform_request: Retrying request after failure (attempt 1 of 3)
Traceback (most recent call last):
  File "/opt/seafile/seafile-server-latest/pro/python/elastic_transport/_transport.py", line 329, in perform_request
    meta, raw_data = node.perform_request(
  File "/opt/seafile/seafile-server-latest/pro/python/elastic_transport/_node/_http_urllib3.py", line 199, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x782070e82250>: Failed to establish a new connection: [Errno 111] Connection refused)
05/19/2024 03:51:41 [INFO] elastic_transport.transport:359 perform_request: HEAD http://127.0.0.1:9200/repo_head [status:N/A duration:0.000s]
05/19/2024 03:51:41 [WARNING] elastic_transport.node_pool:246 mark_dead: Node <Urllib3HttpNode(http://127.0.0.1:9200)> has failed for 3 times in a row, putting on 4 second timeout
05/19/2024 03:51:41 [WARNING] elastic_transport.transport:396 perform_request: Retrying request after failure (attempt 2 of 3)
Traceback (most recent call last):
  File "/opt/seafile/seafile-server-latest/pro/python/elastic_transport/_transport.py", line 329, in perform_request
    meta, raw_data = node.perform_request(
  File "/opt/seafile/seafile-server-latest/pro/python/elastic_transport/_node/_http_urllib3.py", line 199, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x782070e82430>: Failed to establish a new connection: [Errno 111] Connection refused)
05/19/2024 03:51:41 [INFO] elastic_transport.transport:359 perform_request: HEAD http://127.0.0.1:9200/repo_head [status:N/A duration:0.000s]
05/19/2024 03:51:41 [WARNING] elastic_transport.node_pool:246 mark_dead: Node <Urllib3HttpNode(http://127.0.0.1:9200)> has failed for 4 times in a row, putting on 8 second timeout
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/seafile/seafile-server-latest/pro/python/seafes/index_local.py", line 264, in <module>
    main()
  File "/opt/seafile/seafile-server-latest/pro/python/seafes/index_local.py", line 214, in main
    args.func()
  File "/opt/seafile/seafile-server-latest/pro/python/seafes/index_local.py", line 174, in delete_indices
    if es.indices.exists(index=idx):
  File "/opt/seafile/seafile-server-latest/pro/python/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "/opt/seafile/seafile-server-latest/pro/python/elasticsearch/_sync/client/indices.py", line 1111, in exists
    return self.perform_request(  # type: ignore[return-value]
  File "/opt/seafile/seafile-server-latest/pro/python/elasticsearch/_sync/client/_base.py", line 390, in perform_request
    return self._client.perform_request(
  File "/opt/seafile/seafile-server-latest/pro/python/elasticsearch/_sync/client/_base.py", line 286, in perform_request
    meta, resp_body = self.transport.perform_request(
  File "/opt/seafile/seafile-server-latest/pro/python/elastic_transport/_transport.py", line 329, in perform_request
    meta, raw_data = node.perform_request(
  File "/opt/seafile/seafile-server-latest/pro/python/elastic_transport/_node/_http_urllib3.py", line 199, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x782070ef0df0>: Failed to establish a new connection: [Errno 111] Connection refused))

daniel.pan · May 20, 2024, 12:57am

It seems that Seafile server can not connect to ElasticSearch.

If you deploy Seafile pro edition and elasticsearch both with docker, you should use the internal docker network name in seafevents.conf for the address of elasticsearch. Can you post your seafevent.conf and docker-compose.yml contents?

bhugh · May 20, 2024, 9:57am

Aha, that makes sense. With that clue, I tried some different variations for es_host and finally got it working!

See the docker-compose.yml & seafevents.conf files below.

Here are the combinations for es_host that I tried - and what worked and did not work:

es_host = seafile_default  ## DOESN"T WORK ##
es_host = 10.10.10.6   ## WORKS!!! ##
es_host = seafile-elasticsearch   ## DOESN"T WORK ##
es_host = elasticsearch   ## WORKS!!! ##

seafile_default is the name of the network. seafile-elasticsearch is the “container_name” as defined in docker-compose.yml. This is what I had thought the correct es_host should be.

Either the IP address I had specified for the container (10.10.10.6) OR the name of the elasticsearch service as defined in the docker-compose.yml (elasticsearch) work properly.

Note one thing that threw me off is that the elasticsearch contained seems to take a while longer to start up than seafile. When I initially tried using the ip address, I got no results. But I don’t think I waited long enough for the elasticsearch container to start up.

Just FYI, here are my relevant files:

docker-compose.yml:

networks:
  seafile_default:
    ipam:
          driver: default
          config:
              - subnet: "10.10.10.0/24"
                gateway: "10.10.10.1"

services:
  db:
    image: mariadb:10.11
    container_name: seafile-mysql
    restart: always
    environment:
      - MYSQL_ROOT_PASSWORD=my_pw  # Requested, set the root's password of MySQL service.
      - MYSQL_LOG_CONSOLE=true
      - MARIADB_AUTO_UPGRADE=1
    volumes:
      - ./seafile-mysql2/db:/var/lib/mysql  # Requested, specifies the path to MySQL data persistent store.
      - /mnt/seafile_main_data_store_on_W/seafile_db_dumps:/mnt/seafile_db_dumps  # Requested, specifies the path to Elasticsearch data persistent store.
      - /home/mydir/seafile:/mnt/config
    networks:
      seafile_default:
        ipv4_address: 10.10.10.2 

  memcached:
    image: memcached:1.6.18
    container_name: seafile-memcached
    restart: always
    entrypoint: memcached -m 256
    networks:
      seafile_default:
        ipv4_address: 10.10.10.3
          
  elasticsearch:
    image: elasticsearch:8.6.2
    container_name: seafile-elasticsearch
    restart: always
    ports:
      - 9200:9200   # 192.x.x.x is the IP address of the machine
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 4g
    volumes:
     
     - /mnt/seafile_main_data_store_on_W/elasticsearch/data:/usr/share/elasticsearch/data  # Requested, specifies the path to Elasticsearch data persistent store.
    networks:
      seafile_default:
        ipv4_address: 10.10.10.6


  seafile:
    #image: docker.seadrive.org/seafileltd/seafile-pro-mc:latest 
    #find latest version # here: https://manual.seafile.com/changelog/changelog-for-seafile-professional-server/
    image: docker.seadrive.org/seafileltd/seafile-pro-mc:10.0.15
    container_name: seafile

    restart: always
    privileged: true
    ports:
      # NO PORTS NEEDED because we have our external networks & hit them directly from CADDY
      #- "180:80" #Windows uses port 80 for WebDAV services etc, so use something else
      #- "1443:443"  # If https is enabled, cancel the comment.
      - "8000:8000" #backup access port
      
    volumes:
      - /mnt/seafile_main_data_store_on_W:/shared   # Requested, specifies the path to Seafile data persistent store. NEW location on W drive (2024/01/14)
      - type: bind
        source: /mnt/seafile-archive/
        target: /seafile-fuse/
        bind:
          propagation: rshared
    environment:
      - DB_HOST=db
      - DB_ROOT_PASSWD=My_pw  # Requested, the value should be root's password of MySQL service.
      - TIME_ZONE=America/Chicago  # Optional, default is UTC. Should be uncomment and set to your local time zone.
      - SEAFILE_ADMIN_EMAIL=my@email.com # Specifies Seafile admin user, default is 'me@example.com'.
      - SEAFILE_ADMIN_PASSWORD=my_PW     # Specifies Seafile admin password, default is 'asecret'.
      - SEAFILE_SERVER_LETSENCRYPT=false   # Whether to use https or not.
      - SEAFILE_SERVER_HOSTNAME=sf.myhost.org # Specifies your host name if https is enabled.
    depends_on:
      - db
      - memcached
      - elasticsearch
    networks:
      seafile_default:
        ipv4_address: 10.10.10.4

seafevents.conf:

[DATABASE]
type = mysql
host = db
port = 3306
username = seafile
password = my_pw
name = seahub_db



[AUDIT]
enabled = true

[INDEX FILES]
external_es_server = true
## es_host = seafile_default  ## DOESN"T WORK ##
## es_host = 10.10.10.6   ## WORKS!!! ##
## es_host = seafile-elasticsearch   ## DOESN"T WORK ##
es_host = elasticsearch   ## WORKS!!! ##
es_port = 9200
enabled = true
interval = 10m

highlight = fvh

## If true, indexes the contents of office/pdf files while updating search index
## Note: If you change this option from "false" to "true", then you need to clear the search index and update the index again. See the FAQ for details.
index_office_pdf = true

[SEAHUB EMAIL]
enabled = true

## interval of sending Seahub email. Can be s(seconds), m(minutes), h(hours), d(days)
interval = 30m

# Enable statistics
[STATISTICS]
enabled=true

[OFFICE CONVERTER]
port = 8089
host = https://op.myhost.org/
enabled = true
workers = 1