Seafile docker stopped working, Nginx errors though only using http

I noticed that my Seafile today is not accessible. Agents are not syncing, and the web portal giving me a 502 bad gateway error. Looking at the docker logs I have what is shown below. I noticed a lot of NGINX errors though my Seafile is configured for HTTP access and not HTTPS internally since I use a NGINX proxy manger separately for external access so I’m not sure why that is coming up here. Any suggestions how to troubleshoot things? Thank you.

nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:49 [emerg] 431#431: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:50 [emerg] 432#432: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:51 [emerg] 433#433: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:52 [emerg] 434#434: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:53 [emerg] 435#435: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:54 [emerg] 436#436: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:55 [emerg] 437#437: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:56 [emerg] 438#438: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:57 [emerg] 439#439: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:58 [emerg] 440#440: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:31:59 [emerg] 441#441: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:32:00 [emerg] 442#442: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:32:01 [emerg] 443#443: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:32:02 [emerg] 444#444: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (40: Too many levels of symbolic links)
2023/03/07 18:32:03 [emerg] 445#445: open() "/var/log/nginx/access.log" failed (40: Too many levels of symbolic links)
*** Shutting down /scripts/enterpoint.sh (PID 17)...
*** Shutting down runit daemon (PID 16)...
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown...
*** Init system aborted.
*** Killing all processes...
*** Running /etc/my_init.d/01_create_data_links.sh...
*** Booting runit daemon...
*** Runit started as PID 24
*** Running /scripts/enterpoint.sh...
2023-03-08 07:16:12 Waiting Nginx 
2023-03-08 07:16:12 Nginx ready 
2023-03-08 07:16:12 This is an idle script (infinite loop) to keep container running. 
Starting seafile server, please wait ...
** Message: 07:16:14.939: seafile-controller.c(621): No seafevents.
Seafile server started
Done.
Starting seahub at port 8000 ...
Seahub is started
Done.
*** Shutting down /scripts/enterpoint.sh (PID 25)...
*** Shutting down runit daemon (PID 24)...
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown...
*** Init system aborted.
*** Killing all processes...
*** Running /etc/my_init.d/01_create_data_links.sh...
*** Booting runit daemon...
*** Runit started as PID 23
*** Running /scripts/enterpoint.sh...
2023-03-14 18:43:06 Nginx ready 
2023-03-14 18:43:06 This is an idle script (infinite loop) to keep container running. 
nginx: [error] open() "/run/nginx.pid" failed (2: No such file or directory)
Traceback (most recent call last):
  File "/scripts/start.py", line 92, in <module>
    main()
  File "/scripts/start.py", line 53, in main
    call('nginx -s reload')
  File "/scripts/utils.py", line 70, in call
    return subprocess.check_call(*a, **kw)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'nginx -s reload' returned non-zero exit status 1.
*** Shutting down /scripts/enterpoint.sh (PID 24)...
*** Shutting down runit daemon (PID 23)...
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown...
*** Init system aborted.
*** Killing all processes...
*** Running /etc/my_init.d/01_create_data_links.sh...
*** Booting runit daemon...
*** Runit started as PID 24
*** Running /scripts/enterpoint.sh...
2023-03-18 13:44:13 Nginx ready 
2023-03-18 13:44:13 This is an idle script (infinite loop) to keep container running. 
nginx: [error] open() "/run/nginx.pid" failed (2: No such file or directory)
Traceback (most recent call last):
  File "/scripts/start.py", line 92, in <module>
    main()
  File "/scripts/start.py", line 53, in main
    call('nginx -s reload')
  File "/scripts/utils.py", line 70, in call
    return subprocess.check_call(*a, **kw)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'nginx -s reload' returned non-zero exit status 1.

I ran into the same errors when I rebooted my server.

The cause may vary but in my case it was because the /shared directory of my Seafile container is mounted to an NFS mount on my host, and the NFS mount either failed to mount or the container started before the mount was ready. So the /shared directory pointed to a regular empty or mostly-empty directory which didn’t have the directory structure that Seafile expected when it was previously initialized.

I’m not sure how peculiar my setup is with using NFS but I’m going to add my solution because this is the first Google result for the problem.

My docker-compose.yml looked like this:

services:
  db:
    image: mariadb:10.11
    container_name: seafile-mysql
    # ...

  memcached:
    image: memcached:1.6.18
    container_name: seafile-memcached
    # ...
          
  seafile:
    image: seafileltd/seafile-mc:11.0-latest
    container_name: seafile
    ports:
      - "80:80"
#     - "443:443"  # If https is enabled, cancel the comment.
    volumes:
      - /opt/seafile-data:/shared

My /etc/fstab looked like this:

192.168.1.15:/mnt/seafile /opt/seafile-data nfs auto,noexec

The first part of the fix is to add bg option to /etc/fstab so the NFS mount retries until successful:

192.168.1.15:/mnt/seafile /opt/seafile-data nfs auto,noexec,bg

Then remount:

$ sudo mount -a
$ sudo systemctl daemon-reload

The second part of the fix is to make the Seafile Docker compose service depend on the NFS mount being mounted. Put this script in /opt/seafile/check_nfs_mount.sh:

#!/bin/sh

# Check if an argument is provided
if [ $# -eq 0 ]; then
    echo "Error: No mount point specified"
    echo "Usage: $0 <mount_point>"
    exit 2
fi

MOUNT_POINT="$1"

# Use stat to check if the mount point is a mount point and if it's an NFS filesystem
FSTYPE=$(stat -f -c %T "$MOUNT_POINT")

if [ "$FSTYPE" = "nfs" ]; then
    echo "NFS mount at $MOUNT_POINT is available"
    exit 0
elif [ "$FSTYPE" = "nullfs" ]; then
    echo "Mount at $MOUNT_POINT is a bind mount, not an NFS mount"
    exit 1
else
    echo "Mount at $MOUNT_POINT is not an NFS mount (type: $FSTYPE)"
    exit 1
fi

and make it executable:

$ sudo chmod a+x /opt/seafile/check_nfs_mount.sh

Modify docker-compose.yml to add the new service and dependency:

services:
  db:
    image: mariadb:10.11
    container_name: seafile-mysql
    # ...

  memcached:
    image: memcached:1.6.18
    container_name: seafile-memcached
    # ...
          
  seafile:
    image: seafileltd/seafile-mc:11.0-latest
    container_name: seafile
    depends_on:
      db:
        condition: service_started
      memcached:
        condition: service_started
      seafile_mount_checker:
        condition: service_healthy
    ports:
      - "80:80"
#     - "443:443"  # If https is enabled, cancel the comment.
    volumes:
      - seafile_data:/shared

  seafile_mount_checker:
    image: alpine:latest
    volumes:
      - /opt/seafile/check_nfs_mount.sh:/check_nfs_mount.sh:ro
      - seafile_data:/mnt/check:ro
    command: sh -c "while true; do /check_nfs_mount.sh /mnt/check || exit 1; sleep 10; done"
    healthcheck:
      test: ["CMD-SHELL", "/check_nfs_mount.sh /mnt/check"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 0s
    networks:
      - seafile-net
    restart: always

networks:
  seafile-net:

volumes:
  seafile_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /opt/seafile-data

This includes turning the volume into a named volume so the host mount reference can’t get mismatched between the two services (e.g. a umount issued when one service is up and the other is down).

Finally, restart Docker compose to pick up the changes:

$ docker compose up -d

Now whenever I restart my server, Seafile always comes up as I expect without the strange symlink error messages.