Seafile-controller somehow killed seaf-server component periodically

Hi,

Sorry to bother but I spent quite some time debugging but it doesn’t work out. I installed the latest version of Seafile Pro 6.3.8 on Debian 9 today with only one user (me). Everything works perfect except … the seafile-server periodically got killed somehow (doesn’t listen on 8082), which leads to my desktop and mobile client getting 500 internal error. Everytime I tried to restart by ./seafile.sh restart, It’s up again but got down for … I don’t know, 1 hour later. It’s not memory issue. I found this in controller.log which may be helpful:

[12/12/18 19:39:02] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/12/18 19:39:12] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/12/18 19:39:22] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/12/18 19:39:32] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/12/18 19:39:38] seafile-controller.c(790): shutting down ccnet-server ...
[12/12/18 19:39:43] seafile-controller.c(184): starting ccnet-server ...
[12/12/18 19:39:43] seafile-controller.c(88): spawn_process: ccnet-server -F /var/www/seafile/conf -c /var/www/seafile/ccnet -f /var/www/seafile/logs/ccnet.log -d -L /var/www/seafile -P /var/www/seafile/pids/ccnet.pid
[12/12/18 19:39:43] seafile-controller.c(103): spawned ccnet-server, pid 5458 
[12/12/18 19:39:44] seafile-controller.c(779): ccnet daemon connected.
[12/12/18 19:39:44] seafile-controller.c(218): starting seaf-server ...
[12/12/18 19:39:44] seafile-controller.c(88): spawn_process: seaf-server -F /var/www/seafile/conf -c /var/www/seafile/ccnet -d /var/www/seafile/seafile-data -l /var/www/seafile/logs/seafile.log -P /var/www/seafile/pids/seaf-server.pid -f -L /var/www/seafile
[12/12/18 19:39:44] seafile-controller.c(103): spawned seaf-server, pid 5460 
[12/12/18 19:39:44] seafile-controller.c(592): pid file /var/www/seafile/pids/seafevents.pid does not exist
[12/12/18 19:39:44] seafile-controller.c(88): spawn_process: /usr/bin/python2.7 -m seafevents.main --config-file /var/www/seafile/conf/seafevents.conf --logfile /var/www/seafile/logs/seafevents.log -P /var/www/seafile/pids/seafevents.pid
[12/12/18 19:39:44] seafile-controller.c(103): spawned /usr/bin/python2.7, pid 5461 
[12/12/18 19:39:44] seafile-controller.c(747): seafdav is not enabled
[12/12/18 19:39:44] seafile-controller.c(592): pid file /var/www/seafile/pids/elasticsearch.pid does not exist
[12/12/18 19:39:44] seafile-controller.c(88): spawn_process: /var/www/seafile/seafile-pro-server-6.3.8/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/var/www/seafile/logs -Des.path.data=/var/www/seafile/pro-data/search/data -Des.network.host=127.0.0.1 -Des.insecure.allow.root=true -p /var/www/seafile/pids/elasticsearch.pid
[12/12/18 19:39:44] seafile-controller.c(103): spawned /var/www/seafile/seafile-pro-server-6.3.8/pro/elasticsearch/bin/elasticsearch, pid 5462 [12/12/18 20:39:46] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/12/18 20:39:46] seafile-controller.c(617): Seafile license file is invalid. Stop Seafile server.
[12/12/18 20:39:56] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/12/18 20:40:06] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/12/18 20:40:16] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/12/18 20:40:26] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist

I can see that it’s the controller killed the seaf-server component due to license issue. But the doc says I don’t need license for free Pro version. Any suggestions?

I’m comfortable with Linux but just don’t know where I should look up to. All logs in /logs seems to be fine except for this one. Let me know if more info is required.

Thank you so much.

How many users are you using? The pro edition is free for 3 users.

Hi, thank you for your reply.

I’m only user as I stated.

I got a simple work-around… Just check if there’s a process listening on 8082 every 30s and restart the server if not. But that’s temporary solution. Any better idea?

Did you check your config files for errors? Are there any other error messages?

And how often does this happen?

Did you check your config files for errors?

I’ve checked the config and everything seems to be fine. Tell me if you need to see some specific config.

Are there any other error messages?

Yer but it should be the consequence of losing seaf-server. In ccnet.log

...
[12/13/18 14:14:40] ../common/peer.c(950): Local peer down
[12/13/18 14:14:40] ../common/peer.c(950): Local peer down
[12/13/18 14:15:41] ../common/session.c(409): Accepted a local client
[12/13/18 14:15:41] ../common/peer.c(948): libevent got an error! what=33, errno=104 (Connection reset by peer)
[12/13/18 14:15:41] ../common/peer.c(950): Local peer down
[12/13/18 14:21:41] ../common/session.c(409): Accepted a local client
[12/13/18 14:21:41] ../common/peer.c(948): libevent got an error! what=33, errno=104 (Connection reset by peer)
[12/13/18 14:21:41] ../common/peer.c(950): Local peer down
[12/13/18 14:22:29] ../common/session.c(409): Accepted a local client
[12/13/18 14:22:29] ../common/peer.c(948): libevent got an error! what=33, errno=104 (Connection reset by peer)
[12/13/18 14:22:29] ../common/peer.c(950): Local peer down
...

And how often does this happen?

After some experiment I can find that the life span of seaf-server component is 1 hour. In controller.log

[12/13/18 13:13:56] seafile-controller.c(790): shutting down ccnet-server ...
[12/13/18 13:14:39] seafile-controller.c(184): starting ccnet-server ...
[12/13/18 13:14:39] seafile-controller.c(88): spawn_process: ccnet-server -F /var/www/seafile/conf -c /var/www/seafile/ccnet -f /var/www/seafile/logs/ccnet.log -d -L /var/www/seafile -P /var/www/seafile/pids/ccnet.pid
[12/13/18 13:14:39] seafile-controller.c(103): spawned ccnet-server, pid 20933
[12/13/18 13:14:40] seafile-controller.c(779): ccnet daemon connected.
[12/13/18 13:14:40] seafile-controller.c(218): starting seaf-server ...
[12/13/18 13:14:40] seafile-controller.c(88): spawn_process: seaf-server -F /var/www/seafile/conf -c /var/www/seafile/ccnet -d /var/www/seafile/seafile-data -l /var/www/seafile/logs/seafile.log -P /var/www/seafile/pids/seaf-server.pid -f -L /var/www/seafile
[12/13/18 13:14:40] seafile-controller.c(103): spawned seaf-server, pid 20935
[12/13/18 13:14:40] seafile-controller.c(592): pid file /var/www/seafile/pids/seafevents.pid does not exist
[12/13/18 13:14:40] seafile-controller.c(88): spawn_process: /usr/bin/python2.7 -m seafevents.main --config-file /var/www/seafile/conf/seafevents.conf --logfile /var/www/seafile/logs/seafevents.log -P /var/www/seafile/pids/seafevents.pid
[12/13/18 13:14:40] seafile-controller.c(103): spawned /usr/bin/python2.7, pid 20936
[12/13/18 13:14:40] seafile-controller.c(747): seafdav is not enabled
[12/13/18 13:14:40] seafile-controller.c(592): pid file /var/www/seafile/pids/elasticsearch.pid does not exist
[12/13/18 13:14:40] seafile-controller.c(88): spawn_process: /var/www/seafile/seafile-pro-server-6.3.8/pro/elasticsearch/bin/elasticsearch -Des.path.logs=/var/www/seafile/logs -Des.path.data=/var/www/seafile/pro-data/search/data -Des.network.host=127.0.0.1 -Des.insecure.allow.root=true -p /var/www/seafile/pids/elasticsearch.pid
[12/13/18 13:14:40] seafile-controller.c(103): spawned /var/www/seafile/seafile-pro-server-6.3.8/pro/elasticsearch/bin/elasticsearch, pid 20937
[12/13/18 14:14:43] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/13/18 14:14:43] seafile-controller.c(617): Seafile license file is invalid. Stop Seafile server.
[12/13/18 14:14:53] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/13/18 14:15:03] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
[12/13/18 14:15:13] seafile-controller.c(592): pid file /var/www/seafile/pids/seaf-server.pid does not exist
...

You can see that I restart the whole thing at 13:14 but the seafile server is killed at 14:14. I don’t specific settings related to “1 hour” so I guess it’s internal stuff.
And this issue always happens after a fresh installation (yesterday).

:frowning: I don’t really know what I can do.

I am also having this issue. I can see 6.3.8 is retracted. I guess it will be fixed in the next release soon.

1 Like

Oh my godness I didn’t notice 6.3.8 download tarball is gone. Thank you for the remind. Anyway I can rollback to 6.3.7?

It could be confirmed that 6.3.8 is unstable and its tar file is retracted in official source. Best solution is to wait for the next update.

Temporary solution is to use external script to check the status of seaf-server process and restart it if it fails:

#/bin/bash                                                                                                                                     

set -euo pipefail

while true; do
    if ! netstat -l4nt | grep 8082 &>/dev/null; then
        echo "service down at $(date)"
        sudo -u seafile ./seafile.sh restart &> /dev/null
    else
        sleep 30s 
    fi  
done

Update: I upgrade to 6.3.9 and this issue is gone:)

1 Like

Following this post to rollback.

Thanks for this Information!

And yes you could rollback to 6.3.7 - just stop Seafile and download the old Version and run the Minor Upgrade Script (minor-upgrade.sh).