SearpcError: Error received: 511 Unknown service (In _start_service)

Hey there,

we have seen several error messages like SearpcError: Error received: 511 Unknown service (In _start_service) in our logs.

It occured at the URLs /api2/account/info/ and /api2/repos/.

One stack trace pointed to

File "seahub/api2/views.py", line 425, in get
  ret_corrupted=True)

all the other once occured at

File "seahub/api2/views.py", line 267, in get
  info['total'] = seafile_api.get_user_quota(email)

Are these messages indicating an error in our setup or is it something that needs to be fixed in the code?
I can provide you with the more detailed stacktraces, if you wish.

Regards,
Moritz,
Johannes Gutenberg University Mainz

1 Like

Hi Moritz,

The message " SearpcError: Error received: 511 Unknown service (In _start_service)" means the daemon seaf-server was down at that time. Can you check what happens to seaf-server at that time?

Regards,
Daniel Pan

Hi Daniel,

yes, you are right. Apparently, seaf-server crashed several times in rapid succession at that time:

dmesg:

[Sun Nov 20 22:32:39 2016] seaf-server[4019]: segfault at 0 ip 00007fe9ba147e38 sp 00007fe9b2654c78 error 4 in libglib-2.0.so.0[7fe9ba0d6000+120000]
[Sun Nov 20 22:33:27 2016] seaf-server[12041]: segfault at 0 ip 00007f3371410e38 sp 00007f336991dc78 error 4 in libglib-2.0.so.0[7f337139f000+120000]
[Sun Nov 20 22:57:18 2016] seaf-server[12137]: segfault at 0 ip 00007f33637ede38 sp 00007f335bcfac78 error 4 in libglib-2.0.so.0[7f336377c000+120000]
[Sun Nov 20 22:57:59 2016] seaf-server[15711]: segfault at 0 ip 00007f437f689e38 sp 00007f4377b96c78 error 4 in libglib-2.0.so.0[7f437f618000+120000]

controller.log:

[11/20/16 22:32:41] seafile-controller.c(595): seaf-server need restart...
[11/20/16 22:32:41] seafile-controller.c(205): starting seaf-server ...
[11/20/16 22:32:41] seafile-controller.c(87): spawn_process: seaf-server -F /data/seafile/conf -c /data/seafile/ccnet -d /data/seafile_data -l /data/seafile/logs/seafile.log -P /data/seafile/pids/seaf-server.pid
[11/20/16 22:32:41] seafile-controller.c(102): spawned seaf-server, pid 12036
[11/20/16 22:33:31] seafile-controller.c(595): seaf-server need restart...
[11/20/16 22:33:31] seafile-controller.c(205): starting seaf-server ...
[11/20/16 22:33:31] seafile-controller.c(87): spawn_process: seaf-server -F /data/seafile/conf -c /data/seafile/ccnet -d /data/seafile_data -l /data/seafile/logs/seafile.log -P /data/seafile/pids/seaf-server.pid
[11/20/16 22:33:31] seafile-controller.c(102): spawned seaf-server, pid 12133
[11/20/16 22:57:23] seafile-controller.c(595): seaf-server need restart...
[11/20/16 22:57:23] seafile-controller.c(205): starting seaf-server ...
[11/20/16 22:57:23] seafile-controller.c(87): spawn_process: seaf-server -F /data/seafile/conf -c /data/seafile/ccnet -d /data/seafile_data -l /data/seafile/logs/seafile.log -P /data/seafile/pids/seaf-server.pid
[11/20/16 22:57:23] seafile-controller.c(102): spawned seaf-server, pid 15707

seafile.log:

[11/20/2016 09:32:49 PM] filelock-mgr.c(947): 0 expired file locks are unlocked.
[11/20/2016 10:32:42 PM] ../common/mq-mgr.c(61): [mq client] mq cilent is started
[11/20/2016 10:32:42 PM] listen-mgr.c(127): listen on port 12001 for block tranfer
[11/20/2016 10:32:42 PM] filelock-mgr.c(917): Cleaning expired file locks.
[11/20/2016 10:32:42 PM] filelock-mgr.c(947): 1 expired file locks are unlocked.
[11/20/2016 10:33:32 PM] ../common/mq-mgr.c(61): [mq client] mq cilent is started
[11/20/2016 10:33:32 PM] listen-mgr.c(127): listen on port 12001 for block tranfer
[11/20/2016 10:33:32 PM] filelock-mgr.c(917): Cleaning expired file locks.
[11/20/2016 10:33:32 PM] filelock-mgr.c(947): 0 expired file locks are unlocked.
[11/20/2016 10:57:23 PM] ../common/mq-mgr.c(61): [mq client] mq cilent is started

But I don’t find any further information on why this might have happened…

Regards,
Moritz

1 Like

same issue

File “/opt/haiwen/seafile-pro-server-6.2.9/seafile/lib/python2.7/site-packages/ccnet/rpc.py”, line 75, in call_remote_func_sync
req_id = self._start_service(client)
File “/opt/haiwen/seafile-pro-server-6.2.9/seafile/lib/python2.7/site-packages/ccnet/rpc.py”, line 36, in _start_service
raise SearpcError(“Error received: %s %s (In _start_service)” % (rsp.code, rsp.code_msg))
SearpcError: Error received: 511 Unknown service (In _start_service)

2 years, and no solution?
I just did a brand new install of Ubuntu 18.10, added the seafile cli packages following the docs, added the ccnet stuff (not documented), and get the same error. I’m assuming it’s a problem with the socket connection to the daemon or ccnet, as the access to the seafile server itself is working fine (list-remote lists all of the libraries I have available). Checking with strace, I can see that the daemon process is opening the socket just fine:

1556 socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
1556 socket(AF_UNIX, SOCK_STREAM, 0) = 14
1556 access("/root/seafile/seafile-data/seafile.sock", F_OK) = 0
1556 unlink("/root/seafile/seafile-data/seafile.sock") = 0
1556 bind(14, {sa_family=AF_UNIX, sun_path="/root/seafile/seafile-data/seafile.sock"}, 110) = 0
1556 chmod("/root/seafile/seafile-data/seafile.sock", 0700) = 0

though seaf-cli fails accessing the daemon process:

1631 connect(3, {sa_family=AF_UNIX, sun_path="/root/.ccnet/ccnet.sock"}, 25) = -1 ECONNREFUSED (Connection refused)

And yes, the daemon process is still running …

Here’s some output from ccnet.log:

[12/01/18 22:30:49] ccnet-daemon.c(193): starting ccnet client 6.0.4
[12/01/18 22:30:49] …/common/session.c(132): using config file /root/.ccnet/ccnet.conf
[12/01/18 22:30:49] …/common/session.c(455): socket file exists, delete it anyway
[12/01/18 22:30:49] …/common/session.c(484): Listen on /root/.ccnet/ccnet.sock for local clients
[12/01/18 22:30:49] …/common/session.c(290): Update pubinfo file
[12/01/18 22:31:01] …/common/session.c(398): Accepted a local client
[12/01/18 22:31:01] …/common/session.c(398): Accepted a local client
[12/01/18 22:31:01] …/common/peer.c(943): Local peer down
[12/01/18 22:31:01] …/common/peer.c(943): Local peer down
[12/01/18 22:34:22] …/common/session.c(398): Accepted a local client
[12/01/18 22:34:22] …/common/peer.c(943): Local peer down
[12/01/18 22:34:22] …/common/session.c(398): Accepted a local client
[12/01/18 22:34:22] …/common/session.c(398): Accepted a local client
[12/01/18 22:34:22] …/common/peer.c(943): Local peer down
[12/01/18 22:34:22] …/common/peer.c(943): Local peer down

The last 4 lines are always written when seaf-cli status (e.g.) is called …

strace of ccnet shows this:

readv(12, [{iov_base=“seafile-rpcserver”, iov_len=17}], 1) = 17
epoll_ctl(5, EPOLL_CTL_MOD, 12, {EPOLLIN|EPOLLOUT, {u32=12, u64=12}}) = 0
epoll_wait(5, [{EPOLLOUT, {u32=12, u64=12}}], 32, -1) = 1
writev(12, [{iov_base="\1\3\0\24\0\0\3\351511 Unknown service\n", iov_len=28}], 1) = 28

Could it be that the official packages are missing some files or expecting some prerequisites that are neither documented nor listed as prerequisites in the packages???

This thread is about Seafile Server - you seem to be talking about the Client, right?

Which version of the packages did you install?

From Seafile Client 6.2.0 upwards, ccnet is not required anymore.

Used the install instructions in the manual, according to the packages it’s 6.2.4-1 … if ccnet isn’t required anymore, why did the command “seaf-cli” complain about missing modules named ccnet?
Also, “seaf-cli start” EXPLICITLY starts ccnet daemon …

Aha - 6.2.4 was broken and never intended to be installed - in Debian this was assured by an appropriate bug report. Ubuntu apparently seems to have known it better and used that package anyways…

You could try to get the newer, fixed package from the next Ubuntu release somehow, I don’t exactly know how.

After clearing the APT cache, the debian versions did show up … but even though I remove the Ubuntu apt source for Seafile, I still get the 6.2.4-1 version listed:

seafile-cli:
Installiert: (keine)
Installationskandidat: 6.2.4-1
Versionstabelle:
6.2.4-1 500
500 http://archive.ubuntu.com/ubuntu cosmic/universe amd64 Packages
6.2.4 500
500 http://deb.seadrive.org jessie/main amd64 Packages
6.1.8 500
500 http://deb.seadrive.org jessie/main amd64 Packages
6.1.2 500
500 http://deb.seadrive.org jessie/main amd64 Packages
6.1.0 500
500 http://deb.seadrive.org jessie/main amd64 Packages
6.0.7 500
500 http://deb.seadrive.org jessie/main amd64 Packages

And even if I install the older version from the debian sources, it will try to update to the Ubuntu version once I run “apt upgrade” …

This cannot be.

Please post the output of “apt-cache policy seafile-cli”.