Ccnet-server segmentation fault

Seafile Server
1

We are struggling with regular segfaults of ccnet-server and service interruption. I already opened an issue on github (https://github.com/haiwen/seafile-server/issues/68), but I wanted to ask if anyone can help us to debug and possibly fix this.

The segfaults appear to be related to viewing/generating thumbnails in the seafile web UI, but I can’t find a more specific cause.

Is there any way to increase the ccnet-server verbosity? Right now, the ccnet.log only lists:

[11/07/17 12:15:10] ../common/peer.c(943): Local peer down
[11/07/17 12:15:10] ../common/peer.c(943): Local peer down
[11/07/17 12:15:10] ../common/peer.c(943): Local peer down

right after the crash.

Here is an example coredump obtained from coredumpctl:

       Message: Process 26566 (ccnet-server) of user 1001 dumped core.
                
                Stack trace of thread 26714:
                #0  0x000000000040fa9c n/a (ccnet-server)
                #1  0x0000000000410648 n/a (ccnet-server)
                #2  0x0000000000410cba n/a (ccnet-server)
                #3  0x000000000040bcff n/a (ccnet-server)
                #4  0x000000000041d546 n/a (ccnet-server)
                #5  0x00007f37c6e67f0a searpc_server_call_function (libsearpc.so.1)
                #6  0x0000000000424e30 n/a (ccnet-server)
                #7  0x00000000004181bb n/a (ccnet-server)
                #8  0x00000000004280eb n/a (ccnet-server)
                #9  0x00007f37c64022c9 n/a (libglib-2.0.so.0)
                #10 0x00007f37c64014d5 n/a (libglib-2.0.so.0)
                #11 0x00007f37c57ab6ba start_thread (libpthread.so.0)
                #12 0x00007f37c54e13dd __clone (libc.so.6)
                
                Stack trace of thread 26713:
                #0  0x00007f37c70b29a8 n/a (libsqlite3.so.0)
                #1  0x00007f37c70b331c n/a (libsqlite3.so.0)
                #2  0x00007f37c70b1f98 n/a (libsqlite3.so.0)
                #3  0x00007f37c70b1ff9 n/a (libsqlite3.so.0)
                #4  0x00007f37c70b1ff9 n/a (libsqlite3.so.0)
                #5  0x00007f37c70b4569 n/a (libsqlite3.so.0)
                #6  0x00007f37c70b4307 n/a (libsqlite3.so.0)
                #7  0x00007f37c70b22aa n/a (libsqlite3.so.0)
                #8  0x00007f37c70b4668 n/a (libsqlite3.so.0)
                #9  0x00007f37c70dcec6 n/a (libsqlite3.so.0)
                #10 0x00007f37c70dd652 n/a (libsqlite3.so.0)
                #11 0x00007f37c70ef8df n/a (libsqlite3.so.0)
                #12 0x00007f37c70f2784 n/a (libsqlite3.so.0)
                #13 0x00007f37c70f35ce n/a (libsqlite3.so.0)
                #14 0x00007f37c70d5d23 n/a (libsqlite3.so.0)
                #15 0x00007f37c70d605f n/a (libsqlite3.so.0)
                #16 0x00007f37c70d6216 sqlite3_prepare_v2 (libsqlite3.so.0)
                #17 0x000000000042e414 sqlite_prepare_statement (ccnet-server)
                #18 0x000000000042dd89 n/a (ccnet-server)
                #19 0x0000000000410660 n/a (ccnet-server)
                #20 0x0000000000410cba n/a (ccnet-server)
                #21 0x000000000040bcff n/a (ccnet-server)
                #22 0x000000000041d546 n/a (ccnet-server)
                #23 0x00007f37c6e67f0a searpc_server_call_function (libsearpc.so.1)
                #24 0x0000000000424e30 n/a (ccnet-server)
                #25 0x00000000004181bb n/a (ccnet-server)
                #26 0x00000000004280eb n/a (ccnet-server)
                #27 0x00007f37c64022c9 n/a (libglib-2.0.so.0)
                #28 0x00007f37c64014d5 n/a (libglib-2.0.so.0)
                #29 0x00007f37c57ab6ba start_thread (libpthread.so.0)
                #30 0x00007f37c54e13dd __clone (libc.so.6)
                
                Stack trace of thread 26566:
                #0  0x00007f37c54e19d3 epoll_wait (libc.so.6)
                #1  0x00007f37c85a4326 n/a (libevent-2.0.so.5)
                #2  0x00007f37c8592ac6 event_base_loop (libevent-2.0.so.5)
                #3  0x000000000040962a n/a (ccnet-server)
                #4  0x00007f37c53fa830 __libc_start_main (libc.so.6)
                #5  0x0000000000409849 n/a (ccnet-server)
2

It looks like as if ccnet-server accepts a ‘-D FLAGS’ argument to output debug information, but this is not configurable in ccnet.conf and the argv array for ccnet-server is hard coded in seafile-controller.c.

Can I safely try to start the ccnet-server inside seafile-server/seafile/bin manually and pass the -D option?

3

As a super bad workaround, I added rate limiting to nginx (20r/s, large burst, with delay).
This seems to help in avoiding the segfault for now at the price of more sluggish seahub experience.

So I guess it’s some kind of race condition in ccnet-server?

4

same problem

have test on Ubuntu 14 and Ubuntu 16
with
Seafile Server 6.2.5 / Seafile Pro 6.2.13 / Seafile Pro 6.2.11 / Seafile Pro 6.2.13 / Seafile Pro 6.2.9