Hi,
I just ran seaf-fsck on a repo that causes some errors in seafile.log. The process crashed after a while with the following errors:
[seafile@xxx seafile-server-latest]$ ./seaf-fsck.sh -r b6bb5186-4beb-4a98-b064-5b6c7fe5b088
Starting seaf-fsck, please wait ...
[09/24/19 09:26:09] fsck.c(599): Running fsck for repo b6bb5186-4beb-4a98-b064-5b6c7fe5b088.
[09/24/19 09:26:10] fsck.c(425): Checking file system integrity of repo Seafile(b6bb5186)...
[09/24/19 10:14:55] fsck.c(102): Repo[b6bb5186] block b6bb5186-4beb-4a98-b064-5b6c7fe5b088:bad0086f3ebd0c30be114f90736e1491b209f250 is missing.
[09/24/19 10:14:55] ../../common/block-backend-fs.c(85): [block bend] failed to open block bad0086f3ebd0c30be114f90736e1491b209f250 for read: No such file or directory
[09/24/19 10:14:55] ../../common/block-mgr.c(768): Failed to open block b6bb5186-4beb-4a98-b064-5b6c7fe5b088:bad0086f.
*** Error in `/home/seafile/seafile-pro-server-7.0.9/seafile/bin/seaf-fsck': free(): invalid size: 0x000000000170dd60 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x816b9)[0x7f029843d6b9]
/home/seafile/seafile-pro-server-7.0.9/seafile/lib/libglib-2.0.so.0(g_free+0xe)[0x7f029ce10bce]
/home/seafile/seafile-pro-server-7.0.9/seafile/bin/seaf-fsck[0x408723]
/home/seafile/seafile-pro-server-7.0.9/seafile/bin/seaf-fsck[0x4086f6]
/home/seafile/seafile-pro-server-7.0.9/seafile/bin/seaf-fsck[0x4093ed]
/home/seafile/seafile-pro-server-7.0.9/seafile/bin/seaf-fsck[0x409673]
/home/seafile/seafile-pro-server-7.0.9/seafile/bin/seaf-fsck[0x407a08]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f02983de545]
/home/seafile/seafile-pro-server-7.0.9/seafile/bin/seaf-fsck[0x407b19]
======= Memory map: ========
00400000-0044b000 r-xp 00000000 fd:02 4235693 /home/seafile/seafile-pro-server-7.0.9/seafile/bin/seaf-fsck
0064a000-0064b000 r--p 0004a000 fd:02 4235693 /home/seafile/seafile-pro-server-7.0.9/seafile/bin/seaf-fsck
0064b000-0064c000 rw-p 0004b000 fd:02 4235693 /home/seafile/seafile-pro-server-7.0.9/seafile/bin/seaf-fsck
…
I started seaf-fsck again without the repair option, and I’m getting a segmentation fault (line 46 in seaf-fsck.sh):
[09/24/19 13:19:29] ../../common/block-backend-fs.c(85): [block bend] failed to open block bad0086f3ebd0c30be114f90736e1491b209f250 for read: No such file or directory
[09/24/19 13:19:29] ../../common/block-mgr.c(768): Failed to open block b6bb5186-4beb-4a98-b064-5b6c7fe5b088:bad0086f.
./seaf-fsck.sh: Zeile 46: 81293 Speicherzugriffsfehler LD_LIBRARY_PATH=$SEAFILE_LD_LIBRARY_PATH ${seaf_fsck} -c "${default_ccnet_conf_dir}" -d "${seafile_data_dir}" -F "${default_conf_dir}" ${seaf_fsck_opts}
BTW, I’m using Scientific Linux 7.7
Hi,
No problem for me (but with few repos and files)
We’ll look into the problem.
Hi @uosseafile
Can you still reproduce the issue? We cannot reproduce it in our testing environment. So we’ll have to provide you with a debug package, then you send back the back trace to us.
Hi Jonathan,
I just ran seaf-fsck on the same library again with a similar result:
[seafile@xxx seafile-server-latest]$ ./seaf-fsck.sh b6bb5186-...-5b6c7fe5b088
Starting seaf-fsck, please wait ...
[12/10/19 09:48:40] fsck.c(599): Running fsck for repo b6bb5186-...-5b6c7fe5b088.
[12/10/19 09:48:41] fsck.c(425): Checking file system integrity of repo Seafile(b6bb5186)...
[12/10/19 10:38:42] fsck.c(102): Repo[b6bb5186] block b6bb5186-4beb-4a98-b064-5b6c7fe5b088:bad0086f3ebd0c30be114f90736e1491b209f250 is missing.
[12/10/19 10:38:42] ../../common/block-backend-fs.c(85): [block bend] failed to open block bad0086f3ebd0c30be114f90736e1491b209f250 for read: No such file or directory
[12/10/19 10:38:42] ../../common/block-mgr.c(768): Failed to open block b6bb5186-4beb-4a98-b064-5b6c7fe5b088:bad0086f.
[12/10/19 10:39:10] fsck.c(102): Repo[b6bb5186] block b6bb5186-4beb-4a98-b064-5b6c7fe5b088:7eefe2e0c677897c97c305e94296c090f6e4f7bc is missing.
[12/10/19 10:39:10] ../../common/block-backend-fs.c(85): [block bend] failed to open block 7eefe2e0c677897c97c305e94296c090f6e4f7bc for read: No such file or directory
[12/10/19 10:39:10] ../../common/block-mgr.c(768): Failed to open block b6bb5186-4beb-4a98-b064-5b6c7fe5b088:7eefe2e0.
*** Error in `/home/seafile/seafile-pro-server-7.0.10/seafile/bin/seaf-fsck': corrupted size vs. prev_size: 0x0000000001956010 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x80fc7)[0x7f062e689fc7]
/lib64/libc.so.6(+0x8159e)[0x7f062e68a59e]
/home/seafile/seafile-pro-server-7.0.10/seafile/lib/libglib-2.0.so.0(g_free+0xe)[0x7f063305dbce]
/home/seafile/seafile-pro-server-7.0.10/seafile/lib/libglib-2.0.so.0(+0x38c19)[0x7f0633046c19]
/home/seafile/seafile-pro-server-7.0.10/seafile/lib/libglib-2.0.so.0(g_hash_table_remove_all+0x5b)[0x7f0633047a0b]
/home/seafile/seafile-pro-server-7.0.10/seafile/lib/libglib-2.0.so.0(g_hash_table_destroy+0xe)[0x7f0633047a4e]
/home/seafile/seafile-pro-server-7.0.10/seafile/bin/seaf-fsck[0x4093fa]
/home/seafile/seafile-pro-server-7.0.10/seafile/bin/seaf-fsck[0x409673]
/home/seafile/seafile-pro-server-7.0.10/seafile/bin/seaf-fsck[0x407a08]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f062e62b545]
/home/seafile/seafile-pro-server-7.0.10/seafile/bin/seaf-fsck[0x407b19]
======= Memory map: ========
00400000-0044b000 r-xp 00000000 fd:02 7038259 /home/seafile/seafile-pro-server-7.0.10/seafile/bin/seaf-fsck
0064a000-0064b000 r--p 0004a000 fd:02 7038259 /home/seafile/seafile-pro-server-7.0.10/seafile/bin/seaf-fsck
0064b000-0064c000 rw-p 0004b000 fd:02 7038259 /home/seafile/seafile-pro-server-7.0.10/seafile/bin/seaf-fsck
0064c000-0064d000 rw-p 00000000 00:00 0
017c2000-0196e000 rw-p 00000000 00:00 0 [heap]
...
7f062a50b000-7f062a70a000 ---p 00025000 fd:02 522017 /home/seafile/seafile-pro-server-7.0.10/seafile/lib/liblzma.so.5./seaf-fsck.sh: line 46: 105476 Aborted LD_LIBRARY_PATH=$SEAFILE_LD_LIBRARY_PATH ${seaf_fsck} -c "${default_ccnet_conf_dir}" -d "${seafile_data_dir}" -F "${default_conf_dir}" ${seaf_fsck_opts}
It is a very big library with more than 30,000 files.
Hi @uosseafile
You can download the debug version here: https://download.seafile.com/f/57c929b0af234cb89fe0/?dl=1
Before running, please run following command first.
ulimit -c unlimited
After the problem occurs, there should be a ‘core’ file in the current directory. Please run gdb to print the stack trace of the crash.
gdb seafile-server-latest/seafile/bin/seaf-fsck core
Then enter ‘backtrace’ command in gdb prompt.
Hi @Jonathan,
thanks, that’s what I got:
Core was generated by `/home/seafile/seafile-pro-server-7.0.10/seafile/bin/seaf-fsck -c /home/seafile/'.
Program terminated with signal 6, Aborted.
#0 0x00007f6ee98c0377 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) backtrace
#0 0x00007f6ee98c0377 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007f6ee98c1a68 in __GI_abort () at abort.c:90
#2 0x00007f6ee9902ec7 in __libc_message (do_abort=2, fmt=fmt@entry=0x7f6ee9a153f8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007f6ee990afc7 in malloc_printerr (ar_ptr=0x7f6ee9c51760 <main_arena>, ptr=0x1ba1b00, str=0x7f6ee9a12a77 "corrupted size vs. prev_size", action=<optimized out>) at malloc.c:4967
#4 malloc_consolidate (av=av@entry=0x7f6ee9c51760 <main_arena>) at malloc.c:4161
#5 0x00007f6ee990b59e in _int_free (av=0x7f6ee9c51760 <main_arena>, p=0x1c05200, have_lock=0) at malloc.c:4053
#6 0x00007f6eee2debce in g_free () from /home/seafile/seafile-pro-server-7.0.10/seafile/lib/libglib-2.0.so.0
#7 0x00007f6eee2c7c19 in g_hash_table_remove_all_nodes.part.0 () from /home/seafile/seafile-pro-server-7.0.10/seafile/lib/libglib-2.0.so.0
#8 0x00007f6eee2c8a0b in g_hash_table_remove_all () from /home/seafile/seafile-pro-server-7.0.10/seafile/lib/libglib-2.0.so.0
#9 0x00007f6eee2c8a4e in g_hash_table_destroy () from /home/seafile/seafile-pro-server-7.0.10/seafile/lib/libglib-2.0.so.0
#10 0x00000000004093fa in check_and_recover_repo (repair=0, reset=0, repo=0x1b0aec0) at fsck.c:442
#11 repair_repo (repo_id=repo_id@entry=0x1b01d50 "b6bb5186-4beb-4a98-b064-5b6c7fe5b088", repair=repair@entry=0) at fsck.c:659
#12 0x0000000000409673 in repair_repos (max_thread_num=0, repair=0, repo_id_list=0x1adda40) at fsck.c:701
#13 seaf_fsck (repo_id_list=repo_id_list@entry=0x1adda40, repair=repair@entry=0, max_thread_num=max_thread_num@entry=0) at fsck.c:716
#14 0x0000000000407a08 in main (argc=<optimized out>, argv=<optimized out>) at seaf-fsck.c:202
I hope this helps.
Dirk
Hi @uosseafile
Sorry for the late reply. We analyze the issue but could not find any real issue in the code. A possible cause is that the library’s directory hierarchy is too deep to be traversed recursively, causing the fsck program to overrun its stack. Is the library’s directory hierarchy very deep?
1 Like
Hi Jonathan,
the library contains a huge number of files. Maybe the user tried to backup her drive with seafile. I will try to talk to her.
Thanks for your support.
Dirk