Segmentation fault while running seaf-fsck.sh

Seafile Server
1

Hi,

I ran into an issue while doing a seaf-fsck. We noticed data loss (a few files) from our seafile. It’s running quite a long time. We started with version 4.x updated to 5.x and now we run the newest version 6.2.5.
The data loss was notices while running version 6.2.4.

I stopped the seafile server and did a seaf-fsck, which gave me the following output:

[04/27/18 17:59:54] fsck.c(586): Running fsck for repo d1715554-4c69-4322-b555-fe617501f987.
[04/27/18 17:59:54] fsck.c(40): Dir 224fef721334d5709a3adf44e0ee8828b7c54705 is missing.
[04/27/18 17:59:54] fsck.c(631): Repo d1715554 HEAD commit is damaged, need to restore to an old version.
[04/27/18 17:59:54] fsck.c(498): Scanning available commits...
[04/27/18 17:59:54] fsck.c(40): Dir 224fef721334d5709a3adf44e0ee8828b7c54705 is missing.
[04/27/18 17:59:54] fsck.c(40): Dir b311dd27ab3f33d42607557dc49bc14e57ae2628 is missing.
[04/27/18 17:59:54] fsck.c(40): Dir 6ac8399e372794f01a8cba4fa46032f20ba3519d is missing.
[04/27/18 17:59:54] fsck.c(40): Dir 389c9728acde1cfc9eadfb4b5be06c6582af8da5 is missing.
[04/27/18 17:59:54] fsck.c(40): Dir 1bfc5bd3e1418b189c42cf2316ef2bb6d5397e88 is missing.
[04/27/18 17:59:54] fsck.c(40): Dir eeeed3b2428c8a33f7ad7f2c90f1395b2466b649 is missing.
[04/27/18 17:59:54] fsck.c(40): Dir f917baf41e83631ba416b238fc8fc7ab0b1e89d3 is missing.
[04/27/18 17:59:54] fsck.c(40): Dir ab8ba6d87a32eb855e9cff24a40fc7f8c234fdc2 is missing.
[04/27/18 17:59:54] fsck.c(40): Dir d7950af01b34129fc687538790f9c82d5fe8ff4f is missing.
[04/27/18 17:59:54] fsck.c(40): Dir 595ce96ea5e1ec5ac3e52eb88b25bbe4c99683db is missing.
[04/27/18 17:59:54] fsck.c(413): Checking file system integrity of repo (null)(d1715554)...
./seaf-fsck.sh: line 46: 31514 Segmentation fault      LD_LIBRARY_PATH=$SEAFILE_LD_LIBRARY_PATH ${seaf_fsck} -c "${default_ccnet_conf_dir}" -d "${seafile_data_dir}" -F "${default_conf_dir}" ${seaf_fsck_opts}
seaf-fsck run done

Done.

Is there a known bug about seaf-fsck getting crashed like this? Ist there anything we can do to get seaf-fsck.sh running without crash to get all repos checked (some kind of skip option for fsck)?

Kind regards

2

A segmentation fault is when a program tries to access an area of memory that is inaccessible or does not exist. Most often, it’s due to a bad memory stick. Try running an extensive offline memory check on your memory… I suggest memtest86.

3

Hi @h.borgmeyer：
Since we can’t reproduce this problem, I built a debug version of seaf-fsck，could you replace the seaf-fsck under seafile-server-6.2.5/seafile/bin and do fsck again, and a core file will be generated in your execution path, you may need to set limit size of core file by ulimit -c unlimited before running fsck, it will be appreciated if you send us the core file.
Download link: seaf-fsck.
You can send the core file to haikuo.cui@seafile.com, thanks.

1 Like
4

Hi,

thanks @wthess for your reply. The host system uses ECC memory. I don’t think this is the reason for the crash. Indeed it could be.

I’ve sent the core file to haikuo as requested.

I will keep you updated.

Kind regards
Hendrik

5

Even with ECC memory, chips can fail. Can’t tell you how many times I’ve had segmentation faults on Linux proggies that turned out to be bad memory or in a couple of cases, a failing graphics card. Keep us up to date on that core file.

6

Hi,

I got the seaf-fsck tool from haikuo:

[05/18/18 15:33:32] fsck.c(604): Running fsck for repo d1715554-4c69-4322-b555-fe617501f987.
[05/18/18 15:33:32] fsck.c(40): Dir 224fef721334d5709a3adf44e0ee8828b7c54705 is missing.
[05/18/18 15:33:32] fsck.c(649): Repo d1715554 HEAD commit is damaged, need to restore to an old version.
[05/18/18 15:33:32] fsck.c(498): Scanning available commits...
[05/18/18 15:33:32] fsck.c(40): Dir 224fef721334d5709a3adf44e0ee8828b7c54705 is missing.
[05/18/18 15:33:32] fsck.c(40): Dir b311dd27ab3f33d42607557dc49bc14e57ae2628 is missing.
[05/18/18 15:33:32] fsck.c(40): Dir 6ac8399e372794f01a8cba4fa46032f20ba3519d is missing.
[05/18/18 15:33:32] fsck.c(40): Dir 389c9728acde1cfc9eadfb4b5be06c6582af8da5 is missing.
[05/18/18 15:33:32] fsck.c(40): Dir 1bfc5bd3e1418b189c42cf2316ef2bb6d5397e88 is missing.
[05/18/18 15:33:32] fsck.c(40): Dir eeeed3b2428c8a33f7ad7f2c90f1395b2466b649 is missing.
[05/18/18 15:33:32] fsck.c(40): Dir f917baf41e83631ba416b238fc8fc7ab0b1e89d3 is missing.
[05/18/18 15:33:32] fsck.c(40): Dir ab8ba6d87a32eb855e9cff24a40fc7f8c234fdc2 is missing.
[05/18/18 15:33:32] fsck.c(40): Dir d7950af01b34129fc687538790f9c82d5fe8ff4f is missing.
[05/18/18 15:33:32] fsck.c(40): Dir 595ce96ea5e1ec5ac3e52eb88b25bbe4c99683db is missing.
[05/18/18 15:33:32] fsck.c(570): Captain: Oh no! A segmentation fault is coming! The earth is going to be destroyed!!
[05/18/18 15:33:32] fsck.c(571): 3
[05/18/18 15:33:33] fsck.c(573): 2
[05/18/18 15:33:34] fsck.c(575): 1
[05/18/18 15:33:35] fsck.c(577): Nick Fury: Don't worry, this bug has been found by Hendrik Borgmeyer!
[05/18/18 15:33:36] fsck.c(579): Iron Man: Thanks Borgmeyer, I'll consider to give you a suit, the world peace depends on you!
[05/18/18 15:33:37] fsck.c(581): Next mission, Go!
[05/18/18 15:33:38] fsck.c(668): Fsck finished for repo d1715554.

I really like the humour of developers :slight_smile: !
All librarys have been checked. Only this one is broken. We will try to clean this up in the next days. We try to delete this library and create a new one.
I keep you updated about the issue of having data to be lost from seafile. I believe this is due to the broken library.

Thanks @haikuo for helping me.

Kind regards
Hendrik

2 Likes