Recently, one of my Seafile Servers started to deteriorate with messages like this in /var/log/kern.log:
Feb 26 08:20:10 transfer kernel: [227297.467719] pool[14757]: segfault at 8 ip 0000000000442654 sp 00007f8464fd7750 error 4 in seaf-server[400000+105000]
Feb 26 08:20:10 transfer kernel: [227297.467734] Code: c7 44 24 68 00 00 00 00 48 85 db 75 2a e9 9c 01 00 00 0f 1f 40 00 e8 cb 57 fc ff 48 8b 7c 24 68 48 85 ff 74 05 e8 bc 57 fc ff <4
8> 8b 5b 08 48 85 db 0f 84 77 01 00 00 48 8b 2b 48 8d 54 24 60 31
Feb 26 08:52:41 transfer kernel: [229249.383857] pool[17967]: segfault at 8 ip 0000000000442654 sp 00007fccb4ff8750 error 4 in seaf-server[400000+105000]
Feb 26 08:52:41 transfer kernel: [229249.383872] Code: c7 44 24 68 00 00 00 00 48 85 db 75 2a e9 9c 01 00 00 0f 1f 40 00 e8 cb 57 fc ff 48 8b 7c 24 68 48 85 ff 74 05 e8 bc 57 fc ff <4
8> 8b 5b 08 48 85 db 0f 84 77 01 00 00 48 8b 2b 48 8d 54 24 60 31
Feb 26 08:57:39 transfer kernel: [229547.207478] pool[18048]: segfault at 8 ip 0000000000442654 sp 00007f5e197f9750 error 4 in seaf-server[400000+105000]
Feb 26 08:57:39 transfer kernel: [229547.207490] Code: c7 44 24 68 00 00 00 00 48 85 db 75 2a e9 9c 01 00 00 0f 1f 40 00 e8 cb 57 fc ff 48 8b 7c 24 68 48 85 ff 74 05 e8 bc 57 fc ff <4
8> 8b 5b 08 48 85 db 0f 84 77 01 00 00 48 8b 2b 48 8d 54 24 60 31
It’s a virtual machine running on a Hyper-V. Operating system is Debian GNU/Linux 10 (buster).
The user just gets a non-descriptive error. After refreshing the page, usually it works out eventually.
I dared to make an upgrade from 7.1.4 to 7.1.5 but it didn’t fix anything.
I’m assuming there’s a connection between these faults and damaged repos :
[02/26/21 00:18:49] gc-core.c(456): === GC is finished ===
[02/26/21 00:18:49] gc-core.c(460): The following repos are damaged. You can run seaf-fsck to fix them.
[02/26/21 00:18:49] gc-core.c(463): 1e13c35a-a82b-4cac-91e1-d7893371ee31
[02/26/21 00:18:49] gc-core.c(463): 28785a8d-e1d4-4749-a088-d433b367f4d1
[02/26/21 00:18:49] gc-core.c(463): 2d4f54a5-4e20-473a-b1ae-590bd620f3a7
[02/26/21 00:18:49] gc-core.c(463): 31e7cc1f-3c03-49b4-9104-8fdb678c8eef
[02/26/21 00:18:49] gc-core.c(463): 3e0f3a80-e142-4f90-80e1-f3d9a3bf4aba
[02/26/21 00:18:49] gc-core.c(463): 55740408-9a2e-457a-9bb0-c41cbd03c50a
[02/26/21 00:18:49] gc-core.c(463): 56b714ef-8828-4a3d-9121-8685b2b6ab9b
[02/26/21 00:18:49] gc-core.c(463): 71e03931-033a-4617-b4b9-036ec7b1bf76
[02/26/21 00:18:49] gc-core.c(463): 7feff8fd-fddc-4ec6-8f9d-b6a7e68c7a02
[02/26/21 00:18:49] gc-core.c(463): 83954dce-7ad9-4b56-9d6f-08b743e33680
[02/26/21 00:18:49] gc-core.c(463): 8b2375bd-c267-4d6a-ad5c-716f8275f55d
[02/26/21 00:18:49] gc-core.c(463): 8d37713c-e86d-4814-953e-cff826374d63
[02/26/21 00:18:49] gc-core.c(463): 92a719dd-0cb2-4af4-8c69-80bdff0f057a
[02/26/21 00:18:49] gc-core.c(463): 93d032b3-cfdd-424a-b5a4-23982dc7aa98
[02/26/21 00:18:49] gc-core.c(463): 985b9145-86b7-472f-8d79-ed764a2413b5
[02/26/21 00:18:49] gc-core.c(463): 9cacc932-36f7-4345-9e56-19bce57b19a0
[02/26/21 00:18:49] gc-core.c(463): 9e21eb93-cd8c-4084-8c9d-7fd72ef3596e
[02/26/21 00:18:49] gc-core.c(463): bc711457-2f8b-4e4a-8fe4-b130cfe08263
[02/26/21 00:18:49] gc-core.c(463): d9d8601c-f57a-48cd-891c-d7b64c9aad6f
[02/26/21 00:18:49] gc-core.c(463): e1e3f9fb-5e4e-4b93-a0a0-3f5f1eac5d24
[02/26/21 00:18:49] gc-core.c(463): e283790b-cd2d-4ef9-b192-db210593fb6d
[02/26/21 00:18:49] gc-core.c(463): fe9b4056-7b41-43af-838e-542963e71aad
seafserv-gc run done
seaf-fsck is run periodically but it doesn’t fix any of these repos.
This is from seaf-fuse.log:
2021-02-26 00:19:03.104 - <139635882320256> wsgidav.dc.seahub_db INFO : Init seahub database…
2021-02-26 00:19:03.138 - <139635882320256> wsgidav.wsgidav_app INFO : WsgiDAV/3.0.4 Python/3.7.3 Linux-4.19.0-14-amd64-x86_64-with-debian-10.8
2021-02-26 00:19:03.138 - <139635882320256> wsgidav.wsgidav_app INFO : Lock manager: LockManager(LockStorageDict)
2021-02-26 00:19:03.138 - <139635882320256> wsgidav.wsgidav_app INFO : Property manager: None
2021-02-26 00:19:03.138 - <139635882320256> wsgidav.wsgidav_app INFO : Domain controller: SeafileDomainController()
2021-02-26 00:19:03.139 - <139635882320256> wsgidav.wsgidav_app INFO : Registered DAV providers by route:
2021-02-26 00:19:03.139 - <139635882320256> wsgidav.wsgidav_app INFO : - ‘/:dir_browser’: FilesystemProvider for path ‘/home/seafile/seafile-server-7.1.5/seahub/thirdpart/wsgidav/dir_browser/htdocs’ (Read-Only)
2021-02-26 00:19:03.139 - <139635882320256> wsgidav.wsgidav_app INFO : - ‘/seafdav’: SeafileProvider for Seafile (Read-Write)
2021-02-26 00:19:03.139 - <139635882320256> wsgidav.wsgidav_app WARNING : Basic authentication is enabled: It is highly recommended to enable SSL.
2021-02-26 00:19:03.139 - <139635882320256> wsgidav WARNING : Could not import lxml: using xml instead (up to 10% slower). Considerpip install lxml
(see lxml · PyPI).
This, I don’t get either:
root@transfer:~# pip install lxml
-bash: pip: command not found
root@transfer:~# pip3 install lxml
Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (4.5.1)
And seafile.log contains a lot of this:
[02/26/21 05:00:59] repo-mgr.c(256): Commit 43fdecaf3285ec10a42a7b097c8982ad727bcec3 is missing
[02/26/21 05:00:59] repo-mgr.c(322): Repo 31e7cc1f is corrupted.
[02/26/21 05:00:59] repo-mgr.c(256): Commit f4053115119e997642ba917cd3e325c45d8dc6dc is missing
[02/26/21 05:00:59] repo-mgr.c(322): Repo 83954dce is corrupted.
[02/26/21 05:01:21] repo-mgr.c(256): Commit e6d1bb360984253ba8f81f15af04eb02d1e5299d is missing
[02/26/21 05:01:21] repo-mgr.c(322): Repo d9d8601c is corrupted.
[02/26/21 05:01:21] repo-mgr.c(256): Commit 6f02963119789f7e1fb895f2835500779102ac48 is missing
[02/26/21 05:01:21] repo-mgr.c(322): Repo e283790b is corrupted.
[02/26/21 05:01:21] repo-mgr.c(256): Commit c82f16070d9f398763362dc68bada644c0d493a9 is missing
[02/26/21 05:01:21] repo-mgr.c(322): Repo 55740408 is corrupted.
[02/26/21 05:01:21] repo-mgr.c(256): Commit 26fc87e4bc6afae63e49ef6811cf0ffa180cce67 is missing
[02/26/21 05:01:21] repo-mgr.c(322): Repo 71e03931 is corrupted.
[02/26/21 05:01:29] repo-mgr.c(256): Commit ea6031215395bafdcee56e0b556ed0d5adb07560 is missing
[02/26/21 05:01:29] repo-mgr.c(322): Repo 28785a8d is corrupted.
And seahub.log shows a lot of these:
2021-02-26 11:13:50,232 [ERROR] django.request:135 handle_uncaught_exception Internal Server Error: /api2/account/info/
Traceback (most recent call last):
File “/home/seafile/seafile-server-7.1.5/seafile/lib64/python3.6/site-packages/pysearpc/utils.py”, line 30, in sendall
n = fd.send(data[offset:])
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/home/seafile/seafile-server-7.1.5/seahub/thirdpart/django/core/handlers/exception.py”, line 41, in inner
response = get_response(request)
File “/home/seafile/seafile-server-7.1.5/seahub/thirdpart/django/core/handlers/base.py”, line 249, in _legacy_get_response
response = self._get_response(request)
File “/home/seafile/seafile-server-7.1.5/seahub/thirdpart/django/core/handlers/base.py”, line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File “/home/seafile/seafile-server-7.1.5/seahub/thirdpart/django/core/handlers/base.py”, line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File “/home/seafile/seafile-server-7.1.5/seahub/thirdpart/django/views/decorators/csrf.py”, line 58, in wrapped_view
return view_func(*args, **kwargs)
File “/home/seafile/seafile-server-7.1.5/seahub/thirdpart/django/views/generic/base.py”, line 68, in view
return self.dispatch(request, *args, **kwargs)
File “/home/seafile/seafile-server-7.1.5/seahub/seahub/api2/base.py”, line 23, in dispatch
response = super(APIView, self).dispatch(*a, **kw)
File “/home/seafile/seafile-server-7.1.5/seahub/thirdpart/rest_framework/views.py”, line 505, in dispatch
response = self.handle_exception(exc)
File “/home/seafile/seafile-server-7.1.5/seahub/seahub/api2/base.py”, line 20, in handle_exception
return super(APIView, self).handle_exception(exc)
File “/home/seafile/seafile-server-7.1.5/seahub/thirdpart/rest_framework/views.py”, line 465, in handle_exception
self.raise_uncaught_exception(exc)
File “/home/seafile/seafile-server-7.1.5/seahub/thirdpart/rest_framework/views.py”, line 476, in raise_uncaught_exception
raise exc
File “/home/seafile/seafile-server-7.1.5/seahub/thirdpart/rest_framework/views.py”, line 502, in dispatch
response = handler(request, *args, **kwargs)
File “/home/seafile/seafile-server-7.1.5/seahub/seahub/api2/views.py”, line 343, in get
return Response(self._get_account_info(request))
File “/home/seafile/seafile-server-7.1.5/seahub/seahub/api2/views.py”, line 309, in _get_account_info
quota_total = seafile_api.get_user_quota(email)
File “/home/seafile/seafile-server-7.1.5/seafile/lib64/python3.6/site-packages/seaserv/api.py”, line 724, in get_user_quota
return seafserv_threaded_rpc.get_user_quota(username)
File “/home/seafile/seafile-server-7.1.5/seafile/lib64/python3.6/site-packages/pysearpc/client.py”, line 126, in newfunc
ret_str = self.call_remote_func_sync(fcall_str)
File “/home/seafile/seafile-server-7.1.5/seafile/lib64/python3.6/site-packages/pysearpc/named_pipe.py”, line 101, in call_remote_func_sync
ret_str = transport.send(self.service_name, fcall_str)
File “/home/seafile/seafile-server-7.1.5/seafile/lib64/python3.6/site-packages/pysearpc/named_pipe.py”, line 62, in send
sendall(self.pipe, header)
File “/home/seafile/seafile-server-7.1.5/seafile/lib64/python3.6/site-packages/pysearpc/utils.py”, line 32, in sendall
raise NetworkError(‘Failed to write to socket: %s’ % e)
pysearpc.errors.NetworkError: Failed to write to socket: [Errno 32] Broken pipe