Unable to repair damaged library Seafile 9.0.x Pro

This has happened twice so far, a couple weeks apart, since upgrading to 9.0.x pro. The same library (encrypted, approx 80 GB) throws an error and stops syncing. This is one library out of about 15. I run seaf-gc and it says the library is damaged. I run seaf-fsck and it says it successfully restores the library to a prior version, but it fails to sync and seaf-gc still says the library is damaged. I’m also unable to browse the library online.

The first time this happened I tried restoring something like the 10 most recent snapshots (by searching for the snapshot by date created/modified and moving each snapshot that was unsuccessful before running seaf-fsck), but nothing worked. The only solution was to delete the library and start from scratch (losing my library history in the process). Prior to version 9 I had Seafile running for years without issue.

Anyone encounter this, or is there another way to restore a snapshot?

What kind of system and hardware is it running on?

Intel Nuc 6i7kyk with a 2tb m.2 solid state drive running Ubuntu 20.04

It just happened again. The seafile client says “Error: Failed to write data on the client. Please check disk space or folder permissions.” seaf-fsck.sh says it successfully restored the repo to a prior commit, but the repo is still damaged. This is a significant problem.

seaf-gc.sh output is as follows:

Commit xxxxx is missing
Repo xxxxx is damaged, skip GC.

seaf-fsck.sh output is as follows:

Find available commit xxxxx(created at 2022-01-21 16:49:12) for repo xxxxx.
Checking file system integrity of repo bklf(xxxxx)…
Update repo xxxxx status to commit xxxxx.
Fsck finished for repo xxxxx.

seafevents.log displays the following around the time the library stopped syncing:

[2022-01-21 16:49:13,551] [ERROR] error when handle msg: 'NoneType' object has no attribute 'repo_name'
Traceback (most recent call last):
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/seafevents/app/mq_handler.py", line 49, in handle_message
    func(session, msg)
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/seafevents/events/handlers.py", line 82, in RepoUpdateEventHandler
    records = generate_activity_records(added_files, deleted_files,
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/seafevents/events/handlers.py", line 162, in generate_activity_records
    'repo_name': repo.repo_name
AttributeError: 'NoneType' object has no attribute 'repo_name'

index.log has been spamming me with the following message since December 29 (the date I updated to Seafile Pro version 9.0.3):

01/21/2022 17:13:40 [WARNING] seafes:110 thread_task: Request Error: RequestError(400, 'invalid_type_name_exception', "Document mapping type name can't start with '_', found: [_update]")
Traceback (most recent call last):
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/seafes/index_local.py", line 105, in thread_task
    self.fileindexupdater.update_repo(repo_id, commit_id)
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/seafes/file_index_updater.py", line 89, in update_repo
    self.status_index.begin_update_repo(repo_id, old, new)
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/seafes/indexes/repo_status.py", line 108, in begin_update_repo
    self.es.update(index=self.INDEX_NAME, id=repo_id, body=dict(doc=doc))
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/elasticsearch/client/__init__.py", line 2101, in update
    return self.transport.perform_request(
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/elasticsearch/transport.py", line 466, in perform_request
    raise e
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/elasticsearch/transport.py", line 427, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/elasticsearch/connection/http_urllib3.py", line 291, in perform_request
    self._raise_error(response.status, raw_data)
  File "/opt/seafile/seafile-pro-server-9.0.3/pro/python/elasticsearch/connection/base.py", line 328, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.RequestError: RequestError(400, 'invalid_type_name_exception', "Document mapping type name can't start with '_', found: [_update]")

fsck checks and repairs only the latest commit for the library. If the current head commit is missing, it’ll try to choose an older commit to replace it. When choosing this new head commit, it only checks the root folder. So there can still be a chance that some files under that head commit is damaged too.

gc also checks history commits. So basically you also have damaged files in the history. So gc cannot work.

After fsck, all synced clients are forced to resync. You need to resync the library to make sync work again.

From what you provided, it seems that your library is damaged deep into the history. This seems to be a system wide issue. With the current implementation of fsck, you can try run it multiple times until fsck tells you that the current commit is fully sound.

Is there any way to see what files are causing the library corruption? With regards to fsck, every time I run it, it reports that the library was successfully repaired. After fsck, any attempt to resync the library from the client fails, and I’m unable to view the library online. And gc still reports that the library is damaged.

It just happened again. Same library. fsck says it successfully reverted to a prior commit, seaf-gc says the library is still damaged. I’m going to have to downgrade to version 8.x until this is fixed.