Unreachable data

[Dks]

Good afternoon everyone

I’ve been using Seafile since a few years now. Hosted the old way (without docker) on a Debian VM.

Recently, my wife found out that a few files were unreachable through the Android application. For images and videos, a message appear telling us that the loading failed, documents download is stuck around 20-25%. I have the same issue when using the web browser.

The libraries where those files are, are working correctly even though seaf-fsck return unavailable blocks. Debian fsck is clean, so does the disk health in iLO.
When using seaf-fuse, the unrechable files are correctly shown with their correct size but they’re still unreadble. When parsing the library snapshot, the files are in the same state.

What looks strange to me, is what seaf-gc returns (see below). It says “about 517 reachable blocks”. What does it mean ? According to the 6th line there are 517 blocks in that library. Does it mean the whole library is consistent (so there’s no data corruption) ?

025-06-19 13:57:46 gc-core.c(590): GC version 1 repo Maison(4fc822a3-42e4-4dbf-8a0d-f122816e572b)
2025-06-19 13:57:46 gc-core.c(417): GC started. Total block number is 512.
2025-06-19 13:57:46 gc-core.c(41): GC index size is 1024 Byte.
2025-06-19 13:57:46 gc-core.c(442): Populating index.
2025-06-19 13:57:46 gc-core.c(187): Populating index for repo 4fc822a3.
2025-06-19 13:57:48 gc-core.c(271): Traversed 531 commits, 517 blocks.
2025-06-19 13:57:48 gc-core.c(462): Scanning unused blocks.
2025-06-19 13:57:48 gc-core.c(512): GC finished. 512 blocks total, about 517 reachable blocks, 0 blocks can be removed.

The most part of the files concerned by this issue were uploaded +/- when I upgraded Seafile to v10.xxx then v11. If I correctly remember, there was a migration from sqlite to sql. Anyone heard about same kind of issue which could be related to this update ?

seafile.log

2025-06-17 11:42:31 access-file.c(289): Failed to open block 4fc822a3-42e4-4dbf-8a0d-f122816e572b:d5234092dbfc9f84bc41baac5d8f54f68916a4df
2025-06-17 11:42:51 zip-download-mgr.c(462): Total download size 933194614, exceed max download dir size 104857600.
2025-06-17 11:49:19 ../common/seaf-db.c(1012): Failed to execute sql REPLACE INTO Branch (name, repo_id, commit_id) VALUES (?, ?, ?): Lost connection to server during query
2025-06-17 11:49:19 ../common/seaf-db.c(433): The mysql connection has expired, creating a new connection to re-query.
2025-06-17 11:49:47 ../common/block-backend-fs.c(359): Failed to link /opt/seafile/seafile-data/storage/blocks/4fc822a3-42e4-4dbf-8a0d-f122816e572b/a5/8373633a6850e0253f857823f28deacfc3e2c9 to /opt/seafile/seaf>
2025-06-17 11:49:47 repo-op.c(2175): Failed to copy block a58373633a6850e0253f857823f28deacfc3e2c9 from repo 4fc822a3-42e4-4dbf-8a0d-f122816e572b to f9981bb8-41ff-4f40-a0b2-03c845896601.
2025-06-17 11:49:47 repo-op.c(2480): [copy files] Failed to copy file mail-humidite.docx.
2025-06-17 12:57:22 ../common/block-backend-fs.c(70): [block bend] failed to open block d5234092dbfc9f84bc41baac5d8f54f68916a4df for read: No such file or directory
2025-06-17 12:57:22 access-file.c(289): Failed to open block 4fc822a3-42e4-4dbf-8a0d-f122816e572b:d5234092dbfc9f84bc41baac5d8f54f68916a4df
2025-06-17 16:47:06 zip-download-mgr.c(462): Total download size 1418415081, exceed max download dir size 104857600.
2025-06-17 18:28:12 access-file.c(786): Failed to open block 0dd62340-f2d9-4c95-ac39-3a6f35be6123:(null)
2025-06-19 13:32:04 seafile-session.c(65): fileserver: web_token_expire_time = 3600
2025-06-19 13:32:04 seafile-session.c(77): fileserver: max_index_processing_threads= 3

I have the following lines i seafhttp.Error.log (the files concerned by this issue are pretty small, between 2 and 50 Mb, so It’s because of the file size or nginx timeout)

25/06/19 14:35:44 [error] 641#641: *2889 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/cf667a77-a801-459b-b85a-78f3>
2025/06/19 14:35:44 [error] 641#641: *2891 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/15f6882f-f915-49c9-8611-11b0>
2025/06/19 14:35:47 [error] 641#641: *2899 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/55496e7f-cfd4-42dd-b5fc-ef6a>
2025/06/19 14:35:47 [error] 639#639: *2900 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/7d23e8c4-c6a7-4299-b17a-8578>
2025/06/19 14:35:47 [error] 641#641: *2903 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/9d363db1-121e-4117-b95d-bde1>
2025/06/19 14:35:49 [error] 641#641: *2907 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/cd03022c-6a35-4ead-b360-b13a>
2025/06/19 14:35:50 [error] 641#641: *2911 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/fad91f02-3bd3-45ca-b381-b6ae>
2025/06/19 14:35:50 [error] 641#641: *2915 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/13941a96-a1b1-41e5-a535-f736>
2025/06/19 14:35:51 [error] 641#641: *2919 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/56de076b-012b-4c59-8ddb-4ce6>
2025/06/19 14:35:51 [error] 641#641: *2923 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/c0bf6252-0240-4e51-8b3d-80ca>
2025/06/19 14:35:51 [error] 641#641: *2927 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/0a78664f-bd49-43a2-b5a3-bf76>
2025/06/19 14:36:06 [error] 641#641: *2931 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/c0becdb0-91b6-4bee-9335-25af>
2025/06/19 14:36:06 [error] 641#641: *2935 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/42509b3e-be35-4d4c-984e-d8ea>
2025/06/19 14:36:06 [error] 641#641: *2939 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/902c6277-d0f1-4c8d-b7c1-82e6>
2025/06/19 14:36:07 [error] 641#641: *2943 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/45dd8b62-8555-4ab1-8f0d-e356>
2025/06/19 14:36:07 [error] 641#641: *2947 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/fe21c390-bdcf-4522-91e7-9a78>
2025/06/19 14:36:09 [error] 641#641: *2951 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/72ed7e01-84b8-4ba3-b5dc-93f4>
2025/06/19 14:36:09 [error] 641#641: *2955 upstream prematurely closed connection while reading upstream, client: 172.16.1.1, server: cloud..me, request: "GET /seafhttp/files/cd8ce57

Thanks for your help

EDIT:

When showing part of the concerned files (this time, images), I got the following error from django which probably just can’t generate the thumbnail

2025-06-19 14:53:30,869 [WARNING] seahub.thumbnail.utils:144 generate_thumbnail IncompleteRead(0 bytes read, 1957658 more expected)
2025-06-19 14:53:30,870 [WARNING] django.request:241 log_response Bad Request: /thumbnail/79ddbe0b-d561-46f1-81ef-efc7c39c1070/create/
2025-06-19 14:53:31,273 [WARNING] seahub.thumbnail.utils:144 generate_thumbnail IncompleteRead(0 bytes read, 2139473 more expected)
2025-06-19 14:53:31,273 [WARNING] django.request:241 log_response Bad Request: /thumbnail/79ddbe0b-d561-46f1-81ef-efc7c39c1070/create/
2025-06-19 14:53:31,331 [WARNING] seahub.thumbnail.utils:144 generate_thumbnail IncompleteRead(0 bytes read, 2419211 more expected)
2025-06-19 14:53:31,331 [WARNING] django.request:241 log_response Bad Request: /thumbnail/79ddbe0b-d561-46f1-81ef-efc7c39c1070/create/
2025-06-19 14:53:31,406 [WARNING] seahub.thumbnail.utils:144 generate_thumbnail IncompleteRead(0 bytes read, 2479949 more expected)
2025-06-19 14:53:31,407 [WARNING] django.request:241 log_response Bad Request: /thumbnail/79ddbe0b-d561-46f1-81ef-efc7c39c1070/create/
2025-06-19 14:53:31,483 [WARNING] seahub.thumbnail.utils:144 generate_thumbnail IncompleteRead(0 bytes read, 2782401 more expected)
2025-06-19 14:53:31,484 [WARNING] django.request:241 log_response Bad Request: /thumbnail/79ddbe0b-d561-46f1-81ef-efc7c39c1070/create/
2025-06-19 14:53:31,544 [WARNING] seahub.thumbnail.utils:144 generate_thumbnail IncompleteRead(0 bytes read, 3174649 more expected)
2025-06-19 14:53:31,545 [WARNING] django.request:241 log_response Bad Request: /thumbnail/79ddbe0b-d561-46f1-81ef-efc7c39c1070/create/
2025-06-19 14:53:31,625 [WARNING] seahub.thumbnail.utils:144 generate_thumbnail IncompleteRead(0 bytes read, 3773177 more expected)
2025-06-19 14:53:31,625 [WARNING] django.request:241 log_response Bad Request: /thumbnail/79ddbe0b-d561-46f1-81ef-efc7c39c1070/create/
2025-06-19 14:53:31,703 [WARNING] seahub.thumbnail.utils:144 generate_thumbnail IncompleteRead(0 bytes read, 3947721 more expected)
2025-06-19 14:53:31,703 [WARNING] django.request:241 log_response Bad Request: /thumbnail/79ddbe0b-d561-46f1-81ef-efc7c39c1070/create/

[Dks]

(sorry, I’ve hit the caracters limit)

If I try to download one of the files, I’m redirected to seafhttp/files/ac04f730-ac62-4811-b2f6-4ad2308fab0f/IMG20240318101907.jpg which generate the following error in seahub.log

2025-06-19 14:56:00,628 [WARNING] django.request:241 log_response Not Found: /api/v2.1/repos/79ddbe0b-d561-46f1-81ef-efc7c39c1070/dir/

[tomservo]

I hate to see something like this go without a reply, so I will reply even though I don’t have much good news. I think your library is missing files, and the data is gone if you don’t have a backup, based on you reporting that seaf-fsck reports missing blocks.

@Dks

What looks strange to me, is what seaf-gc returns (see below). It says “about 517 reachable blocks”. What does it mean ? According to the 6th line there are 517 blocks in that library. Does it mean the whole library is consistent (so there’s no data corruption) ?

I don’t think that’s what that means. AFAIK, seaf-gc’s job is to look for blocks in the library that aren’t referenced by the library anymore (including files that haven’t yet expired from the library history). It does this using a bloom filter, which is a fancy programming trick to very quickly compare 2 sets of data and find those items in one set that aren’t in the other. One side-effect of using this trick to make the GC fast is that it only compares the sets in one direction (“is this block referred to by the library?”), it doesn’t check the other way (“is there a block referred to by this library that is missing from the disk?”). It is the seaf-fsck that checks for that second question. What I think it is saying is that out of the 517 blocks it checked, they were all used by the library, and there aren’t any that can be removed. It’s possible that there were supposed to be 600, or 900, or any number, gc doesn’t know how many there should be because it isn’t asking that question.

[mlazzarotto]

It is my understanding that the files may still be there, but the database could be corrupted.
You can try to recover your data without relying on the database.
Check this out Exporting Libraries to File System, it’s the official Seafile documentation.
If that doesn’t work, you can try Tutorial: recover your data even if your database is corrupted

[Dks]

Hello guys,

Thank you both for your replies. I still hope that my data is still somewhere around, but I agree that it smells quite bad

I do have “many” backups, but the oldests are in the same state

I’ve just tried to export one of the libraries containing unreadable file, but it can’t find the block

Unfortunately, the seafile_recovery tool (Tutorial: recover your data even if your database is corrupted) is quite old and is using deprecated packages I’ll try to install an older version of golang in a test server

EDIT1: I finally made the seafile_recovery tool working. Unfortunately, it still not able to reach the data I want BUT thanks to the error message :
“2025/06/28 20:01:07 open /seafile/storage/blocks/4fc822a3-42e4-4dbf-8a0d-f122816e572b/d5/234092dbfc9f84bc41baac5d8f54f68916a4df: no such file or directory”

i was able to retrieve this specific block in an older backup. I’m trying to restore it rn