Get commit info failed: commit [...] is missing

Seafile Server
1

Hi,

First i want to thank you for seafile, it’s an excellent piece of software and i am very grateful for it.

I have upgraded from seafile-server-4.2.3 to seafile-server-7.1.5 on a CentOS 6.10 server.
It was intense but i did it, and i am now hunting the last few kinks.

One error in particular is appearing frequently in my logs. Here is a 10 minutes extract :

[11/17/2020 02:40:05 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:40:36 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:41:07 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:41:38 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:42:09 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:42:40 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:43:11 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:43:42 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:44:13 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:44:44 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:45:15 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:45:46 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:46:17 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:46:48 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:47:19 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:47:50 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:48:21 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:48:52 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:49:23 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:49:54 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/17/2020 02:50:25 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.

I would like to find which library raises this commit so i can destroy it and reupload it.

What i have done so far :

  • Ran seaf-gc.sh (-r and normal) without incidents.
  • Ran seaf-fsck.sh --repair without incidents.
  • Dumped the seafile ans seahub databases and searched through the mysql dump for any data containing the commit or part of it but found nothing.
  • Searched the storage folder for the commit but found nothing.

Can anyone advise me why seafile reaches for this particular commit ?
How can i pinpoint the faulty library ?

Thank you very much.
Lewis

2

If you run the garbage collector, it should find the same missing commit when it traverses through the history, and then report it. Then you can pinpoint the correct library, as it runs the garbage collector one library at the time.

3

Just be sure to turn seafile off before running GC as it might remove completely new blocks if you are unlucky.

4

Gronis,

Thank you for taking interest in my issue.
As you can see, i already ran seaf-gc.sh with and without -r option

I will try to run it again, maybe i missed something.

5

Gronis,

I stopped my server and ran seaf-gc.sh again. It finds some errors :

[…]

[11/18/20 12:41:03] gc-core.c(460): The following repos are damaged. You can run seaf-fsck to fix them.
[11/18/20 12:41:03] gc-core.c(463): 2b9cfd0f-e5bb-4593-88e8-272d2ba01bbd
[11/18/20 12:41:03] gc-core.c(463): 367448d0-b872-4d79-9bab-26c5e1d4995d
[11/18/20 12:41:03] gc-core.c(463): 3f21f935-cb27-4af8-8840-237a5e72e200
[11/18/20 12:41:03] gc-core.c(463): 50cd8072-3ff9-41aa-9ac1-15c52521ef73
[11/18/20 12:41:03] gc-core.c(463): 5e35df3b-500a-4e5d-a131-733977d8efcf
[11/18/20 12:41:03] gc-core.c(463): 8e4b86d9-a2e8-4148-860b-a26c83504080
[11/18/20 12:41:03] gc-core.c(463): 9058cdef-763a-44c5-bfd6-19a9d848e34c
[11/18/20 12:41:03] gc-core.c(463): 9193b852-45a0-46c8-b78d-1b0c0deb9314
[11/18/20 12:41:03] gc-core.c(463): 9b2300eb-fc09-45cc-9835-a6aa06cc6286
[11/18/20 12:41:03] gc-core.c(463): a26a2cd7-a4e3-4989-bb42-2707aee958e1
[11/18/20 12:41:03] gc-core.c(463): a69ab6bd-ed3f-42a4-8028-19f000078656
[11/18/20 12:41:03] gc-core.c(463): b13777d3-f286-4d1d-9798-5663510555ec
[11/18/20 12:41:03] gc-core.c(463): b216deac-1e50-4f64-b08d-c4165a3b12ff
[11/18/20 12:41:03] gc-core.c(463): ca582658-8460-4ba6-8f15-04894b4242e1
[11/18/20 12:41:03] gc-core.c(463): d21a1d94-477f-4431-9c29-4acdbd21ff2e
[11/18/20 12:41:03] gc-core.c(463): e548dd5b-e206-40c2-b4d8-299c6bf20f2b
[11/18/20 12:41:03] gc-core.c(463): f1474861-d670-487d-8963-7b1ef5866831
[11/18/20 12:41:03] gc-core.c(463): f8885f08-e120-4825-9571-0869325ebc83
[11/18/20 12:41:03] gc-core.c(463): fbe4df66-ff8a-4d7d-bb86-3387326b72f7
seafserv-gc run done

Done.

I ran seaf-fsck.sh --repair on those libraries once again :

-bash-4.1$ ./seaf-fsck.sh --repair 2b9cfd0f-e5bb-4593-88e8-272d2ba01bbd 367448d0-b872-4d79-bab-26c5e1d4995d 3f21f935-cb27-4af8-8840-237a5e72e200 50cd8072-3ff9-41aa-9ac1-15c52521ef73 5e35df3b-500a-4e5d-a131-733977d8efcf 8e4b86d9-a2e8-4148-860b-a26c83504080 9058cdef-763a-44c5-bfd6-19a9d848e34c 9193b852-45a0-46c8-b78d-1b0c0deb9314 9b2300eb-fc09-45cc-9835-a6aa06cc6286 a26a2cd7-a4e3-4989-bb42-2707aee958e1 a69ab6bd-ed3f-42a4-8028-19f000078656 b13777d3-f286-4d1d-9798-5663510555ec b216deac-1e50-4f64-b08d-c4165a3b12ff ca582658-8460-4ba6-8f15-04894b4242e1 d21a1d94-477f-4431-9c29-4acdbd21ff2e e548dd5b-e206-40c2-b4d8-299c6bf20f2b f1474861-d670-487d-8963-7b1ef5866831 f8885f08-e120-4825-9571-0869325ebc83 fbe4df66-ff8a-4d7d-bb86-3387326b72f7 > /tmp/seaf-fsck.log

Starting seaf-fsck, please wait …

[11/18/20 12:51:22] fsck.c(592): Running fsck for repo 2b9cfd0f-e5bb-4593-88e8-272d2ba01bbd.
[11/18/20 12:51:22] fsck.c(418): Checking file system integrity of repo Poste Restante(2b9cfd0f)…
[11/18/20 12:51:22] fsck.c(656): Fsck finished for repo 2b9cfd0f.

[…]

[11/18/20 13:04:06] fsck.c(592): Running fsck for repo fbe4df66-ff8a-4d7d-bb86-3387326b72f7.
[11/18/20 13:04:06] fsck.c(418): Checking file system integrity of repo Ma bibliothèque(fbe4df66)…
[11/18/20 13:04:38] fsck.c(656): Fsck finished for repo fbe4df66.

seaf-fsck run done

Done.

But when i start the server again, the error is still there:

[11/18/2020 01:08:05 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:08:36 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:09:07 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:09:38 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:10:05 PM] start to serve on pipe client
[11/18/2020 01:10:09 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:10:22 PM] start to serve on pipe client
[11/18/2020 01:10:40 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:11:11 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:11:42 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:12:13 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:12:22 PM] start to serve on pipe client
[11/18/2020 01:12:44 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:13:15 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:13:46 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:14:17 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:14:48 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:15:19 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:15:50 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:16:21 PM] start to serve on pipe client
[11/18/2020 01:16:21 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:16:52 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.
[11/18/2020 01:17:23 PM] http-server.c(1272): Get commit info failed: commit bb36e05ba739f2ca1648de45526d2e7a4e4aed0c is missing.

6

After several retries of the garbage collector and FSCK, things are still the same.

I am back to my original question. How can i track and delete this commit manually when the garbage collector won’t do it ?