Seaf-gc suddenly fails collect existing blocks for all repos

Seafile Server
1

Hi there,

probably since the upgrade to server pro 7.0.8 (from 6.x) seaf-gc stopped working. I see errors like this:

[10/28/19 13:06:48] gc-core.c(773): GC version 1 repo Datenhalde(fcd15cb1-65ab-4544-a253-4dbc413edd6c)
[10/28/19 13:06:48] …/…/common/block-backend-ceph.c(630): Failed to open objects list ctx: Success.
[10/28/19 13:06:48] gc-core.c(524): Failed to collect existing blocks for repo fcd15cb1, stop GC.

for all libraries of all users. Furthermore, I see this on my seafile test cluster and my production cluster.

At the same time, seaf-fsck cannot see any problem for all the libraries I checked.

Any hints welcome.

Best,
Hp

2

Hey Jonathan, Daniel,

I really need some help on this.

I get the same error for newly created libraries. so this is clearly not an issue of libraries having corrupt histories.

Best,
Hp

3

We will look into the problem. For urgent problem, you can send to support@seafile.com

4

ok, thank you.

it is not particularly urgent, as everything else works fine, and we have enough storage to let the garbage accumulate a while :wink:

5

Hi @hkunz

We found a possible cause. Have you upgraded your ceph cluster to Iluminous? As their release note stated: https://ceph.io/releases/v12-2-0-luminous-released/, the API we used to list objects are removed.

6

no, we changed nothing on our ceph cluster. it is running jewel.

7

could it be that our ceph version (10.2.11) is too old for seafile 7?

8

What happens if you execute this command: seaf-gc.sh -r ? Same error?

9

I get no error running seaf-gc.sh -r. but there is no deleted library, so nothing really happens. and it does not solve our problem, of course :slight_smile:

10

Of course, i just wanted to check your feedback, as we use ceph as well - and maybe i can help. As this option will call another part of the script, this feedback may help the devs. Can you try to delete a library (and library trash) and try it again?

11

I see. Just tried what you suggested (deleted a library, cleared the trash). When I run seaf-gc the first time I get:

[11/11/19 23:46:49] gc-core.c(673): === Repos deleted by users ===
[11/11/19 23:46:49] gc-core.c(697): Start to GC deleted repo 50aac6a2-c77e-48bd-9f33-e6ab11355e02.
[11/11/19 23:46:49] gc-core.c(643): Deleting commits for repo 50aac6a2-c77e-48bd-9f33-e6ab11355e02.
[11/11/19 23:46:49] …/…/common/obj-backend-ceph.c(467): Failed to open objects list ctx: Success.

nevertheless, the library (or at least the “reference” to it) was removed. If I run seaf-gc again, it does not try to remove the library.

Thanks,
Hp

12

What OS do you use? We can provide you with a debug package.

13

debian buster

14

Hi @hkunz

Please try this package and send the log message: https://download.seafile.com/f/78d0471ffd774a62bca7/?dl=1

15

Hi @Jonathan,

I untared the package and just ran seaf-gc.sh from it (without upgrading the server itself, i.e. seafile and seahub services running were still 7.0.8).

But seaf-gc.sh worked without errors! Many thanks!

Did you fix something in the package you provided? Or can I just upgrade to the “normal” seafile 7.0.10?

Best,
Hp

16

Have you installed librados in your system? What’s the version?

The reason that this package works for you may be that it uses the bundled librados from the package. (You haven’t removed them from the package.)

Which Seafile Pro package do you use? The version with Ubuntu in the name or not?

17

I have installed

librados2, version 12.2.11+dfsg1-2.1+b1

indeed, I remove the bundled rados libraries (but not in the new version of seafile you sent me). so, your theory is correct :slight_smile:

so, was this a mistake? should seafile use the bundled rados libraries?

are there some general guidelines which bundled libraries to remove and which not? I remember I had to remove the ldap libraries for example, in orther to get ldap auth working.

I always used the seafile pro package without ubuntun in its name.

Best,
Hp

18

Hi @Jonathan,

could you give me a hint here? (when to remove bundled libraries and when not to)

Best,
Hp

19

Hi @hkunz

If you use Debian, you should now use the package with Ubuntu in the name. With this you don’t need to remove the bundled libraries.

20

perfekt :slight_smile: