Recovering from S3 storage failure?

#1

Hi, I have a problem where my seafile client is not syncing anymore with an “Server Error” log.

seafile logs show:

[12/19/2020 09:23:45 AM] ../common/obj-backend-s3.c(234): Get object 750552b893c28d98d7b3b227ca6a57285a30a0e5 error 412. Response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PreconditionFailed</Code><Message>At least one of the pre-conditions you specified did not hold</Message><Key>9e506b6-8bd2-4087-875c-5fcd7f6eb99c/750552b893c28d98d7b3b227ca6a57285a30a0e5</Key><BucketName>rogeliodh-seafile-fs-objects</BucketName><Resource>/rogeliodh-seafile-fs-objects/79e506b6-8bd2-4087-875c-5fcd7f6eb99c/750552b893c28d98d7b3b227ca6a57285a30a0e5</Resource><RequestId>1652277A85860581</RequestId><HostId>8733a752-f950-4352-af56-91cd85ffd344</HostId></Error>}
[12/19/2020 09:23:45 AM] ../common/fs-mgr.c(1916): [fs mgr] Failed to read dir 750552b893c28d98d7b3b227ca6a57285a30a0e5.

and making a repair fails with:

root@33270eed6e44:/opt/seafile/seafile-server-latest# ./seaf-fsck.sh --repair 79e506b6-8bd2-4087-875c-5fcd7f6eb99c                                                                                                                                                                                                                                        
Starting seaf-fsck, please wait ...    
[12/19/20 10:01:44] fsck.c(606): Running fsck for repo 79e506b6-8bd2-4087-875c-5fcd7f6eb99c.
[12/19/20 10:01:44] fsck.c(431): Checking file system integrity of repo b2_store(79e506b6)...
[12/19/20 10:01:45] ../../common/obj-backend-s3.c(234): Get object 2e06aede655bbe0d73e34901786c7b0a44554c73 error 412. Response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PreconditionFailed</Code><Message>At least one of the pre-conditions you specified did not hold</Message><Key>79e506b6-8bd2-4087-875c-5fcd7f6eb99c/2e06aede655bbe0d73e34901786c7b0a44554c73</Key><BucketName>rogeliodh-seafile-fs-objects</BucketName><Resource>/rogeliodh-seafile-fs-objects/79e506b6-8bd2-4087-875c-5fcd7f6eb99c/2e06aede655bbe0d73e34901786c7b0a44554c73</Resource><RequestId>1652298D7398CACC</RequestId><HostId>8733a752-f950-4352-af56-91cd85ffd344</HostId></Error>
[12/19/20 10:01:45] ../../common/fs-mgr.c(2937): [fs mgr] Failed to read dir 79e506b6-8bd2-4087-875c-5fcd7f6eb99c:2e06aede655bbe0d73e34901786c7b0a44554c73.
[12/19/20 10:01:45] fsck.c(670): Fsck finished for repo 79e506b6.

and, effectively, that object is missing from my S3 storage (not sure why).

Any suggestions on how to recover from this situation?

Unfortuntely, I don’t have database backups. So only ways to fix this would be to re-upload everything or fix the corruption in the storage. I thought fsck would be able to remove the missing object but it does not offer to perform any operation.

#2

I seem to have run into this as well, not sure when maybe sometime last week, last successful change to the library was about 5 days ago.

Edit: restoring an older version of the DB will not help, it may even make it worse (I tried)

#3

@rogeliodh I don’t suppose you are using Minio at all are you possibly even in S3 gateway mode?

If so try rolling it back to RELEASE.2020-12-16T05-05-17Z

I tried setting up nextcloud using Minio as an S3 gateway to Backblaze B2 (want caching) and found it serving 0 byte files, this is when I got suspicious so tried a download on the Minio browser and got an error.

Then I stepped back through the releases till I found one that worked, having found RELEASE.2020-12-16T05-05-17Z fixed my test nextcloud setup I tried rolling back the Minio instances I use between seafile and B2 and then that also fixed my problem with seafile.

#4

Thanks, yes, I’m using minio and had the exact same problem. Reverting to that version, it works. Thanks.

Do you know if it is a known regression?

#5

https://github.com/minio/minio/issues/11172#event-4150671037 - Essentially the anwser I got was pretty much tough it’s not going to be fixed, Backblaze said they don’t support ETAG so it’s not going anywhere soon.

I stopped using minio/seafile and went back to nextcloud connecting to B2 directly, it’s a shame but seafile didn’t seem to want to talk to B2 directly and whilst I prefer it to nextcloud at least with nextcloud the files are stored directly on B2 so if the integration breaks I can still get them directly off the backend (I don’t bother with NC’s encryption)

#6

that’s a shame, I used minio because seafile authentication to backblaze S3 API didn’t work, now minio as a gateway doesn’t work :frowning:

I’m thinking in returning to nextcloud too…