Recovering from S3 storage failure?

Hi, I have a problem where my seafile client is not syncing anymore with an “Server Error” log.

seafile logs show:

[12/19/2020 09:23:45 AM] ../common/obj-backend-s3.c(234): Get object 750552b893c28d98d7b3b227ca6a57285a30a0e5 error 412. Response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PreconditionFailed</Code><Message>At least one of the pre-conditions you specified did not hold</Message><Key>9e506b6-8bd2-4087-875c-5fcd7f6eb99c/750552b893c28d98d7b3b227ca6a57285a30a0e5</Key><BucketName>rogeliodh-seafile-fs-objects</BucketName><Resource>/rogeliodh-seafile-fs-objects/79e506b6-8bd2-4087-875c-5fcd7f6eb99c/750552b893c28d98d7b3b227ca6a57285a30a0e5</Resource><RequestId>1652277A85860581</RequestId><HostId>8733a752-f950-4352-af56-91cd85ffd344</HostId></Error>}
[12/19/2020 09:23:45 AM] ../common/fs-mgr.c(1916): [fs mgr] Failed to read dir 750552b893c28d98d7b3b227ca6a57285a30a0e5.

and making a repair fails with:

root@33270eed6e44:/opt/seafile/seafile-server-latest# ./seaf-fsck.sh --repair 79e506b6-8bd2-4087-875c-5fcd7f6eb99c                                                                                                                                                                                                                                        
Starting seaf-fsck, please wait ...    
[12/19/20 10:01:44] fsck.c(606): Running fsck for repo 79e506b6-8bd2-4087-875c-5fcd7f6eb99c.
[12/19/20 10:01:44] fsck.c(431): Checking file system integrity of repo b2_store(79e506b6)...
[12/19/20 10:01:45] ../../common/obj-backend-s3.c(234): Get object 2e06aede655bbe0d73e34901786c7b0a44554c73 error 412. Response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PreconditionFailed</Code><Message>At least one of the pre-conditions you specified did not hold</Message><Key>79e506b6-8bd2-4087-875c-5fcd7f6eb99c/2e06aede655bbe0d73e34901786c7b0a44554c73</Key><BucketName>rogeliodh-seafile-fs-objects</BucketName><Resource>/rogeliodh-seafile-fs-objects/79e506b6-8bd2-4087-875c-5fcd7f6eb99c/2e06aede655bbe0d73e34901786c7b0a44554c73</Resource><RequestId>1652298D7398CACC</RequestId><HostId>8733a752-f950-4352-af56-91cd85ffd344</HostId></Error>
[12/19/20 10:01:45] ../../common/fs-mgr.c(2937): [fs mgr] Failed to read dir 79e506b6-8bd2-4087-875c-5fcd7f6eb99c:2e06aede655bbe0d73e34901786c7b0a44554c73.
[12/19/20 10:01:45] fsck.c(670): Fsck finished for repo 79e506b6.

and, effectively, that object is missing from my S3 storage (not sure why).

Any suggestions on how to recover from this situation?

Unfortuntely, I don’t have database backups. So only ways to fix this would be to re-upload everything or fix the corruption in the storage. I thought fsck would be able to remove the missing object but it does not offer to perform any operation.

I seem to have run into this as well, not sure when maybe sometime last week, last successful change to the library was about 5 days ago.

Edit: restoring an older version of the DB will not help, it may even make it worse (I tried)

@rogeliodh I don’t suppose you are using Minio at all are you possibly even in S3 gateway mode?

If so try rolling it back to RELEASE.2020-12-16T05-05-17Z

I tried setting up nextcloud using Minio as an S3 gateway to Backblaze B2 (want caching) and found it serving 0 byte files, this is when I got suspicious so tried a download on the Minio browser and got an error.

Then I stepped back through the releases till I found one that worked, having found RELEASE.2020-12-16T05-05-17Z fixed my test nextcloud setup I tried rolling back the Minio instances I use between seafile and B2 and then that also fixed my problem with seafile.

Thanks, yes, I’m using minio and had the exact same problem. Reverting to that version, it works. Thanks.

Do you know if it is a known regression?

https://github.com/minio/minio/issues/11172#event-4150671037 - Essentially the anwser I got was pretty much tough it’s not going to be fixed, Backblaze said they don’t support ETAG so it’s not going anywhere soon.

I stopped using minio/seafile and went back to nextcloud connecting to B2 directly, it’s a shame but seafile didn’t seem to want to talk to B2 directly and whilst I prefer it to nextcloud at least with nextcloud the files are stored directly on B2 so if the integration breaks I can still get them directly off the backend (I don’t bother with NC’s encryption)

that’s a shame, I used minio because seafile authentication to backblaze S3 API didn’t work, now minio as a gateway doesn’t work :frowning:

I’m thinking in returning to nextcloud too…