Recovering from S3 storage failure?

Hi, I have a problem where my seafile client is not syncing anymore with an “Server Error” log.

seafile logs show:

[12/19/2020 09:23:45 AM] ../common/obj-backend-s3.c(234): Get object 750552b893c28d98d7b3b227ca6a57285a30a0e5 error 412. Response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PreconditionFailed</Code><Message>At least one of the pre-conditions you specified did not hold</Message><Key>9e506b6-8bd2-4087-875c-5fcd7f6eb99c/750552b893c28d98d7b3b227ca6a57285a30a0e5</Key><BucketName>rogeliodh-seafile-fs-objects</BucketName><Resource>/rogeliodh-seafile-fs-objects/79e506b6-8bd2-4087-875c-5fcd7f6eb99c/750552b893c28d98d7b3b227ca6a57285a30a0e5</Resource><RequestId>1652277A85860581</RequestId><HostId>8733a752-f950-4352-af56-91cd85ffd344</HostId></Error>}
[12/19/2020 09:23:45 AM] ../common/fs-mgr.c(1916): [fs mgr] Failed to read dir 750552b893c28d98d7b3b227ca6a57285a30a0e5.

and making a repair fails with:

root@33270eed6e44:/opt/seafile/seafile-server-latest# ./ --repair 79e506b6-8bd2-4087-875c-5fcd7f6eb99c                                                                                                                                                                                                                                        
Starting seaf-fsck, please wait ...    
[12/19/20 10:01:44] fsck.c(606): Running fsck for repo 79e506b6-8bd2-4087-875c-5fcd7f6eb99c.
[12/19/20 10:01:44] fsck.c(431): Checking file system integrity of repo b2_store(79e506b6)...
[12/19/20 10:01:45] ../../common/obj-backend-s3.c(234): Get object 2e06aede655bbe0d73e34901786c7b0a44554c73 error 412. Response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PreconditionFailed</Code><Message>At least one of the pre-conditions you specified did not hold</Message><Key>79e506b6-8bd2-4087-875c-5fcd7f6eb99c/2e06aede655bbe0d73e34901786c7b0a44554c73</Key><BucketName>rogeliodh-seafile-fs-objects</BucketName><Resource>/rogeliodh-seafile-fs-objects/79e506b6-8bd2-4087-875c-5fcd7f6eb99c/2e06aede655bbe0d73e34901786c7b0a44554c73</Resource><RequestId>1652298D7398CACC</RequestId><HostId>8733a752-f950-4352-af56-91cd85ffd344</HostId></Error>
[12/19/20 10:01:45] ../../common/fs-mgr.c(2937): [fs mgr] Failed to read dir 79e506b6-8bd2-4087-875c-5fcd7f6eb99c:2e06aede655bbe0d73e34901786c7b0a44554c73.
[12/19/20 10:01:45] fsck.c(670): Fsck finished for repo 79e506b6.

and, effectively, that object is missing from my S3 storage (not sure why).

Any suggestions on how to recover from this situation?

Unfortuntely, I don’t have database backups. So only ways to fix this would be to re-upload everything or fix the corruption in the storage. I thought fsck would be able to remove the missing object but it does not offer to perform any operation.

I seem to have run into this as well, not sure when maybe sometime last week, last successful change to the library was about 5 days ago.

Edit: restoring an older version of the DB will not help, it may even make it worse (I tried)

@rogeliodh I don’t suppose you are using Minio at all are you possibly even in S3 gateway mode?

If so try rolling it back to RELEASE.2020-12-16T05-05-17Z

I tried setting up nextcloud using Minio as an S3 gateway to Backblaze B2 (want caching) and found it serving 0 byte files, this is when I got suspicious so tried a download on the Minio browser and got an error.

Then I stepped back through the releases till I found one that worked, having found RELEASE.2020-12-16T05-05-17Z fixed my test nextcloud setup I tried rolling back the Minio instances I use between seafile and B2 and then that also fixed my problem with seafile.

Thanks, yes, I’m using minio and had the exact same problem. Reverting to that version, it works. Thanks.

Do you know if it is a known regression? - Essentially the anwser I got was pretty much tough it’s not going to be fixed, Backblaze said they don’t support ETAG so it’s not going anywhere soon.

I stopped using minio/seafile and went back to nextcloud connecting to B2 directly, it’s a shame but seafile didn’t seem to want to talk to B2 directly and whilst I prefer it to nextcloud at least with nextcloud the files are stored directly on B2 so if the integration breaks I can still get them directly off the backend (I don’t bother with NC’s encryption)

that’s a shame, I used minio because seafile authentication to backblaze S3 API didn’t work, now minio as a gateway doesn’t work :frowning:

I’m thinking in returning to nextcloud too…

I came here because I wanted to switch from nextcloud to seafile :smiley:
My idea was this: https:// forum. seafile. com/t/free-cheap-hosted-pro-server-allowing-s3-backend/13937/2 – getting a cheap hosted pro and then connecting backblaze b2. I’ve done that with nextcloud before, but it’s unstable. Regularly the nextcloud client gets stuck; different issues, like filenames too long, a file is being set as read-only (and then can’t be deleted), a bunch of weird errors so you need to manually debug, log into the server, really struggle with deleting files… it’s a pain in the ass. But looks like seafile isn’t a solution either? :smiley:

This is an interesting conclusion in a thread where people used a third party software with an incompatible object storage that pretends to be compatible with S3 but in reality isn’t.

1 Like

Doubt that will work as S3 needs to be configured in multiple settings locations it’s not like external storage in nextcloud, setting Seafile to use S3 makes it use only S3 (I think).

Indeed the problem here is services claiming to be compatible with S3, but in reality they only implement a subset of the S3 features.

That said I do find the minio devs attitude a bit poor, it’s not the first time they’ve made a change that breaks something, it’s not reasonable to expect an end user to know that the functionality they’ve been using the past X months was something that just happened to work but wasn’t following the spec properly.

Fair enough they need to make updates but they could communicate better over such issues, tends to be very much we’re right you’re wrong tough.

1 Like