Corrupted library - how to solve?

Hi,

One of my libraries is not updating, due to one of the files failing to sync:

[10/06/17 19:48:46] http-tx-mgr.c(771): libcurl failed to GET https://domain.com/seafhttp/repo/c2544e0d-a31a-4c0a-9161-d7d58eaad1cc/block/538a5fbd6fc83803a60127e06ed7b499822e7b57: Transferred a partial file.
[10/06/17 19:48:46] repo-mgr.c(5207): Transfer failed.
[10/06/17 19:48:46] http-tx-mgr.c(1132): Transfer repo ‘c2544e0d’: (‘normal’, ‘data’) → (‘error’, ‘finished’)
[10/06/17 19:48:46] sync-mgr.c(764): Repo ‘Projects’ sync state transition from downloading to ‘error’: ‘Error occured in download.’.

All other libraries are syncing OK.

I can see this file located on the server at:

…/seafile-data/storage/blocks/c2544e0d-a31a-4c0a-9161-d7d58eaad1cc/53/8a5fbd6fc83803a60127e06ed7b499822e7b57

I’m guessing that something has got corrupted. Either the file, or some metadata of the file in the database.

So, how can I remove the corrupted file to get the library syncing again?

I actually had this exact same problem a few months ago, but I forgot how I fixed it. Considering it has happened again, I would like to find the optimal way to fix it without risking breaking my installation.

Running seaf-fsck.sh shows no errors. I think that both times I’ve had this error, it’s been after I’ve upgraded Seafile Server. I guess some files integrity were not complete at the moment that Seafile shut down for the upgrade, perhaps? Can’t think of any other reason why it has occured.

At first I would try 2 things:

  1. copy file 8a5fbd6fc83803a60127e06ed7b499822e7b57 to other place to ensure it can be copied.
  2. check the file SHA-1 checksum. It should be 538a5fbd6fc83803a60127e06ed7b499822e7b57
1 Like

Thanks for the reply. The SHA1 checksum is correct, and I can copy the file.

I can also read the contents of the file (MS Word doc, it seems). Also, the permissions of the file is the same as all the other block files.

So I’m really not sure what the issue is - not sure why the Seafile client can’t download this block file, and also not sure why this keeps happening.

Can you download this file with browser? I mean both file from web interface and 8a5fbd6fc83803a60127e06ed7b499822e7b57 chunk.

1 Like

How can I identify which file the chunk is a part of, so that I can find it in the web interface?

For accessing it directly, should I just point my browser to the URL that is being reported as failed in the client log, i.e.:

https://domain.com/seafhttp/repo/c2544e0d-a31a-4c0a-9161-d7d58eaad1cc/block/538a5fbd6fc83803a60127e06ed7b499822e7b57

When I try this, I get a 400 BAD REQUEST status. But, when I change the URL to another block, I also get a 400, so I guess I’m not doing that right.

Thank you for you help.

Update: Some other clients have also got issues. The same file is failing to download on the other clients, which is causing the same library to be out of sync.

But also, on the other clients, some new files created on the client’s device are failing to upload with a 500 status:

[10/08/17 18:40:19] http-tx-mgr.c(3155): Bad response code for PUT https://domain.com/seafhttp/repo/b5247191-b0c6-41da-8665-1c74caa0fda0/block/97cf22876690d487d3e3a9be734dc5a537b113bb: 500.
[10/08/17 18:40:19] http-tx-mgr.c(1132): Transfer repo ‘b5247191’: (‘normal’, ‘data’) → (‘error’, ‘finished’)
[10/08/17 18:40:19] sync-mgr.c(764): Repo ‘Admin’ sync state transition from uploading to ‘error’: ‘Error occured in upload.’.

Anyone have any idea what is going on here? It seems clear that the upgrade to 6.1.9 has broken things, but I’m not sure how/why.

Is there some way I can access support? I have a Pro license but it’s the 9 user special offer, which I don’t think comes with any support.

My installation seems to be completely broken and need to fix it urgently. I don’t mind paying for support. Thanks.

The only way to fix a similar isues i had a year back was to DELETE all local cache of seafile client. So it seems it was a client index corruption and not server corruption for me back than, maybe it is the same for you. Maybe also do reinstall just in case.

By default this is C:\Users\user_name\Seafile folder.

Are there any error in seafile.log at the server side?

No, I don’t see any errors in seafile.log on the server.

I tried uninstalling the Seafile syncing client, and installing the latest version. I removed the Seafile account info (i.e. the seafile-data directory), but did not remove the actual library files. It did not resolve the problem.

This is a wired problem.

Since you can read the content of the file 538a5fbd6fc83803a60127e06ed7b499822e7b57 and it seems to be a word file, can you find the corresponding file and move it from the web interface?

And what’s the size of the block 538a5fbd6fc83803a60127e06ed7b499822e7b57?

There should be some error corresponding to the 500 error. Can you also check the log at the Nginx?

1 Like

Thanks, I checked the Nginx log and it is a permissions error. Permissions of block files are like this:

ls -la /var/www/domain.com/seafile-data/storage/blocks/c2544e0d-a31a-4c0a-9161-d7d58eaad1cc/53/
drwxrwsr-x. 2 seafile www 4096 Oct 8 15:57 .
drwxrwsr-x. 258 seafile www 8192 Oct 29 2016 …
-rw–w----. 1 seafile www 452364 Oct 29 2016 01affc8878efb9103e861a31dc1d18d8fef834
-rw-------. 1 seafile www 1533407 Oct 2 12:11 8a5fbd6fc83803a60127e06ed7b499822e7b57

Seafile runs as seafile, Nginx runs as www. Seafile is also in the www group. As you can see, I have the sticky bit set on the Seafile directory, so that all files get the www group. But Nginx cannot read the blocks as there is no group read permission.

What should the permissions usually be for the storage/blocks directory? I am curious how this happened… it was working OK for about 10 months, and I did a normal upgrade from 6.0.10 to 6.1.9, using the minor upgrade shell script.

Someone please correct me if I am wrong, but nginx does not need direct read/write permissions for the seahub-data folder at all. Nginx uses the proxy_pass statement to send data requests directly to Seahub / seaf-server, who run as the “seafile” user (or whatever user you defined) and by association have the necessary permissions to read the data.

The sticky bit and group read permissions for www are not necessary, and actually dangerous as you are opening up vulnerabilities if Nginx were to be compromised.

Please post your Nginx configuration.

3 Likes

You are right that it’s not necessary, it’s inherited from the /var/www directory. I overwrite that if necessary. However in this case I did not do so because I don’t see it as a big deal - vulnerabilities in Seafile will be far, far easier to exploit than any in Nginx (which is kept up to date).

Nevertheless, the question remains as to what the default file permissions should be for blocks (all blocks in my installation are 0620 and 0600. And also, how they got to these permissions, because I did not change anything except perform the upgrade from 6.0.10 to 6.1.9.

Also, if what you are saying is correct, that the permissions of the Nginx proxy_pass mean that only Seafile needs access, then it does not explain why the following block cannot be read, since it is readable by the Seafile user but not by the Nginx user:

ls -la /var/www/domain.com/seafile-data/storage/blocks/c2544e0d-a31a-4c0a-9161-d7d58eaad1cc/53/8a5fbd6fc83803a60127e06ed7b499822e7b57
-rw-------. 1 seafile www 1533407 Oct 2 12:11 8a5fbd6fc83803a60127e06ed7b499822e7b57

Please provide your Nginx configuration and the log entries that show Nginx with a permissions error.

Redacted Nginx config here: https://ghostbin.com/paste/kujsq

Nginx error log:

2017/10/09 19:31:43 [crit] 11244#0: *132752 open() “/var/lib/nginx/tmp/proxy/0/04/0000023040” failed (13: Permission denied) while reading upstream, client: 192.168.1.1, server: domain.com, request: “GET /seafhttp/repo/c2544e0d-a31a-4c0a-9161-d7d58eaad1cc/block/538a5fbd6fc83803a60127e06ed7b499822e7b57 HTTP/1.1”, upstream: “http://127.0.0.1:8082/repo/c2544e0d-a31a-4c0a-9161-d7d58eaad1cc/block/538a5fbd6fc83803a60127e06ed7b499822e7b57”, host: “domain.com

Ty!

The error nginx is giving is not saying it can’t read the files under seafile-data…, it’s saying it can’t read “/var/lib/nginx/tmp/proxy/0/04/0000023040”.

Are all files under /var/lib/nginx owned by the nginx user?

chown -R www /var/lib/nginx/*

1 Like

Oh wow, I should definitely read more carefully! :flushed:

Yes, that’s it - the timing of the Seafile upgrade must have been coincidental, it seems that an Nginx upgrade botched the permissions of old proxy directory.

Thanks a lot! PM me if you have a way I can send you the equivalent of a beer or a coffee (BTC or whatever).

1 Like