Random "permission denied" error

Hi,

I’m using Seafile on multiple situations, and it’s running quite well.
But since a week there is some permission denied errors that occurs randomly.

When I run seaf-fsck I get this :

[05/08/18 23:01:00] fsck.c(586): Running fsck for repo 1d8ac55f-b21e-478c-8794-e2ab76db9c8d.
[05/08/18 23:01:00] fsck.c(413): Checking file system integrity of repo tek kxkm(1d8ac55f)...
[05/08/18 23:01:01] ../../common/block-backend-fs.c(72): [block bend] failed to open block fd7a4b830efaa96204de4328f8e808fa7498f43c for read: Permission denied
[05/08/18 23:01:01] ../../common/block-mgr.c(232): Failed to open block 1d8ac55f-b21e-478c-8794-e2ab76db9c8d:fd7a4b83.
[05/08/18 23:01:01] fsck.c(650): Fsck finished for repo 1d8ac55f.

And the seafile.log :
[05/09/2018 12:00:32 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block fd7a4b830efaa96204de4328f8e808fa7498f43c for read: Permission denied [05/09/2018 12:00:32 AM] access-file.c(266): Failed to open block 1d8ac55f-b21e-478c-8794-e2ab76db9c8d:fd7a4b830efaa96204de4328f8e808fa7498f43c

A simple seaf-fsck.sh -r $lib_id solve the issue, but the problem come back latter, on this library or on another.

I’m running seafile-server6.2.5 on linux (debian 9)
The seafile-data directory is stored over a NFS disk, but permission seems to be fine.

From the root directory of seafile (on the same level as logs, conf etc…) running find ./ \! -readable give no output.

Do you have any idea ?
Thanks,
Olivier.

Are you using the stock database included with Seafile, or are you using another database?

Hi,

I’m using SQLite as described here : https://manual.seafile.com/deploy/using_sqlite.html
Do you think that it’s not related to file permission but some corruption in the database ?

An error occurred 10 min ago, and I’ve found something new.

Error on lib : 4dc5181d-6e0c-4551-8a7e-5ef846de7f56
Running : ./seaf-fsck -r 4dc5181d-6e0c-4551-8a7e-5ef846de7f56

[05/09/18 10:37:38] fsck.c(413): Checking file system integrity of repo Budgets(4dc5181d)...
[05/09/18 10:37:38] ../../common/block-backend-fs.c(72): [block bend] failed to open block 994644295d56efcd27f7f64b37c6a70de19ef02a for read: Permission denied
[05/09/18 10:37:38] ../../common/block-mgr.c(232): Failed to open block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:99464429.
[05/09/18 10:37:38] fsck.c(650): Fsck finished for repo 4dc5181d.

So nothing solved…
But after that I’ve tried to check if a file in the seafile-data directory isn’t readable : find ./ \! -readable
And I get nothing…

Just check if the problem is still here (without trying to repair) : ./seaf-fsck 4dc5181d-6e0c-4551-8a7e-5ef846de7f56

[05/09/18 10:39:30] fsck.c(586): Running fsck for repo 4dc5181d-6e0c-4551-8a7e-5ef846de7f56.
[05/09/18 10:39:30] fsck.c(413): Checking file system integrity of repo Budgets(4dc5181d)...
[05/09/18 10:39:30] fsck.c(650): Fsck finished for repo 4dc5181d. 

The problem gone… without repair, just after the find command.
Does someone has a clue about what is going on ?

Seafile version : 6.2.5
System : GNU/Linux Debian 9
Database : SQLite
Seafile-data storage location : NFS

There are a lot of things that can cause “permission denied” errors. It’s just a matter of narrowing down what the problem is. Right now, without more information, it’s hard to say, but it does not sound like a permissions problem with the file itself, especially since it’s working now. But, it could happen again. Log files are our best shot right now, including system logs that could indicate a problem elsewhere in your system.

I had asked about the database program you are using so that we might look in those log files for clues, but you are using the stock database.

We just need to keep an eye on it and see if we can track it down.

An error occurred again…

Here is the last part of my seafile.log :

[05/09/2018 10:34:42 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:35:02 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:7b1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3 at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/7b/1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3: Permission denied.
[05/09/2018 10:35:02 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:35:13 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:35:33 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:7b1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3 at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/7b/1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3: Permission denied.
[05/09/2018 10:35:33 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:35:45 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:36:04 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:7b1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3 at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/7b/1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3: Permission denied.
[05/09/2018 10:36:04 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:36:16 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:36:35 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:7b1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3 at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/7b/1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3: Permission denied.
[05/09/2018 10:36:35 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:36:47 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:36:59 AM] size-sched.c(96): Repo size compute queue size is 0
[05/09/2018 10:37:06 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:7b1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3 at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/7b/1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3: Permission denied.
[05/09/2018 10:37:06 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:37:18 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:37:37 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:7b1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3 at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/7b/1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3: Permission denied.
[05/09/2018 10:37:37 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:37:49 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:38:08 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:7b1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3 at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/7b/1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3: Permission denied.
[05/09/2018 10:38:08 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:38:20 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:38:39 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:7b1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3 at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/7b/1f41bd5c3c79588cf6c0e6dedb84ef182eb3e3: Permission denied.
[05/09/2018 10:38:39 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:38:51 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block 4dc5181d-6e0c-4551-8a7e-5ef846de7f56:994644295d56efcd27f7f64b37c6a70de19ef02a at /srv/seafile/nas-nfs/seafile-data/storage/blocks/4dc5181d-6e0c-4551-8a7e-5ef846de7f56/99/4644295d56efcd27f7f64b37c6a70de19ef02a: Permission denied.
[05/09/2018 10:41:59 AM] size-sched.c(96): Repo size compute queue size is 0
[05/09/2018 10:46:59 AM] size-sched.c(96): Repo size compute queue size is 0
[05/09/2018 10:51:59 AM] size-sched.c(96): Repo size compute queue size is 0
[05/09/2018 10:56:59 AM] size-sched.c(96): Repo size compute queue size is 0
[05/09/2018 11:00:32 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 3fce8b6578a2aad2a1e36598fccda01f56cfeb51 for read: Permission denied
[05/09/2018 11:00:32 AM] access-file.c(266): Failed to open block f266a30e-0bd2-40d6-ae9e-e4b90d8eff39:3fce8b6578a2aad2a1e36598fccda01f56cfeb51
[05/09/2018 11:00:47 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 4f27565a19555afcce877485acb44a147b3bd08e for read: Permission denied
[05/09/2018 11:00:47 AM] access-file.c(266): Failed to open block 04fba02d-809c-420b-896f-923d721bea9d:4f27565a19555afcce877485acb44a147b3bd08e
[05/09/2018 11:01:59 AM] size-sched.c(96): Repo size compute queue size is 0
[05/09/2018 11:02:25 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 4f27565a19555afcce877485acb44a147b3bd08e for read: Permission denied
[05/09/2018 11:02:25 AM] access-file.c(266): Failed to open block 04fba02d-809c-420b-896f-923d721bea9d:4f27565a19555afcce877485acb44a147b3bd08e
[05/09/2018 11:02:55 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 3fce8b6578a2aad2a1e36598fccda01f56cfeb51 for read: Permission denied
[05/09/2018 11:02:55 AM] access-file.c(266): Failed to open block f266a30e-0bd2-40d6-ae9e-e4b90d8eff39:3fce8b6578a2aad2a1e36598fccda01f56cfeb51
[05/09/2018 11:06:59 AM] size-sched.c(96): Repo size compute queue size is 0 

Is it possible that the problem is related with Collabora ?

Nothing interesting on my system logs …

You mean Collabora Office?

Looking at your log files, it does appear quite random. Is this happening while Seafile is idle, or while you are trying to sync/upload? What clients are you utilizing? What version?

The NFS you are using… Are you using RAID?

Anything you can tell us about your setup may be helpful. Did it work fine prior to a week ago?

Do you monitor the NFS response times? Could there be timeouts?

Just to make sure: Did you run the command with the same user you run seafile with?

Hi,

Thanks for your time.

You mean Collabora Office?

I mead CODE, so the Oline version of Collabora based on LibreOffice Online. It runs on a docker container on the same server.

Here is the head of my log file for each line with “fail” grep -i "fail" seafile.log | head

[04/22/2018 03:11:31 PM] repo-op.c(5163): Failed to get commit a31f496e-0383-4e54-8720-731a3e9d2d61:(null).
[04/23/2018 04:58:59 PM] ../common/seaf-db.c(887): Error exec prep stmt: sqlite3_step failed: database table is locked.
[05/04/2018 09:46:27 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block ec7b2c7962423b325e37d99099f8c6f8918528ab for read: Permission denied
[05/04/2018 09:46:27 AM] access-file.c(266): Failed to open block beb29d67-55c1-43f8-9897-5929393d6779:ec7b2c7962423b325e37d99099f8c6f8918528ab
[05/04/2018 09:47:00 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block ec7b2c7962423b325e37d99099f8c6f8918528ab for read: Permission denied
[05/04/2018 09:47:00 AM] access-file.c(266): Failed to open block beb29d67-55c1-43f8-9897-5929393d6779:ec7b2c7962423b325e37d99099f8c6f8918528ab
[05/04/2018 09:47:38 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block ec7b2c7962423b325e37d99099f8c6f8918528ab for read: Permission denied
[05/04/2018 09:47:38 AM] access-file.c(266): Failed to open block beb29d67-55c1-43f8-9897-5929393d6779:ec7b2c7962423b325e37d99099f8c6f8918528ab
[05/04/2018 09:47:51 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5 for read: Permission denied
[05/04/2018 09:47:51 AM] access-file.c(266): Failed to open block beb29d67-55c1-43f8-9897-5929393d6779:9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5
[05/04/2018 10:46:13 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 92657af15b00a5b3f2845c9e380368b9b6279068 for read: Permission denied
[05/04/2018 10:46:13 AM] access-file.c(266): Failed to open block 04fba02d-809c-420b-896f-923d721bea9d:92657af15b00a5b3f2845c9e380368b9b6279068
[05/04/2018 10:46:24 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 92657af15b00a5b3f2845c9e380368b9b6279068 for read: Permission denied
[05/04/2018 10:46:24 AM] access-file.c(266): Failed to open block 04fba02d-809c-420b-896f-923d721bea9d:92657af15b00a5b3f2845c9e380368b9b6279068
[05/04/2018 10:48:33 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 92657af15b00a5b3f2845c9e380368b9b6279068 for read: Permission denied
[05/04/2018 10:48:33 AM] access-file.c(266): Failed to open block 04fba02d-809c-420b-896f-923d721bea9d:92657af15b00a5b3f2845c9e380368b9b6279068
[05/04/2018 10:50:33 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 92657af15b00a5b3f2845c9e380368b9b6279068 for read: Permission denied
[05/04/2018 10:50:33 AM] access-file.c(266): Failed to open block 04fba02d-809c-420b-896f-923d721bea9d:92657af15b00a5b3f2845c9e380368b9b6279068
[05/04/2018 10:52:35 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 92657af15b00a5b3f2845c9e380368b9b6279068 for read: Permission denied
[05/04/2018 10:52:35 AM] access-file.c(266): Failed to open block 04fba02d-809c-420b-896f-923d721bea9d:92657af15b00a5b3f2845c9e380368b9b6279068
[05/04/2018 10:54:36 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 92657af15b00a5b3f2845c9e380368b9b6279068 for read: Permission denied
[05/04/2018 10:54:36 AM] access-file.c(266): Failed to open block 04fba02d-809c-420b-896f-923d721bea9d:92657af15b00a5b3f2845c9e380368b9b6279068
[05/04/2018 11:05:50 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5 for read: Permission denied
[05/04/2018 11:05:50 AM] access-file.c(266): Failed to open block beb29d67-55c1-43f8-9897-5929393d6779:9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5
[05/04/2018 11:06:01 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block ec7b2c7962423b325e37d99099f8c6f8918528ab for read: Permission denied
[05/04/2018 11:06:01 AM] access-file.c(266): Failed to open block beb29d67-55c1-43f8-9897-5929393d6779:ec7b2c7962423b325e37d99099f8c6f8918528ab
[05/04/2018 11:11:26 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5 for read: Permission denied
[05/04/2018 11:11:26 AM] access-file.c(266): Failed to open block beb29d67-55c1-43f8-9897-5929393d6779:9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5
[05/04/2018 11:11:51 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5 for read: Permission denied
[05/04/2018 11:11:51 AM] access-file.c(266): Failed to open block beb29d67-55c1-43f8-9897-5929393d6779:9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5
[05/04/2018 11:14:25 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5 for read: Permission denied
[05/04/2018 11:14:25 AM] access-file.c(266): Failed to open block beb29d67-55c1-43f8-9897-5929393d6779:9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5
[05/04/2018 11:14:54 AM] ../common/block-backend-fs.c(72): [block bend] failed to open block 9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5 for read: Permission denied
[05/04/2018 11:14:54 AM] access-file.c(266): Failed to open block beb29d67-55c1-43f8-9897-5929393d6779:9c939dca8b78900ac0bf0b7e8e6d41fdffde94b5
[05/04/2018 11:15:58 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block beb29d67-55c1-43f8-9897-5929393d6779:91417b391c9c6fbc3037b85d040c5d07a74cdb69 at /srv/seafile/nas-nfs/seafile-data/s$
[05/04/2018 11:16:02 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block beb29d67-55c1-43f8-9897-5929393d6779:91417b391c9c6fbc3037b85d040c5d07a74cdb69 at /srv/seafile/nas-nfs/seafile-data/s$
[05/04/2018 11:16:07 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block beb29d67-55c1-43f8-9897-5929393d6779:91417b391c9c6fbc3037b85d040c5d07a74cdb69 at /srv/seafile/nas-nfs/seafile-data/s$
[05/04/2018 11:16:12 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block beb29d67-55c1-43f8-9897-5929393d6779:91417b391c9c6fbc3037b85d040c5d07a74cdb69 at /srv/seafile/nas-nfs/seafile-data/s$
[05/04/2018 11:16:17 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block beb29d67-55c1-43f8-9897-5929393d6779:91417b391c9c6fbc3037b85d040c5d07a74cdb69 at /srv/seafile/nas-nfs/seafile-data/s$
[05/04/2018 11:16:22 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block beb29d67-55c1-43f8-9897-5929393d6779:91417b391c9c6fbc3037b85d040c5d07a74cdb69 at /srv/seafile/nas-nfs/seafile-data/s$
[05/04/2018 11:16:27 AM] ../common/block-backend-fs.c(225): [block bend] Failed to stat block beb29d67-55c1-43f8-9897-5929393d6779:91417b391c9c6fbc3037b85d040c5d07a74cdb69 at /srv/seafile/nas-nfs/seafile-data/s$

So the problem is quite new (nothing similar before 04 may).

NFS storage is done by a NAS Synology over a SHR RAID (done by Synology, quite similar to RAID 5).
Permission are :
Client : 10.2.0.242

  • Privilege = R/W
  • Squash = No Mapping
  • Asynchronous = Yes
  • Non-Privileged port = Allowed
  • Cross-mount = Denied

Systemd mount file for NFS :

[Unit]
Description=Mount NAS Seafile at boot
Requires=systemd-networkd.service
After=network-online.target
Wants=network-online.target

[Mount]
What=10.2.0.4:/volume1/Seafile
Where=/srv/seafile/nas-nfs
Type=nfs
TimeoutSec=30

[Install]
WantedBy=multi-user.target

Do you monitor the NFS response times? Could there be timeouts?

Not directly, but I monitor general I/O on my system and there is nothing strange, perf seems good.
Maybe I will use something like NFS collectd plugin or nfsiostat in the future.

Just to make sure: Did you run the command with the same user you run seafile with?

Yep, all is running with the seafile user (kxkm) which is an unprivileged user.

Does the file seafile-data/storage/blocks/beb29d67-55c1-43f8-9897-5929393d6779/9c/939dca8b78900ac0bf0b7e8e6d41fdffde94b5 exist?

And seafile-data/storage/blocks/beb29d67-55c1-43f8-9897-5929393d6779/91/417b391c9c6fbc3037b85d040c5d07a74cdb69 ?

(Please notice, that I added a folder to the first path and added the second one)

You mean : seafile-data/storage/blocks/beb29d67-55c1-43f8-9897-5929393d6779/9c/939dca8b78900ac0bf0b7e8e6d41fdffde94b5 ? (I think you’ve forgot the library id)

So yes, this file exist :

touch seafile-data/storage/blocks/beb29d67-55c1-43f8-9897-5929393d6779/9c/939dca8b78900ac0bf0b7e8e6d41fdffde94b5 && echo "OK"
OK

(Same thing for the second path)

I’m a bit confused now. touch creates the file if it doesn’t exist and updates it’s timestamp if it does exist, doesn’t it? So the above command always returns ok?

I’d check with ls -l. Size should be larger than 0 (otherwise touch created the file).

I’m a bit confused now. Touch creates the file if it doesn’t exist and updates it’s timestamp if it does exist, doesn’t it? So the above command always returns ok?

You’re totally right ! touch is not the command to use here, so to be sure that the file already exist (before my touch command) I’ve just check the size with ls and it seems that there aren’t empty new files :

ls -lah seafile-data/storage/blocks/beb29d67-55c1-43f8-9897-5929393d6779/91/417b391c9c6fbc3037b85d040c5d07a74cdb69
-rw------- 1 kxkm kxkm 102K mai    9 12:10 seafile-data/storage/blocks/beb29d67-55c1-43f8-9897-5929393d6779/91/417b391c9c6fbc3037b85d040c5d07a74cdb69

Ok, at least for now I don’t have another idea. Maybe someone else has.

As for Collabra, I can’t be certain. Unfortunately, I don’t use Collabra, so I have no experience with it. But, being that it’s an online Office product, I don’t see how it could affect the Seafile blocks.

I’m leaning toward it having something to do with the NFS, at this point, since the OS is what is reporting the “permission denied” error, which is simply being passed to the Seafile function, and also because nothing new was added or changed on the system on May 4, or close thereby.

Anyway, my insomnia bout is over now and I must get rest. You are in good hands with @shoeper, and hopefully he can nail it down for you while I’m sleeping. :wink: I’ll keep an eye on the thread, as I’m interested in finding out what’s going on.

Having a look at that, it looks like some blocks are not being written properly to the NFS.

Have a good rest and thanks for your help !

I agree that it’s seems to be a NFS problem but… why now ? The current installation is working like a charm since more than a month.

How can I investigate deeply the NFS service to see what’s going on ?

Could be a hardware issue interfering with file operations, or even a cable. A fsck (Linux’s fsck) on the NFS might reveal something. Could be as simple as a faulty file system needing repaired.

Anyway, have a good day!

Maybe this helps: https://wiki.archlinux.org/index.php/NFS/Troubleshooting

If you’re using async I’d try using sync (as with async there is no guarantee for the client (seafile) whether a file has actually been written to disk or not).

(I’m not using NFS. )