Seafile is continously corrupting files

Hello,

we are using Seafile in our company. For a while now we are facing the issue that Seafile is repeatedly corrupting files and we are unsure how its happening. We are using the repository and at some point the client will give an error message for the repo. I then go to the server and run seaf-fsck.sh on the repo in question and it removes the corrupted files and we have to resync all clients. It is happening like once per day now.

This is becoming more and more of a problem because we are losing the corrupted files in the process. We already switched the server machine that seafile is running on to make sure it is not a hardware issue. We also switched an external HDD that seafile was being used on by one of our team members, but the problem persists.

All clients use client version 6.0.4 on Windows 7, Windows 10 or Mac OS X.

The server is currently 6.0.9 but I believe we did have the problem in earlier version as well. Can anyone help? This is driving us mad.

Thanks

From my point of view this a very likely an issue with the server hardware or configuration. I run Seafile here for more than two years now without a single corruption or failure.

Can you give us more details about your setup and which hardware is being used?

Sure, this is a dedicated Ubuntu 16.04 machine which is exclusively used by Seafile. Specs:
Intel® Xeon®
E3-1230v3
4 x 3,3 GHz
16 GB RAM
2 x 2 TB in Raid 1

As I said we already switched the machine and upgrade to this one last week in case of a hardware issue, but this did not help.

We configured seafile to use nginx with https. The maximum concurrent users on the repos is 3 people. I was suspecting that the Mac ._ files could cause a problem, today I created a seafile-ignore.txt with the content ._* to prevent hidden Mac files from being synced.

Let me know if any info is missing.

Which file system do you use and have there been any power outages?

Could you share some logs? One the server they can be found in your “seafile directory”/logs on the clients there is an option to open the log folder. If you don’t want to share them publicly you can also send me a link as PM (or upload here - max upload rate is 1,6 MiB/s). Zipping the logs reduces their size significantly.

Hi,

sorry for the delay with the easter holidays. So no, the servers did not have any power outages, the second machine is very fresh and there were absolutely no interruptions of power or any other hardware failures and still it is corrupting files.

We did see another corruption today, here are some logs (only including lines from today):

SERVER:

ccnet.log

[04/19/17 01:27:48] …/common/peer.c(943): Local peer down
[04/19/17 07:07:50] …/common/session.c(398): Accepted a local client
[04/19/17 07:07:50] …/common/session.c(398): Accepted a local client
[04/19/17 07:07:50] …/common/peer.c(943): Local peer down
[04/19/17 07:50:23] …/common/session.c(398): Accepted a local client
[04/19/17 07:50:23] …/common/session.c(398): Accepted a local client
[04/19/17 07:50:23] …/common/peer.c(943): Local peer down
[04/19/17 07:50:23] …/common/peer.c(943): Local peer down
[04/19/17 08:28:24] …/common/session.c(398): Accepted a local client
[04/19/17 08:28:24] …/common/peer.c(943): Local peer down
[04/19/17 08:33:22] …/common/session.c(398): Accepted a local client
[04/19/17 08:33:22] …/common/peer.c(943): Local peer down
[04/19/17 08:49:53] …/common/session.c(398): Accepted a local client
[04/19/17 08:49:53] …/common/session.c(398): Accepted a local client
[04/19/17 08:49:53] …/common/peer.c(943): Local peer down
[04/19/17 08:49:53] …/common/peer.c(943): Local peer down
[04/19/17 09:38:14] …/common/session.c(398): Accepted a local client
[04/19/17 09:38:14] …/common/peer.c(943): Local peer down

controller.log:
no entries for today

seafile.log (this looks promising):

[04/19/2017 09:41:32 AM] …/common/fs-mgr.c(2722): dir /marketing/store/doko/screens/version3 doesn’t exist in repo 7cded088.
[04/19/2017 12:39:36 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 12:41:37 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 12:43:35 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 12:45:37 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 12:47:44 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 12:50:03 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 12:52:26 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 12:54:21 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 12:56:21 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 12:58:32 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:00:44 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:02:43 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:05:52 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:08:28 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:11:50 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:14:17 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:16:36 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:19:59 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:20:06 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:22:49 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:24:21 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:25:17 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:27:34 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:28:14 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:30:01 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:31:00 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:32:55 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:33:53 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:35:43 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:36:23 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:38:52 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:41:14 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:41:51 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:44:46 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:45:13 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:48:32 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:49:35 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:50:57 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:54:20 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:54:36 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:57:54 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 01:58:34 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:01:21 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:01:53 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:05:00 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:05:06 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:08:46 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:09:11 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:12:27 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:12:48 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:15:24 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:16:29 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:20:01 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:21:07 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:23:02 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:24:27 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:25:20 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:27:40 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:28:39 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:30:33 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:32:34 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:34:12 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:36:09 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:37:30 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:39:08 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:41:33 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:41:52 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:44:50 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:46:07 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:47:14 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:50:09 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:50:16 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:53:15 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:53:57 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:57:19 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 02:57:19 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:00:23 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:00:31 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:03:07 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:03:12 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:06:35 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:06:45 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:08:10 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:09:58 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:10:15 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:11:43 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:13:20 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:13:55 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:15:09 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:23:25 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:23:31 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:25:17 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:25:37 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:27:25 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:27:35 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:29:21 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:29:33 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:31:18 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:31:39 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:33:10 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:33:29 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:35:10 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:35:14 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:36:53 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:37:20 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:38:41 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:39:02 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:40:40 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:41:26 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:41:50 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:43:41 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:43:44 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:45:26 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:45:30 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:47:24 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:47:34 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:49:12 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:49:13 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:51:03 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:51:10 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:53:04 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:53:18 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:55:05 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.
[04/19/2017 03:55:35 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.

seahub.log:
empty

seahub.django_request.log:

2017-04-19 06:20:20,904 [WARNING] django.request:170 get_response Not Found: /cgi-bin/webcm
2017-04-19 09:19:07,048 [WARNING] django.request:170 get_response Not Found: /plugins/weathermap/editor.php

MY CLIENT (Windows 7):

applet.log:

[04/19/17 10:04:46]starting ccnet: (“-c”, “C:/Users/Ruben/ccnet”)
[04/19/17 10:04:47]trying to connect to ccnet daemon…

[04/19/17 10:04:47]connected to ccnet daemon

[04/19/17 10:04:48]starting seaf-daemon: (“-c”, “C:/Users/Ruben/ccnet”, “-d”, “E:/Seafile/seafile-data”, “-w”, “E:/Seafile”)
[04/19/17 10:04:49][Rpc Client] connected to daemon
[04/19/17 10:04:49][Rpc Client] connected to daemon
[04/19/17 10:04:49][MessageListener] connected to daemon
[04/19/17 10:04:49]Unable to get config value download_limit: Config not exists
[04/19/17 10:04:49]Unable to get config value upload_limit: Config not exists
[04/19/17 10:04:49]Starting the network status detector
[04/19/17 10:04:49][Rpc Client] connected to daemon
[04/19/17 10:04:49][Rpc Client] connected to daemon
[04/19/17 10:04:51]The latest version is 5.0.4
[04/19/17 16:17:32][network detector] got a network failure
[04/19/17 16:17:32][api] network error for URL/api2/account/info/: Connection timed out

[04/19/17 16:17:32][api] network error for URL/api2/ping/: Connection timed out

[04/19/17 16:17:50][network detector] resetting the qt network access manager
[04/19/17 16:18:42][network detector] got a network failure
[04/19/17 16:18:42][api] network error for URL/api2/ping/: Connection timed out

[04/19/17 16:18:56][network detector] resetting the qt network access manager
[04/19/17 16:19:42][network detector] got a network failure
[04/19/17 16:19:42][api] network error for URL/api2/ping/: Connection timed out

[04/19/17 16:20:02][network detector] resetting the qt network access manager

ccnet.log:

[04/19/17 10:04:46] ccnet-daemon.c(193): starting ccnet client 6.0.4
[04/19/17 10:04:46] ccnet-daemon.c(195): ccnet source code version e9995b4c40132f9678c4ee354253d8524638ddd7
[04/19/17 10:04:46] …/common/session.c(132): using config file C:/Users/Ruben/ccnet\ccnet.conf
[04/19/17 10:04:46] …/common/session.c(418): Listen on 127.0.0.1 13419
[04/19/17 10:04:46] …/common/session.c(290): Update pubinfo file
[04/19/17 10:04:47] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:48] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:48] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:48] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:49] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:49] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:49] …/common/peer.c(943): Local peer down
[04/19/17 10:04:49] …/common/peer.c(943): Local peer down
[04/19/17 10:04:49] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:49] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:49] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:49] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:49] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:49] …/common/session.c(398): Accepted a local client
[04/19/17 10:04:49] …/common/session.c(398): Accepted a local client

seafile.log (I am only including a portion which I assume to be interesting, because it’s a lot):

[04/19/17 13:17:06] sync-mgr.c(702): Repo ‘spiele-palast-design’ sync state transition from ‘downloading’ to ‘synchronized’.
[04/19/17 13:17:07] sync-mgr.c(1516): Removing blocks for repo spiele-palast-design(a4ba31fb).
[04/19/17 13:17:08] sync-mgr.c(702): Repo ‘spiele-palast-design’ sync state transition from ‘synchronized’ to ‘committing’.
[04/19/17 13:17:08] repo-mgr.c(3730): All events are processed for repo a4ba31fb-44dc-4c2c-a61a-2b4f6b61ddcf.
[04/19/17 13:17:08] sync-mgr.c(702): Repo ‘spiele-palast-design’ sync state transition from ‘committing’ to ‘synchronized’.
[04/19/17 13:28:06] sync-mgr.c(702): Repo ‘spiele-palast-design-current’ sync state transition from ‘initializing’ to ‘downloading’.
[04/19/17 13:28:06] http-tx-mgr.c(4262): Download with HTTP sync protocol version 1.
[04/19/17 13:28:06] http-tx-mgr.c(1109): Transfer repo ‘7cded088’: (‘normal’, ‘init’) → (‘normal’, ‘check’)
[04/19/17 13:28:06] http-tx-mgr.c(1109): Transfer repo ‘7cded088’: (‘normal’, ‘check’) → (‘normal’, ‘commit’)
[04/19/17 13:28:06] http-tx-mgr.c(1109): Transfer repo ‘7cded088’: (‘normal’, ‘commit’) → (‘normal’, ‘fs’)
[04/19/17 13:28:07] http-tx-mgr.c(1109): Transfer repo ‘7cded088’: (‘normal’, ‘fs’) → (‘normal’, ‘data’)
[04/19/17 13:36:00] http-tx-mgr.c(1109): Transfer repo ‘7cded088’: (‘normal’, ‘data’) → (‘finished’, ‘finished’)
[04/19/17 13:36:00] sync-mgr.c(702): Repo ‘spiele-palast-design-current’ sync state transition from ‘downloading’ to ‘synchronized’.
[04/19/17 13:36:01] sync-mgr.c(1516): Removing blocks for repo spiele-palast-design-current(7cded088).
[04/19/17 13:36:02] sync-mgr.c(702): Repo ‘spiele-palast-design-current’ sync state transition from ‘synchronized’ to ‘committing’.
[04/19/17 13:36:02] repo-mgr.c(3730): All events are processed for repo 7cded088-f671-4f71-9858-b73aeed17feb.
[04/19/17 13:36:02] sync-mgr.c(702): Repo ‘spiele-palast-design-current’ sync state transition from ‘committing’ to ‘synchronized’.
[04/19/17 13:37:36] sync-mgr.c(702): Repo ‘spiele-palast-design-current’ sync state transition from ‘initializing’ to ‘downloading’.
[04/19/17 13:37:36] http-tx-mgr.c(4262): Download with HTTP sync protocol version 1.
[04/19/17 13:37:36] http-tx-mgr.c(1109): Transfer repo ‘7cded088’: (‘normal’, ‘init’) → (‘normal’, ‘check’)
[04/19/17 13:37:36] http-tx-mgr.c(1109): Transfer repo ‘7cded088’: (‘normal’, ‘check’) → (‘normal’, ‘commit’)
[04/19/17 13:37:36] http-tx-mgr.c(1109): Transfer repo ‘7cded088’: (‘normal’, ‘commit’) → (‘normal’, ‘fs’)
[04/19/17 13:37:36] http-tx-mgr.c(1109): Transfer repo ‘7cded088’: (‘normal’, ‘fs’) → (‘normal’, ‘data’)
[04/19/17 13:39:35] http-tx-mgr.c(4061): Bad response code for GET URL/seafhttp/repo/7cded088-f671-4f71-9858-b73aeed17feb/block/ff82bf11d2b61b2c1c831a28a9c8ed553470870d: 500.
[04/19/17 13:39:35] repo-mgr.c(5198): Transfer failed.
[04/19/17 13:39:35] http-tx-mgr.c(1109): Transfer repo ‘7cded088’: (‘normal’, ‘data’) → (‘error’, ‘finished’)
[04/19/17 13:39:35] sync-mgr.c(764): Repo ‘spiele-palast-design-current’ sync state transition from downloading to ‘error’: ‘Error occured in download.’.

Let me know if this is sufficient or if you need more. Also let me know if you need logs from the other clients.

Thanks for your support!

@daniel.pan

is a contrast to, not sure if that’s correct.

… one request failure

and some errors that it tried to allocate 0 bytes

[04/19/2017 12:39:36 PM] http-server.c(1374): Failed to allocate 0 bytes memeory.

This is problematic and likely the cause, or a good indication of the cause, of your corruption.

Is your server running out of memory? Check the ulimits that are configured for the user account the seafile server is running under and ensure it’s not hitting any resource limits. Check /var/log/messages for any OS-level indications of problems that are being logged around the same time these “failed to allocate” messages are being logged.

Thanks for your suggestions.

ulimits are installation default, ulimit -n shows 1024. Is there any guideline how to configure this or other values for Seafile? I followed the installation procedure in the wiki but I don’t think there was any mentioning of ulimit.

I also checked /var/log/syslog but there are no entries around the time. Any other important logs I should check? I am not an expert on Ubuntu so I am not sure what to look for. I opened all the logs that are in /var/log/ (except the subfolders) but nothing looks related.

About the 5.0.4 version: This is interesting. When I rightclick my Seafile Client Trace Icon and hit About, it definitely says 6.0.4. Maybe there is just a typo in the ccnet version?

The server disk is only at 19% capacity. The server has 16GB of RAM so it should not easily run out of memory, but I dont know how to check that. Is there a limit that I have to configure in the seafile config?

Thanks in advance.

This warning prints when the client tries to fetch a zero-sized block from the server, which won’t happen normally. I suspect it’s due to some metadata corruption on the server. Can you post the output of seaf-fsck on that library?

Hello,

I don’t have the original output of seaf-fsck from that day but we did have another corruption today. This is the output of seaf-fsck:

[04/21/17 10:51:00] fsck.c(586): Running fsck for repo 7cded088-f671-4f71-9858-b73aeed17feb.
[04/21/17 10:51:00] fsck.c(413): Checking file system integrity of repo spiele-palast-design-current(7cded088)…
[04/21/17 10:51:00] fsck.c(105): Block 0e6c96cb5605f14808bb2a92272e6c5b901ec559 is damaged, remove it.
[04/21/17 10:51:00] fsck.c(182): File /ui/ui_studie/legacy (SFConflict [USER-NAME] 2017-04-04-10-53-24) (SFConflict [USER-NAME] 2017-04-18-15-28-38) (SFConflict [USER-NAME] 2017-04-20-13-20-59).psd(a7e18024) is damaged, recreate an empty file.
[04/21/17 10:51:06] fsck.c(391): Update repo 7cded088 status to commit 7eeeae7c.
[04/21/17 10:51:06] fsck.c(650): Fsck finished for repo 7cded088.

seaf-fsck run done

The output always looks like that after we had a corruption, but it is always different files.

Are the files of any specific type like psd or is the type random as well?

I am not 100% sure but I do not remember any case where it was not a psd file. This is our marketing repository, so there are a lot of psd files.

Ok, so we cannot exclude a problem with psd files in specific.

Another thing: Do you use nginx / apache and have access logs enabled? If so, can you check which client uploaded the block? Using nginx default logging such requests look like

IP - - [21/Apr/2017:09:10:37 +0200] "PUT /seafhttp/repo/REPO-ID/block/BLOCK-ID HTTP/1.1" 200 0 "-" "Seafile/6.0.4 (Windows NT)"

So in your case you could grep for something like this:

zcat access-logs | grep \"PUT" | grep "block/0e6c96cb5605f14808bb2a92272e6c5b901ec559"

The interesting thing would be if corrupted files are all uploaded form the same client.

Neat, sounds like a good strategy. Access logs are enabled. Here is the entry in question:

[IP] - - [21/Apr/2017:09:52:17 +0100] “PUT /seafhttp/repo/7cded088-f671-4f71-9858-b73aeed17feb/block/0e6c96cb5605f14808bb2a92272e6c5b901ec559 HTTP/1.1” 200 0 “-” “Seafile/6.0.4 (Windows NT)”

Is there a way to identify the user through this entry other than IP? We are all sitting in the same office so IP will not be of much use.

PS: So it is already clear that the broken files are uploaded from a client and not corrupted by the server?

I’d say it is still unclear but wouldn’t exclude any option (except server hardware). As the server hardware seems to be ok (ecc, quality hardware in comparison to most desktop hardware, you also switched the hardware) I think we can exclude that option.

So a bug on the server (software) or client (either hardware or software) is the issue. As I’m pretty good informed on Seafile and would have read about corruption issues I currently think it is most likely that there is a problem with the client hardware. If that’s not the case it most likely is something relating to psd files but that’s not that likely.

Didn’t thought about that. If you have the audit log enabled you can download the log (file update) for that day and look at the update at the time of the log entry. This way you can find out which account made the update.

Ok great, thank you. I will keep an eye on that and gather data. Also I have advised my colleague to switch the HDD and see if that prevents further corruptions. I will post back once I have news on this. Thanks for your continous support on this issue.