Seafile Server is getting slow

Seafile Server
1

In the last weeks, my local seafile server is very unresponsive and has syncing problems. It also causes to crash my client apps (WindowsX64) quite often.

server seahub.log

2019-07-05 16:40:05,011 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-05 16:41:15,280 [ERROR] django.request:135 handle_uncaught_exception Internal Server Error: /server-status
Traceback (most recent call last):
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/core/handlers/exception.py”, line 41, in inner
response = get_response(request)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/core/handlers/base.py”, line 244, in _legacy_get_response
response = middleware_method(request)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/seahub/base/middleware.py”, line 69, in process_request
self.get_from_db()
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/seahub/base/middleware.py”, line 53, in get_from_db
refresh_cache()
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/seahub/notifications/utils.py”, line 22, in refresh_cache
NOTIFICATION_CACHE_TIMEOUT)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/core/cache/backends/filebased.py”, line 57, in set
f.write(zlib.compress(pickle.dumps(value, pickle.HIGHEST_PROTOCOL)))
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/query.py”, line 202, in getstate
self._fetch_all()
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/query.py”, line 53, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
OperationalError: database is locked
2019-07-05 16:41:21,578 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-05 16:45:04,916 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-05 16:46:15,307 [ERROR] django.request:135 handle_uncaught_exception Internal Server Error: /server-status
Traceback (most recent call last):
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/core/handlers/exception.py”, line 41, in inner
response = get_response(request)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/core/handlers/base.py”, line 244, in _legacy_get_response
response = middleware_method(request)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/seahub/base/middleware.py”, line 69, in process_request
self.get_from_db()
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/seahub/base/middleware.py”, line 53, in get_from_db
refresh_cache()
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/seahub/notifications/utils.py”, line 22, in refresh_cache
NOTIFICATION_CACHE_TIMEOUT)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/core/cache/backends/filebased.py”, line 57, in set
f.write(zlib.compress(pickle.dumps(value, pickle.HIGHEST_PROTOCOL)))
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/query.py”, line 202, in getstate
self._fetch_all()
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/query.py”, line 53, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
OperationalError: database is locked
2019-07-05 16:46:25,486 [ERROR] django.request:135 handle_uncaught_exception Internal Server Error: /server-status
Traceback (most recent call last):
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/core/handlers/exception.py”, line 41, in inner
response = get_response(request)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/core/handlers/base.py”, line 244, in _legacy_get_response
response = middleware_method(request)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/seahub/base/middleware.py”, line 69, in process_request
self.get_from_db()
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/seahub/base/middleware.py”, line 53, in get_from_db
refresh_cache()
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/seahub/notifications/utils.py”, line 22, in refresh_cache
NOTIFICATION_CACHE_TIMEOUT)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/core/cache/backends/filebased.py”, line 57, in set
f.write(zlib.compress(pickle.dumps(value, pickle.HIGHEST_PROTOCOL)))
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/query.py”, line 202, in getstate
self._fetch_all()
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/query.py”, line 53, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File “/VG/SEAFILE/seafile-server-7.0.2/seahub/thirdpart/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
OperationalError: database is locked
[…]
2019-07-08 14:16:17,410 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:20:04,054 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:20:08,074 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:20:14,083 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:29:49,046 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:29:49,147 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:29:49,211 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:30:04,154 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:30:08,468 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:30:14,004 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:35:03,506 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:36:27,498 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:36:32,559 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:40:04,403 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:40:28,942 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:40:34,190 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:45:04,578 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:45:08,782 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:45:13,973 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:50:04,046 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:51:09,612 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:51:14,828 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:55:03,997 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:55:38,438 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 14:55:43,504 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:00:04,010 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:00:08,187 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:00:12,915 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:05:03,668 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:06:09,211 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:06:14,477 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:10:04,063 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:11:46,542 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:11:46,636 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:15:20,937 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:15:25,457 [WARNING] django.request:152 get_response Not Found: /server-status
2019-07-08 15:15:31,043 [WARNING] django.request:152 get_response Not Found: /server-status

client seafile.log 
[07/08/19 17:20:38] http-tx-mgr.c(954): libcurl failed to PUT http://192.168.0.100/seafhttp/repo/bcc8b60a-72f2-4172-9f80-184c734806c2/commit/HEAD/?head=bd7f683cff91eea8b5f82b0aabb411a848043fff: Timeout was reached.
[07/08/19 17:20:38] http-tx-mgr.c(3813): Failed to update branch of repo bcc8b60a.
[07/08/19 17:20:38] http-tx-mgr.c(1181): Transfer repo 'bcc8b60a': ('normal', 'update-branch') --> ('error', 'finished')
[07/08/19 17:20:38] sync-mgr.c(621): Repo 'FUTUR III - MAN' sync state transition from uploading to 'error': 'Error occurred in upload.'.
[07/08/19 17:20:56] http-tx-mgr.c(1070): libcurl failed to POST http://192.168.0.100/seafhttp/repo/head-commits-multi/: Timeout was reached.
[07/08/19 17:21:54] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/018fc26e-4d37-47ae-b5bf-05cfe7ab2bd0/commit/HEAD: Timeout was reached.
[07/08/19 17:21:54] sync-mgr.c(621): Repo 'Alkoholprävention' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:21:54] sync-mgr.c(559): Repo 'FUTUR III - MAN' sync state transition from 'synchronized' to 'uploading'.
[07/08/19 17:21:54] http-tx-mgr.c(1181): Transfer repo 'bcc8b60a': ('normal', 'init') --> ('normal', 'check')
[07/08/19 17:21:55] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/5eceea92-f88c-4723-8bbc-d165b48fcf81/commit/HEAD: Timeout was reached.
[07/08/19 17:21:55] sync-mgr.c(621): Repo 'Deine Tierwelt' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:21:56] [07/08/19 17:21:56] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/20db3577-74d3-4bf9-99c6-8f4a1290d5c6/commit/HEAD: Timeout was reached.
http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/f9ae8ec4-8a6f-428c-a1d1-86653e8588f9/commit/HEAD: Timeout was reached.
[07/08/19 17:21:56] sync-mgr.c(621): Repo 'Kleefeld-Buchholz Genossenschaft' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:21:56] sync-mgr.c(621): Repo 'AEJ' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:26:54] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/bcc8b60a-72f2-4172-9f80-184c734806c2/permission-check/?op=upload&client_id=a07329a3c0ec21567e34448b5ff2b448d3999298&client_name=ELEMENTK-2019: Timeout was reached.
[07/08/19 17:26:54] http-tx-mgr.c(3708): Upload permission denied for repo bcc8b60a on server http://192.168.0.100.
[07/08/19 17:26:54] http-tx-mgr.c(1181): Transfer repo 'bcc8b60a': ('normal', 'check') --> ('error', 'finished')
[07/08/19 17:26:54] sync-mgr.c(621): Repo 'FUTUR III - MAN' sync state transition from uploading to 'error': 'Error occurred in upload.'.
[07/08/19 17:26:55] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/82b25903-f143-4f5e-ad4b-875ac07159e9/commit/HEAD: Timeout was reached.
[07/08/19 17:26:55] sync-mgr.c(621): Repo '2019-07 Interview Sven' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:26:56] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/50ed43ba-0d56-4389-93e8-30428de024b7/commit/HEAD: Timeout was reached.
[07/08/19 17:26:56] sync-mgr.c(621): Repo 'LEITHART' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:26:56] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/6e4e472d-527c-4359-b03e-c25129683298/commit/HEAD: Timeout was reached.
[07/08/19 17:26:56] sync-mgr.c(621): Repo 'Stiftung Niedersachsen' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:28:07] http-tx-mgr.c(1181): Transfer repo 'c16eba26': ('normal', 'data') --> ('normal', 'update-branch')
[07/08/19 17:28:07] http-tx-mgr.c(1181): Transfer repo 'c16eba26': ('normal', 'update-branch') --> ('finished', 'finished')
[07/08/19 17:28:07] sync-mgr.c(559): Repo 'Projekte Online' sync state transition from 'uploading' to 'initializing'.
[07/08/19 17:28:07] sync-mgr.c(864): Removing blocks for repo Projekte Online(c16eba26).
[07/08/19 17:28:08] sync-mgr.c(559): Repo 'Projekte Online' sync state transition from 'synchronized' to 'committing'.
[07/08/19 17:28:08] repo-mgr.c(2131): Failed to stat Futur III/Stiftung Niedersachsen/Output/06 LINK Tabea Golgath.mp4._00_: No such file or directory.
[07/08/19 17:28:08] repo-mgr.c(2131): Failed to stat Futur III/Stiftung Niedersachsen/Output/06 LINK Tabea Golgath.mp4._00_: No such file or directory.
[07/08/19 17:28:08] repo-mgr.c(1811): Failed to stat E:/Seafile/Projekte Online/Futur III/Stiftung Niedersachsen/Output/06 LINK Tabea Golgath.mp4._01_.
[07/08/19 17:28:15] repo-mgr.c(1811): Failed to stat E:/Seafile/Projekte Online/Futur III/Stiftung Niedersachsen/Output/06 LINK Tabea Golgath.mp4._00_.
[07/08/19 17:28:15] repo-mgr.c(3817): All events are processed for repo c16eba26-cdb0-45b8-b3c7-e69ddc15f789.
[07/08/19 17:28:15] repo-mgr.c(4029): No change to the fs tree of repo c16eba26-cdb0-45b8-b3c7-e69ddc15f789
[07/08/19 17:28:15] repo-mgr.c(3917): checking Futur III/Stiftung Niedersachsen/Output/06 LINK Tabea Golgath.mp4 in changeset.
[07/08/19 17:28:15] sync-mgr.c(559): Repo 'Projekte Online' sync state transition from 'committing' to 'synchronized'.
[07/08/19 17:28:15] sync-mgr.c(559): Repo 'Projekte Online' sync state transition from 'synchronized' to 'committing'.
[07/08/19 17:28:15] repo-mgr.c(3673): All events are processed for repo c16eba26-cdb0-45b8-b3c7-e69ddc15f789.
[07/08/19 17:28:15] sync-mgr.c(559): Repo 'Projekte Online' sync state transition from 'committing' to 'synchronized'.
[07/08/19 17:29:23] http-tx-mgr.c(1070): libcurl failed to POST http://192.168.0.100/seafhttp/repo/head-commits-multi/: Timeout was reached.
[07/08/19 17:30:32] sync-mgr.c(559): Repo 'FUTUR III - MAN' sync state transition from 'synchronized' to 'uploading'.
[07/08/19 17:30:32] http-tx-mgr.c(1181): Transfer repo 'bcc8b60a': ('normal', 'init') --> ('normal', 'check')
[07/08/19 17:35:32] [07/08/19 17:35:32] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/bcc8b60a-72f2-4172-9f80-184c734806c2/permission-check/?op=upload&client_id=a07329a3c0ec21567e34448b5ff2b448d3999298&client_name=ELEMENTK-2019: Timeout was reached.
http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/50ed43ba-0d56-4389-93e8-30428de024b7/commit/HEAD: Timeout was reached.
[07/08/19 17:35:32] http-tx-mgr.c(3708): Upload permission denied for repo bcc8b60a on server http://192.168.0.100.
[07/08/19 17:35:32] sync-mgr.c(621): Repo 'LEITHART' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:35:32] http-tx-mgr.c(1181): Transfer repo 'bcc8b60a': ('normal', 'check') --> ('error', 'finished')
[07/08/19 17:35:32] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/6e4e472d-527c-4359-b03e-c25129683298/commit/HEAD: Timeout was reached.
[07/08/19 17:35:32] sync-mgr.c(621): Repo 'FUTUR III - MAN' sync state transition from uploading to 'error': 'Error occurred in upload.'.
[07/08/19 17:35:32] sync-mgr.c(621): Repo 'Stiftung Niedersachsen' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:35:32] sync-mgr.c(559): Repo 'FUTUR III - MAN' sync state transition from 'synchronized' to 'uploading'.
[07/08/19 17:35:32] http-tx-mgr.c(1181): Transfer repo 'bcc8b60a': ('normal', 'init') --> ('normal', 'check')
[07/08/19 17:36:07] http-tx-mgr.c(1070): libcurl failed to POST http://192.168.0.100/seafhttp/repo/head-commits-multi/: Timeout was reached.
[07/08/19 17:37:05] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/20db3577-74d3-4bf9-99c6-8f4a1290d5c6/commit/HEAD: Timeout was reached.
[07/08/19 17:37:05] sync-mgr.c(621): Repo 'Kleefeld-Buchholz Genossenschaft' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:37:05] http-tx-mgr.c(808): libcurl failed to GET http://192.168.0.100/seafhttp/repo/82b25903-f143-4f5e-ad4b-875ac07159e9/commit/HEAD: Timeout was reached.
[07/08/19 17:37:05] sync-mgr.c(621): Repo '2019-07 Interview Sven' sync state transition from initializing to 'error': 'Failed to get sync info from server.'.
[07/08/19 17:40:30] http-tx-mgr.c(1181): Transfer repo 'bcc8b60a': ('normal', 'check') --> ('normal', 'commit')
[07/08/19 17:40:35] sync-mgr.c(864): Removing blocks for repo Projekte Online(c16eba26).
[07/08/19 17:42:16] http-tx-mgr.c(1181): Transfer repo 'bcc8b60a': ('normal', 'commit') --> ('normal', 'fs')
[07/08/19 17:42:16] http-tx-mgr.c(1181): Transfer repo 'bcc8b60a': ('normal', 'fs') --> ('normal', 'data')
[07/08/19 17:42:16] http-tx-mgr.c(1181): Transfer repo 'bcc8b60a': ('normal', 'data') --> ('normal', 'update-branch')

My system config looks like:

  • Debian 8
  • Seafile directory on a LVM with btrfs filesystem (27 TB)
  • Seafile server version 7.0.2
  • PostgreSQL database (seahub.db is about 25 MB big) <== could this be the performance bottleneck?
  • Seafile client version 7.0.0

And finally the relevant lines from my config:

gunicorn.conf

daemon = True
workers = 1
timeout = 1200

seafile.conf

worker_threads = 1
max_indexing_threads = 1

seafile.conf

worker_threads = 1
max_indexing_threads = 1

Thanks in advance!

2

25MB isn’t very big for a database. That will all fit in RAM easily, so I don’t think it would make a big difference to speed. There are a few things to check that might help to figure out what part is being slow. Is the web interface also slow? Are uploads and downloads from the web interface slow too? Are plain reads and writes to that same disk slow? Are there any errors in the logs (look in both the seafile logs and the syslog)?

I would also look for clues in atop while your server is relatively idle and while it is syncing some files. Look for any process using lot of CPU consistently. On the memory line if free + cache + buff isn’t much then you might need more memory.

On the line for your disk, look for the “busy”. Steady 100% (or close to it) means that this is going as fast as this disk can go. Sometimes that’s only a few MB if you are reading/writing heavily fragmented file system which can happen after btrfs has been used for a long time (a natural consequence of being a copy-on-write filesystem).

Anyway hopefully something there helps you narrow down what’s going on.

1 Like
3

Hi, thanks for your reply. The disk activity was indeed quite high and now I temporarily stopped the seafile server.
I now started a defragmentation of the btrfs and after its been finished I will do a balance, too. Maybe that will help.

Another option I found was the nodatacow option on the mount. Lets see, if it will make a difference.

4

I also think that it is very likely that io is an issue. How many disks do you use? Expect the disk operations to be mostly small random read operations. It could help to add an SSD as cache - but I don’t know what the best solution is for dowing that with btrfs, but think there are also some generic solutions available on linux.

The 25 MiB for the database isn’t an issue at all - it is actually very small.