Seafile Server Becomes Unresponsive After A Few Minutes

Our seafile server (seafile-server-6.2.5 with MySQL 5.6 on Debian 8.11) has started to become unresponsive and I have had to keep stopping and restarting the seafile and seahub processes to get it to respond again.

I start the processes with:

./seahub.sh start
./seafile.sh start

and not ./seahub.sh start-fastcgi

By unresponsive, the client will just show a swirl:

And the browser will just keep running its loading animation. I don’t get any bad gateway messages etc.

Logs

The logs within /home/seafile/logs don’t appear to show anything helpful.

There is nothing in seahub.log that relates to today when I am starting the server and it eventually stops responding.

seafile.log:

8/29/2018 10:11:36 AM] listen-mgr.c(120): listen on port 12001 for block tranfer
[08/29/2018 10:11:37 AM] size-sched.c(96): Repo size compute queue size is 0
[08/29/2018 10:16:37 AM] size-sched.c(96): Repo size compute queue size is 0
[08/29/2018 10:21:37 AM] size-sched.c(96): Repo size compute queue size is 0
[08/29/2018 10:26:37 AM] size-sched.c(96): Repo size compute queue size is 0
[08/29/2018 10:31:37 AM] size-sched.c(96): Repo size compute queue size is 0
[08/29/2018 10:34:10 AM] Disconnected from daemon
[08/29/18 10:34:45] http-server.c(161): fileserver: worker_threads = 10
[08/29/18 10:34:45] http-server.c(176): fileserver: fixed_block_size = 8388608
[08/29/18 10:34:45] http-server.c(191): fileserver: web_token_expire_time = 3600
[08/29/18 10:34:45] http-server.c(206): fileserver: max_indexing_threads = 1
[08/29/2018 10:34:45 AM] ../common/mq-mgr.c(54): [mq client] mq cilent is started
[08/29/2018 10:34:45 AM] listen-mgr.c(120): listen on port 12001 for block tranfer
[08/29/2018 10:34:46 AM] size-sched.c(96): Repo size compute queue size is 0
[08/29/2018 10:39:46 AM] size-sched.c(96): Repo size compute queue size is 0
[08/29/2018 10:44:46 AM] size-sched.c(96): Repo size compute queue size is 0

ccnet.log:

[08/29/18 10:34:44] ../common/session.c(132): using config file /mnt/data/admin/conf/ccnet.conf
[08/29/18 10:34:44] ../common/session.c(455): socket file exists, delete it anyway
[08/29/18 10:34:44] ../common/session.c(484): Listen on /mnt/data/admin/ccnet/ccnet.sock for local clients
[08/29/18 10:34:44] ../common/session.c(290): Update pubinfo file
[08/29/18 10:34:44] ../common/connect-mgr.c(515): Opened port 10001 to listen for incoming peer connections
[08/29/18 10:34:45] ../common/session.c(398): Accepted a local client
[08/29/18 10:34:45] ../common/session.c(398): Accepted a local client
[08/29/18 10:34:45] ../common/session.c(398): Accepted a local client
[08/29/18 10:34:45] ../common/session.c(398): Accepted a local client
[08/29/18 10:34:47] ../common/session.c(398): Accepted a local client
[08/29/18 10:34:47] ../common/peer.c(943): Local peer down
[08/29/18 10:34:48] ../common/session.c(398): Accepted a local client
[08/29/18 10:34:52] ../common/session.c(398): Accepted a local client
[08/29/18 10:34:52] ../common/session.c(398): Accepted a local client
[08/29/18 10:34:57] ../common/session.c(398): Accepted a local client
[08/29/18 10:35:41] ../common/session.c(398): Accepted a local client
[08/29/18 10:36:36] ../common/session.c(398): Accepted a local client

controller.log:

[08/29/18 10:11:36] seafile-controller.c(528): seafdav not enabled.
[08/29/18 10:34:10] seafile-controller.c(566): shutting down ccnet-server ...
[08/29/18 10:34:44] seafile-controller.c(154): starting ccnet-server ...
[08/29/18 10:34:44] seafile-controller.c(73): spawn_process: ccnet-server -F /mnt/data/admin/conf -c /mnt/data/admin/ccnet -f /mnt/data/admin/logs/ccnet.log -d -P /mnt/data/admin/pids/ccnet.pid
[08/29/18 10:34:44] seafile-controller.c(88): spawned ccnet-server, pid 25096
[08/29/18 10:34:45] seafile-controller.c(555): ccnet daemon connected.
[08/29/18 10:34:45] seafile-controller.c(186): starting seaf-server ...
[08/29/18 10:34:45] seafile-controller.c(73): spawn_process: seaf-server -F /mnt/data/admin/conf -c /mnt/data/admin/ccnet -d /home/ubuntu/seafile/seafile-data -l /mnt/data/admin/logs/seafile.log -P /mnt/data/admin/pids/seaf-server.pid
[08/29/18 10:34:45] seafile-controller.c(88): spawned seaf-server, pid 25117
[08/29/18 10:34:45] seafile-controller.c(528): seafdav not enabled.

Nginx

Since the issue started, I have put an nginx (1.12.2) reverse proxy in place moved myself aacross to using it by editing my client connection details to remove the port 8000. I have also moved myself and one other person across across so far (by removing the :8000 from the connection settings in the client).

The nginx error log shows:

2018/08/29 10:09:23 [notice] 3220#3220: signal process started
2018/08/29 10:11:16 [notice] 4890#4890: signal process started
2018/08/29 10:34:14 [error] 4892#4892: *259 connect() failed (111: Connection refused) while connecting to upstream, client: 59.179.26.73, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
2018/08/29 10:34:18 [error] 4894#4894: *192 connect() failed (111: Connection refused) while connecting to upstream, client: 178.62.56.51, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
2018/08/29 10:34:44 [error] 4894#4894: *288 connect() failed (111: Connection refused) while connecting to upstream, client: 59.179.26.73, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"

The nginx access log shows:

59.179.26.73 - - [29/Aug/2018:10:42:30 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:42:50 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
178.62.56.51 - - [29/Aug/2018:10:43:20 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
178.62.56.51 - - [29/Aug/2018:10:43:50 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:43:52 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:44:21 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:44:22 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:44:51 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:44:53 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:45:21 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:45:24 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:45:51 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
178.62.56.51 - - [29/Aug/2018:10:46:21 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:46:45 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:46:51 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:47:16 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:47:21 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:47:46 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:47:51 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:48:16 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:48:21 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:48:47 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:48:52 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:49:17 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:49:22 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:49:48 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:49:52 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:50:18 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:50:22 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:50:49 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:50:52 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:51:19 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:51:22 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:51:49 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:51:52 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:52:20 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:52:22 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.179.26.73 - - [29/Aug/2018:10:52:51 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:52:53 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"
59.177.203.155 - - [29/Aug/2018:10:53:22 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 1969 "-" "Seafile/6.2.4 (Windows NT)"
178.62.56.51 - - [29/Aug/2018:10:53:23 +0000] "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1" 200 411 "-" "Seafile/6.1.8 (Linux)"

MySQL error log

The mysql error log:

180829 09:50:58 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
180829 10:04:16 mysqld_safe Logging to '/var/log/mysql/error.log'.
180829 10:04:16 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2018-08-29 10:04:16 0 [Note] /usr/sbin/mysqld (mysqld 5.6.41) starting as process 655 ...
2018-08-29 10:04:16 655 [Note] Plugin 'FEDERATED' is disabled.
2018-08-29 10:04:16 655 [ERROR] Function 'innodb' already exists
2018-08-29 10:04:16 655 [Warning] Couldn't load plugin named 'innodb' with soname 'ha_innodb.so'.
2018-08-29 10:04:16 655 [ERROR] Function 'federated' already exists
2018-08-29 10:04:16 655 [Warning] Couldn't load plugin named 'federated' with soname 'ha_federated.so'.
2018-08-29 10:04:16 655 [ERROR] Function 'blackhole' already exists
2018-08-29 10:04:16 655 [Warning] Couldn't load plugin named 'blackhole' with soname 'ha_blackhole.so'.
2018-08-29 10:04:16 655 [ERROR] Function 'archive' already exists
2018-08-29 10:04:16 655 [Warning] Couldn't load plugin named 'archive' with soname 'ha_archive.so'.
2018-08-29 10:04:16 655 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-08-29 10:04:16 655 [Note] InnoDB: The InnoDB memory heap is disabled
2018-08-29 10:04:16 655 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-08-29 10:04:16 655 [Note] InnoDB: Memory barrier is not used
2018-08-29 10:04:16 655 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-08-29 10:04:16 655 [Note] InnoDB: Using Linux native AIO
2018-08-29 10:04:16 655 [Note] InnoDB: Using CPU crc32 instructions
2018-08-29 10:04:16 655 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-08-29 10:04:16 655 [Note] InnoDB: Completed initialization of buffer pool
2018-08-29 10:04:16 655 [Note] InnoDB: Highest supported file format is Barracuda.
2018-08-29 10:04:17 655 [Note] InnoDB: 128 rollback segment(s) are active.
2018-08-29 10:04:17 655 [Note] InnoDB: Waiting for purge to start
2018-08-29 10:04:17 655 [Note] InnoDB: 5.6.41 started; log sequence number 525129698
2018-08-29 10:04:17 655 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
2018-08-29 10:04:17 655 [Note]   - '127.0.0.1' resolves to '127.0.0.1';
2018-08-29 10:04:17 655 [Note] Server socket created on IP: '127.0.0.1'.
2018-08-29 10:04:17 655 [Note] Event Scheduler: Loaded 0 events
2018-08-29 10:04:17 655 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.41'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)

Seaf FSCK

I ran a full seaf-fsck yesterday overnight and that found/fixed some bad blocks in a few libraries.

MysQL repair

I also ran a mysql repair operation:
mysqlcheck -u root -p --all-databases --auto-repair

Clear Cache

I tried clearing the cache with:

sudo rm -rf /tmp/seahub_cache

Question

Are there any logs that I haven’t checked? Has anyone else found/fixed this issue?

Update - 29 Aug 13:29

I have since enabled memcached but still experiencing the issue.

Deployment Setup

We are using AWS with a t3.medium with 2 vCPUs, 4GB of RAM, 1.5 TB of magnetic hdd storage which seafile is set to use, but the MySQL database is on the / partition is only at 10% and is SSD based. The block storage is an LVG made up of 500GB LVM chunks as we have needed to add more storage over time. It’s currently quite full, but still has 70GB free (does seafile require x% free at all times?).

I already tried increasing the instance to the maximum possible t3, but that had no effect. Even on the t3 medium, the server appears to be hardly doing anything at all with no CPU usage and lots of spare RAM just getting used as a cache/buffer for the disk (yellow lines).

Nginx Logs

After switching more and more people across to connecting on port 80 through Nginx, I can now see these error logs under /var/log/nginx

seahub.error.log:

    2018/08/30 09:15:20 [error] 601#601: *5712 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 51.52.10.14, server: seafile.xxxx.com, request: "GET /api2/repos/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/repos/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *5714 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 51.52.10.14, server: seafile.xxxx.com, request: "GET /api2/events/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/events/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *5734 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 51.52.10.14, server: seafile.xxxx.com, request: "GET /api2/account/info/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/account/info/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *5812 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 178.62.56.51, server: seafile.xxxx.com, request: "GET /api2/events/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/events/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *5861 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 92.207.255.103, server: seafile.xxxx.com, request: "GET /api2/repos/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/repos/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *5914 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 130.105.147.229, server: seafile.xxxx.com, request: "GET /api/v2.1/notifications/?_=1535611664577 HTTP/1.1", upstream: "http://127.0.0.1:8000/api/v2.1/notifications/?_=1535611664577", host: "seafile.xxxx.com", referrer: "http://seafile.xxxx.com/"
    2018/08/30 09:15:20 [error] 601#601: *5921 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 59.179.26.73, server: seafile.xxxx.com, request: "GET /api/v2.1/notifications/?_=1535619217242 HTTP/1.1", upstream: "http://127.0.0.1:8000/api/v2.1/notifications/?_=1535619217242", host: "seafile.xxxx.com", referrer: "http://seafile.xxxx.com/"
    2018/08/30 09:15:20 [error] 601#601: *5983 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 178.62.56.51, server: seafile.xxxx.com, request: "GET /api2/account/info/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/account/info/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *6013 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 195.147.152.100, server: seafile.xxxx.com, request: "GET /api2/account/info/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/account/info/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *6044 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 220.240.176.49, server: seafile.xxxx.com, request: "GET /api2/account/info/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/account/info/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *6096 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 92.207.255.103, server: seafile.xxxx.com, request: "GET /api2/account/info/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/account/info/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *6123 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 59.179.26.73, server: seafile.xxxx.com, request: "GET /api2/account/info/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/account/info/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *6177 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 104.237.86.107, server: seafile.xxxx.com, request: "GET /api2/repos/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/repos/", host: "seafile.xxxx.com"
    2018/08/30 09:15:20 [error] 601#601: *6188 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 178.62.56.51, server: seafile.xxxx.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "seafile.xxxx.com"
    2018/08/30 09:15:21 [error] 601#601: *6188 connect() failed (111: Connection refused) while connecting to upstream, client: 178.62.56.51, server: seafile.xxxx.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "seafile.xxxx.com"
    2018/08/30 09:15:23 [error] 602#602: *6029 connect() failed (111: Connection refused) while connecting to upstream, client: 59.179.26.73, server: seafile.xxxx.com, request: "GET /api2/ping/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/ping/", host: "seafile.xxxx.com"
    2018/08/30 09:15:23 [error] 602#602: *6029 connect() failed (111: Connection refused) while connecting to upstream, client: 59.179.26.73, server: seafile.xxxx.com, request: "GET /api2/account/info/ HTTP/1.1", upstream: "http://127.0.0.1:8000/api2/account/info/", host: "seafile.xxxx.com"

error.log:

    2018/08/30 08:53:25 [error] 602#602: *3726 connect() failed (111: Connection refused) while connecting to upstream, client: 51.52.10.14, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 08:53:26 [error] 601#601: *2588 connect() failed (111: Connection refused) while connecting to upstream, client: 92.207.255.103, server: seafile.xxxx.com, request: "GET /seafhttp/repo/fe43ee8e-a4e5-4861-9976-84be110f0226/permission-check/?op=download&client_id=e3de94cb1b8e1685277a189816a156899134019c&client_name=dell-laptop HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/fe43ee8e-a4e5-4861-9976-84be110f0226/permission-check/?op=download&client_id=e3de94cb1b8e1685277a189816a156899134019c&client_name=dell-laptop", host: "seafile.xxxx.com"
    2018/08/30 08:53:28 [error] 601#601: *3927 connect() failed (111: Connection refused) while connecting to upstream, client: 130.105.147.229, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 08:53:32 [error] 601#601: *2525 connect() failed (111: Connection refused) while connecting to upstream, client: 92.207.255.103, server: seafile.xxxx.com, request: "GET /seafhttp/repo/8598d9bf-3927-48c8-b024-2ea9200c9782/commit/HEAD HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/8598d9bf-3927-48c8-b024-2ea9200c9782/commit/HEAD", host: "seafile.xxxx.com"
    2018/08/30 08:53:34 [error] 601#601: *3931 connect() failed (111: Connection refused) while connecting to upstream, client: 31.53.208.67, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 08:53:35 [error] 601#601: *2249 connect() failed (111: Connection refused) while connecting to upstream, client: 178.62.56.51, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 09:14:53 [error] 602#602: *6169 connect() failed (111: Connection refused) while connecting to upstream, client: 220.240.176.49, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 09:14:54 [error] 601#601: *5696 connect() failed (111: Connection refused) while connecting to upstream, client: 51.52.10.14, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 09:14:59 [error] 601#601: *5952 connect() failed (111: Connection refused) while connecting to upstream, client: 130.105.147.229, server: seafile.xxxx.com, request: "GET /seafhttp/repo/312a7786-3c57-4cd7-a7a3-10af33c5be65/permission-check/?op=upload&client_id=cc840bbb9d99ce3a0ee708095398ffc198d1390e&client_name=DESKTOP-4D9ETFT HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/312a7786-3c57-4cd7-a7a3-10af33c5be65/permission-check/?op=upload&client_id=cc840bbb9d99ce3a0ee708095398ffc198d1390e&client_name=DESKTOP-4D9ETFT", host: "seafile.xxxx.com"
    2018/08/30 09:15:02 [error] 601#601: *100 connect() failed (111: Connection refused) while connecting to upstream, client: 104.237.86.107, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 09:15:07 [error] 602#602: *4008 connect() failed (111: Connection refused) while connecting to upstream, client: 92.207.255.103, server: seafile.xxxx.com, request: "GET /seafhttp/repo/312a7786-3c57-4cd7-a7a3-10af33c5be65/commit/HEAD HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/312a7786-3c57-4cd7-a7a3-10af33c5be65/commit/HEAD", host: "seafile.xxxx.com"
    2018/08/30 09:15:08 [error] 601#601: *5869 connect() failed (111: Connection refused) while connecting to upstream, client: 92.207.255.103, server: seafile.xxxx.com, request: "GET /seafhttp/repo/fe43ee8e-a4e5-4861-9976-84be110f0226/permission-check/?op=download&client_id=e3de94cb1b8e1685277a189816a156899134019c&client_name=dell-laptop HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/fe43ee8e-a4e5-4861-9976-84be110f0226/permission-check/?op=download&client_id=e3de94cb1b8e1685277a189816a156899134019c&client_name=dell-laptop", host: "seafile.xxxx.com"
    2018/08/30 09:15:10 [error] 601#601: *6184 connect() failed (111: Connection refused) while connecting to upstream, client: 59.179.26.73, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 09:15:10 [error] 601#601: *4011 connect() failed (111: Connection refused) while connecting to upstream, client: 178.62.56.51, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 09:15:14 [error] 602#602: *5741 connect() failed (111: Connection refused) while connecting to upstream, client: 92.207.255.103, server: seafile.xxxx.com, request: "GET /seafhttp/repo/8598d9bf-3927-48c8-b024-2ea9200c9782/commit/HEAD HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/8598d9bf-3927-48c8-b024-2ea9200c9782/commit/HEAD", host: "seafile.xxxx.com"
    2018/08/30 09:15:15 [error] 601#601: *6055 connect() failed (111: Connection refused) while connecting to upstream, client: 92.207.255.103, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 09:15:16 [error] 602#602: *5765 connect() failed (111: Connection refused) while connecting to upstream, client: 51.52.10.14, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 09:15:16 [error] 602#602: *5114 connect() failed (111: Connection refused) while connecting to upstream, client: 51.52.10.14, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"
    2018/08/30 09:15:18 [error] 602#602: *4933 connect() failed (111: Connection refused) while connecting to upstream, client: 81.158.106.174, server: seafile.xxxx.com, request: "GET /seafhttp/repo/36c23fc9-a91c-48fc-9b42-81b523c7a787/commit/HEAD HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/36c23fc9-a91c-48fc-9b42-81b523c7a787/commit/HEAD", host: "seafile.xxxx.com"
    2018/08/30 09:15:19 [error] 602#602: *4937 connect() failed (111: Connection refused) while connecting to upstream, client: 81.158.106.174, server: seafile.xxxx.com, request: "GET /seafhttp/repo/636e71f4-74c6-468d-ab10-6d1a7ad9bcb3/commit/HEAD HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/636e71f4-74c6-468d-ab10-6d1a7ad9bcb3/commit/HEAD", host: "seafile.xxxx.com"
    2018/08/30 09:15:19 [error] 602#602: *4935 connect() failed (111: Connection refused) while connecting to upstream, client: 81.158.106.174, server: seafile.xxxx.com, request: "GET /seafhttp/repo/c0cf3dde-6ac0-4112-b419-283f6d72acfa/commit/HEAD HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/c0cf3dde-6ac0-4112-b419-283f6d72acfa/commit/HEAD", host: "seafile.xxxx.com"
    2018/08/30 09:15:22 [error] 602#602: *5991 connect() failed (111: Connection refused) while connecting to upstream, client: 130.105.147.229, server: seafile.xxxx.com, request: "POST /seafhttp/repo/head-commits-multi/ HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/head-commits-multi/", host: "seafile.xxxx.com"

Nginx Connection Stats

I followed this nixcraft post on getting nginx to output statistics.

After restarting the seafile processes, the output would be something like below:

Active connections: 66 
server accepts handled requests
 71 71 220 
Reading: 0 Writing: 1 Waiting: 65 

However, eventually it will move to something like:

Active connections: 48 
server accepts handled requests
 324 324 1362 
Reading: 0 Writing: 16 Waiting: 32 

Whenever “Writing” gets above something like 4 (and usually when it rises, it rises rapidly to 16 or 20), then this is when the seahub web interface becomes unresponsive. Perhaps there is a way that I can gracefully handle this by tweaking some configuration settings either in seahub or in nginx?

Pretty certain I have achieved stability now after having blocked off port 8000 through the firewall. This means that people can only go through the nginx frontend proxy now. I think that some people were still hitting the website throught the browser with :8000, perhaps bookmarked etc.