Linux Client (seaf-cli) failing to download large libraries with 'unknown error'

On Ubuntu 18.04 I’m unable to download larger libraries successfully although smaller libraries sync correctly.
The failing libraries log the same type of errors (redacted sample below):

[04/24/20 10:43:19] http-tx-mgr.c(4145): Bad response code for POST https://seafile.redacted.co.uk/seafhttp/repo/60b9212b-9be3-4525-8d3d-3d17f917adb8/pack-fs/: 0.
[04/24/20 10:43:19] http-tx-mgr.c(4572): Failed to get fs objects for repo 60b9212b on server https://seafile.redacted.co.uk.
[04/24/20 10:43:19] http-tx-mgr.c(1157): Transfer repo '60b9212b': ('normal', 'fs') --> ('error', 'finished')
[04/24/20 10:43:19] clone-mgr.c(697): Transition clone state for 60b9212b from [fetch] to [error]: Unknown error.

The account has read-only access to all libraries.

How can I best investigate / resolve this issue?
I’ve posted my nginx configuration here in another question which sadly has had no feedback.

BUMP

It’s definitely a timeout based on client logs.

Shame that no-one’s able to help.

Did you look at ram and swap usage of your servers? when I copy a huge repo I usually get a very big load on ram and swap.

Also did you run a file system check? I see that in your previous post you did garbage collect and so. I had a file system issue and some of my libraries were erroring on some clients due to a server file system issue.

I looked at your logs, this is the api that time outs:

clientipaddress [28/Apr/2020:14:21:59 +0100] "POST /seafhttp/repo/d596f397-2163-4dbb-b1e1-ce1130b68ea2/pack-fs/ HTTP/1.1" 408 0 "-" "Seafile/7.0.4 (Linux)"

pack file system? if it is big you might literally be running out of resources

I found the code that handles that API call,


it looks that it gathers some info about the repository, and it logs warnings accordingly if something goes wrong. Can you also post your seafile and seahub logs as well? Not just the nginx (which only has info on the http calls)

This will be an interesting find.

Thanks for responding.

I’m now getting a proper error in the client-side logs after fiddling with the nginx timeouts.

Client logs:

[04/30/20 08:47:00] http-tx-mgr.c(1157): Transfer repo '54d15b3e': ('normal', 'fs') --> ('error', 'finished')
[04/30/20 08:47:00] sync-mgr.c(621): Repo 'RemoteBackupScripts' sync state transition from downloading to 'error': 'Data transfer timed out. Please check network or firewall'.
[04/30/20 08:47:00] Couldn't prepare query, error:1->'no such table: Certs'
        SELECT cert FROM Certs;
[04/30/20 08:47:00] sync-mgr.c(582): Repo 'RemoteBackupScripts' sync state transition from 'initializing' to 'downloading'.
[04/30/20 08:47:00] http-tx-mgr.c(1157): Transfer repo '54d15b3e': ('normal', 'init') --> ('normal', 'check')
[04/30/20 08:47:00] http-tx-mgr.c(1157): Transfer repo '54d15b3e': ('normal', 'check') --> ('normal', 'commit')
[04/30/20 08:47:00] http-tx-mgr.c(1157): Transfer repo '54d15b3e': ('normal', 'commit') --> ('normal', 'fs')
[04/30/20 08:47:10] Couldn't prepare query, error:1->'no such table: Certs'
        SELECT cert FROM Certs;
[04/30/20 08:50:45] http-tx-mgr.c(1045): libcurl failed to POST https://seafile.mydomain.com/seafhttp/repo/7f6a1a7e-25ec-4cf4-a896-5d26c5a11a02/pack-fs/: Timeout was reached.
[04/30/20 08:50:45] http-tx-mgr.c(4572): Failed to get fs objects for repo 7f6a1a7e on server https://seafile.mydomain.com.
[04/30/20 08:50:45] http-tx-mgr.c(1157): Transfer repo '7f6a1a7e': ('normal', 'fs') --> ('error', 'finished')
[04/30/20 08:50:45] sync-mgr.c(621): Repo 'Company' sync state transition from downloading to 'error': 'Data transfer timed out. Please check network or firewall'.
[04/30/20 08:50:45] sync-mgr.c(582): Repo 'Company' sync state transition from 'initializing' to 'downloading'.
[04/30/20 08:50:45] http-tx-mgr.c(1157): Transfer repo '7f6a1a7e': ('normal', 'init') --> ('normal', 'check')
[04/30/20 08:50:45] http-tx-mgr.c(1157): Transfer repo '7f6a1a7e': ('normal', 'check') --> ('normal', 'commit')
[04/30/20 08:50:45] http-tx-mgr.c(1157): Transfer repo '7f6a1a7e': ('normal', 'commit') --> ('normal', 'fs')
[04/30/20 08:51:10] Couldn't prepare query, error:1->'no such table: Certs'
        SELECT cert FROM Certs;
[04/30/20 08:51:33] http-tx-mgr.c(1045): libcurl failed to POST https://seafile.mydomain.com/seafhttp/repo/f0707b23-81b7-49b5-9da9-588efa62705a/pack-fs/: Timeout was reached.
[04/30/20 08:51:33] http-tx-mgr.c(4572): Failed to get fs objects for repo f0707b23 on server https://seafile.mydomain.com.
[04/30/20 08:51:34] http-tx-mgr.c(1157): Transfer repo 'f0707b23': ('normal', 'fs') --> ('error', 'finished')
[04/30/20 08:51:34] sync-mgr.c(621): Repo 'SubversionRepositories' sync state transition from downloading to 'error': 'Data transfer timed out. Please check network or firewall'.
[04/30/20 08:51:34] sync-mgr.c(582): Repo 'SubversionRepositories' sync state transition from 'initializing' to 'downloading'.
[04/30/20 08:51:34] http-tx-mgr.c(1157): Transfer repo 'f0707b23': ('normal', 'init') --> ('normal', 'check')
[04/30/20 08:51:34] http-tx-mgr.c(1157): Transfer repo 'f0707b23': ('normal', 'check') --> ('normal', 'commit')

The ‘Company’ repo is massive (currently 37406 / 124.0GB) however ‘RemoteBackupScripts’ is only small (43 / 124.8KB).

Server-side logs

seafile.log for the last 24 hours:

[04/29/20 08:12:52] http-server.c(220): fileserver: worker_threads = 10
[04/29/20 08:12:52] http-server.c(233): fileserver: backlog = 32
[04/29/20 08:12:52] http-server.c(248): fileserver: fixed_block_size = 8388608
[04/29/20 08:12:52] http-server.c(263): fileserver: web_token_expire_time = 3600
[04/29/20 08:12:52] http-server.c(278): fileserver: max_indexing_threads = 1
[04/29/20 08:12:52] http-server.c(293): fileserver: max_index_processing_threads= 3
[04/29/20 08:12:52] http-server.c(315): fileserver: cluster_shared_temp_file_mode = 600
[04/29/20 08:12:52] http-server.c(393): fileserver: enable_async_indexing = 0
[04/29/20 08:12:52] http-server.c(405): fileserver: async_indexing_threshold = 700
[04/29/20 08:12:52] http-server.c(418): fileserver: fs_id_list_request_timeout = -1
[04/29/20 08:12:52] ../common/mq-mgr.c(61): [mq client] mq cilent is started
[04/29/20 08:12:52] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 09:12:52] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 10:12:52] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 11:12:53] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 12:12:53] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 13:12:53] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 14:12:53] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 14:48:07] repo-mgr.c(3478): Commit a1a745eb8800a34a81876b2f5229091dcd971388 not found in repo 78daf96e-1383-4142-b6d6-4690b5eba31e
[04/29/20 14:48:14] repo-mgr.c(3478): Commit a1a745eb8800a34a81876b2f5229091dcd971388 not found in repo 78daf96e-1383-4142-b6d6-4690b5eba31e
[04/29/20 15:12:53] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 16:12:54] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 17:12:54] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 18:12:54] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 19:12:54] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 20:12:54] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 21:12:54] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 22:12:54] filelock-mgr.c(973): Cleaning expired file locks.
[04/29/20 23:12:54] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 00:12:54] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 01:12:54] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 02:12:55] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 03:12:55] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 04:12:55] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 05:12:55] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 06:12:55] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 07:12:55] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 08:12:55] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 09:12:55] filelock-mgr.c(973): Cleaning expired file locks.
[04/30/20 09:40:07] repo-mgr.c(3478): Commit a1a745eb8800a34a81876b2f5229091dcd971388 not found in repo 78daf96e-1383-4142-b6d6-4690b5eba31e

seahub.log for the last 24 hours

2020-04-29 08:12:56,169 [INFO] seafevents.db:74 create_engine_from_conf [seafevents] database: mysql, name: seahub_db
2020-04-29 08:12:56,192 [INFO] seafevents.db:74 create_engine_from_conf [seafevents] database: mysql, name: seafile_db
2020-04-29 08:12:56,383 [INFO] seafevents.app.config:127 load_file_history_config The file with the following suffix will be recorded into the file history: md,txt,doc,docx,xls,xlsx,ppt,pptx
2020-04-29 08:12:56,389 [INFO] seafevents.db:74 create_engine_from_conf [seafevents] database: mysql, name: seahub_db
2020-04-29 08:12:56,392 [INFO] seafevents:118 is_audit_enabled audit is enabled
2020-04-29 08:13:48,591 [INFO] seafes:162 load_seafevents_conf [seafes] use highlighter fvh
2020-04-29 08:14:00,547 [INFO] seafes:162 load_seafevents_conf [seafes] use highlighter fvh
2020-04-29 08:16:49,318 [INFO] seafes:162 load_seafevents_conf [seafes] use highlighter fvh
2020-04-29 08:17:49,821 [INFO] seafes:162 load_seafevents_conf [seafes] use highlighter fvh
2020-04-29 08:18:50,337 [INFO] seafes:162 load_seafevents_conf [seafes] use highlighter fvh
2020-04-29 10:42:50,023 [WARNING] django.request:152 get_response Not Found: /favicon.ico
2020-04-29 14:49:23,547 [WARNING] seahub.utils.licenseparse:38 parse_license [Errno 2] No such file or directory: '/opt/seafile/seafile-pro-server-7.0.14/seahub/seahub/../../../seafile-license.txt'
2020-04-29 14:50:04,232 [WARNING] seahub.utils.licenseparse:38 parse_license [Errno 2] No such file or directory: '/opt/seafile/seafile-pro-server-7.0.14/seahub/seahub/../../../seafile-license.txt'
2020-04-29 21:38:51,700 [WARNING] seahub.utils.licenseparse:38 parse_license [Errno 2] No such file or directory: '/opt/seafile/seafile-pro-server-7.0.14/seahub/seahub/../../../seafile-license.txt'
2020-04-29 21:39:29,482 [WARNING] seahub.utils.licenseparse:38 parse_license [Errno 2] No such file or directory: '/opt/seafile/seafile-pro-server-7.0.14/seahub/seahub/../../../seafile-license.txt'

EDIT:
Apologies - I missed this question

I ran seaf-fsck.sh before rebuilding the repos, and after the syncs first began to hang again.

EDIT 2
Using the windows sync client and the same account credentials as the failing seaf-cli I’ve been able to sync two of the troublesome repos without issue. Windows client on a different (200/20) connection, to the Linux client (30/10), but both passing through nginx.

I’ve spent the afternoon sniffing traffic on both firewalls and checking logs after clearing down the seaf-cli client yet again, and achieved absolutely nothing apart from burning another day trying with the smallest problem repo (128 KB total).

In the client logs I can see the same behaviour as before for the problem repos:

  • A libcurl failure to post to pack-fs reporting timeout
  • A TX-MGR logged error “Failed to get fs objects for repo…”
  • A clone state of error with ‘Data transfer timed out. Please check network or firewall’

This seems to occur on a 5 minute loop from the client.

HTTPS timeouts on the Watchguard are set to ridiculously high levels (1 hour) to remove it from the equation.
All the timeouts I can find in nginx and seafile configurations are set to 3600 and I can find nothing to look at in the server-side logs except in seafhttp.access.log from nginx which seems to be successfully passing the request through:

"GET /seafhttp/repo/54d15b3e-5f2c-4afa-9d03-b17d12e9fb26/fs-id-list/?server-head=0e92e8c8102524f7fcc97555f344ccc16d9746be HTTP/1.1" 200 1936 "-" "Seafile/7.0.4 (Linux)" 0.000

This only leaves me looking at the Meraki MX gateway in front of the client as a possible issue, so I’ve whitelisted the relevant policies and can see no problems with that when reviewing the traffic.

However none of the above can explain why:

  1. the problem repos sync without issue to a Windows client using the same account credentials
  2. Some repos sync without a problem to the seaf-cli client.

I think I might reset the seaf-cli client again tonight in case there’s some corruption there, however I am at a complete loss.

TazUk,

In order to understand what is happening, we need the logs of seafile(most important) and seahub at the moment you trigger a download with seaf-cli on linux (it fails when you do this)

1)Does it fail immediately? or it takes time?

2)Also you didn’t answer my question about system resources. How much ram do you have? do you see swapping? What cpu do you have? Also how complicated is your network layout?

  1. Also what did seafile fsck say? did it give you any bad blocks?

  2. When you are downloading with the windows client how long does it take and are you downloading from SCRATCH?

  3. Also the query that is timing out is a POST query, not GET.

The desktop clients might not behave the same way with the cli, here is a link explaining further

https://seafile.gitbook.io/seafile-server-manual/developing/server-components

@kaangoksal

It’s the Meraki gateway causing the issue and blocking the transfer, but only for the

I’ve just moved the linux seaf-cli box to a new location outside the Meraki perimeter and all repos now appear to be downloading without issue.

Similarly I’ve also placed a Windows laptop using the sync client within the Meraki perimeter, which is able to sync the repos without issue.

Therefore the combination of linux seaf-cli client behind the Meraki firewall is timing out.
Quite why this only occurs for some repos, I have no idea - I welcome any suggestions.