I’m using seafile with linux seadrive client, and need to upload netcdf files of about 0.5Tb size.
My configuration uses debian 10, seafile 7.0.5, nginx with ssl as proxy, and percona mysql as database. Seadrive client is 1.0.9 with default 10Gb cache. Connection is 1Gbps both in server and client, with only Gb switches in-between
I’ve been able to copy a 100Gb file to seadrive and get it uploaded without problems (just wait), but when trying a 500Gb file, it gets copied to seadrive folder (takes several hours) but never gets uploaded to server.
Do you have any tip in configuration or performance tuning that I could have missing? I’ve followed the official recipe
Finally it gots uploaded!. May be it just needed some basic nginx tunning, or just more patience to get the file uploaded (seadrive client machine was very CPU busy!). We have used rsync -a folder seadrive/My\ Libraries/test_repo/ to copy the file.
Anyway, I attach the (sucessful) client log here:
[01/15/20 18:59:01] file-cache-mgr.c(2653): 1 files in cache, total size is 458018349056
[01/15/20 18:59:01] file-cache-mgr.c(2674): Cache cleaning done.
[01/15/20 18:59:01] file-cache-mgr.c(2676): Removed 0 files, cleaned up 0 MB
[01/15/20 18:59:52] cdc.c(166): File size changed while chunking.
[01/15/20 18:59:52] fs-mgr.c(376): Failed to chunk file with CDC.
[01/15/20 18:59:52] sync-mgr.c(2574): Failed to index file big_file_folder/.big500gb.nc.R4mwBn in repo aaaaaaaa-bbbb-4500-9aea-123456789012, skip.
[01/15/20 18:59:52] sync-mgr.c(2865): All operations of repo test_repo(aaaaaaaa) have been processed.
[01/15/20 18:59:56] sync-mgr.c(586): Repo 'test_repo' sync state transition from 'committing' to 'synchronized'.
[01/15/20 19:09:01] file-cache-mgr.c(2645): cache size limit is 20000000000
[01/15/20 19:09:01] file-cache-mgr.c(2651): Cleaning cache space.
[01/15/20 19:09:01] file-cache-mgr.c(2653): 1 files in cache, total size is 470899449856
[01/15/20 19:09:01] file-cache-mgr.c(2674): Cache cleaning done.
[01/15/20 19:09:01] file-cache-mgr.c(2676): Removed 0 files, cleaned up 0 MB
[01/15/20 19:16:19] sync-mgr.c(586): Repo 'test_repo' sync state transition from 'synchronized' to 'committing'.
[01/15/20 19:16:19] file-cache-mgr.c(2237): Failed to stat /home/user/.seadrive/data/file-cache/aaaaaaaa-bbbb-4500-9aea-123456789012/big_file_folder/.big500gb.nc.R4mwBn: No such file or directory.
[01/15/20 19:16:19] sync-mgr.c(2574): Failed to index file big_file_folder/.big500gb.nc.R4mwBn in repo aaaaaaaa-bbbb-4500-9aea-123456789012, skip.
[01/15/20 19:19:01] file-cache-mgr.c(2645): cache size limit is 20000000000
[01/15/20 19:19:01] file-cache-mgr.c(2651): Cleaning cache space.
[01/15/20 19:19:01] file-cache-mgr.c(2653): 1 files in cache, total size is 482967022096
[01/15/20 19:19:01] file-cache-mgr.c(2674): Cache cleaning done.
[01/15/20 19:19:01] file-cache-mgr.c(2676): Removed 0 files, cleaned up 0 MB
...
[01/15/20 21:39:02] file-cache-mgr.c(2645): cache size limit is 20000000000
[01/15/20 21:39:02] file-cache-mgr.c(2651): Cleaning cache space.
[01/15/20 21:39:02] file-cache-mgr.c(2653): 1 files in cache, total size is 482967022096
[01/15/20 21:39:02] file-cache-mgr.c(2674): Cache cleaning done.
[01/15/20 21:39:02] file-cache-mgr.c(2676): Removed 0 files, cleaned up 0 MB
[01/15/20 21:42:45] sync-mgr.c(2802): Creating partial commit after rename big_file_folder/.big500gb.nc.R4mwBn in repo test_repo(aaaaaaaa).
[01/15/20 21:42:46] sync-mgr.c(586): Repo 'test_repo' sync state transition from 'committing' to 'uploading'.
[01/15/20 21:42:46] http-tx-mgr.c(4057): Upload with HTTP sync protocol version 2.
[01/15/20 21:42:46] http-tx-mgr.c(1276): Transfer repo 'aaaaaaaa': ('normal', 'init') --> ('normal', 'check')
[01/15/20 21:42:46] http-tx-mgr.c(1276): Transfer repo 'aaaaaaaa': ('normal', 'check') --> ('normal', 'commit')
[01/15/20 21:42:46] http-tx-mgr.c(1276): Transfer repo 'aaaaaaaa': ('normal', 'commit') --> ('normal', 'fs')
[01/15/20 21:42:46] http-tx-mgr.c(1276): Transfer repo 'aaaaaaaa': ('normal', 'fs') --> ('normal', 'data')
[01/15/20 21:49:02] file-cache-mgr.c(2645): cache size limit is 20000000000
[01/15/20 21:49:02] file-cache-mgr.c(2651): Cleaning cache space.
[01/15/20 21:49:02] file-cache-mgr.c(2653): 1 files in cache, total size is 482967022096
[01/15/20 21:49:02] file-cache-mgr.c(2062): Failed to get path big_file_folder/big500gb.nc for repo aaaaaaaa.
[01/15/20 21:49:02] file-cache-mgr.c(2674): Cache cleaning done.
[01/15/20 21:49:02] file-cache-mgr.c(2676): Removed 0 files, cleaned up 0 MB
...
[01/15/20 23:39:02] file-cache-mgr.c(2645): cache size limit is 20000000000
[01/15/20 23:39:02] file-cache-mgr.c(2651): Cleaning cache space.
[01/15/20 23:39:02] file-cache-mgr.c(2653): 1 files in cache, total size is 482967022096
[01/15/20 23:39:02] file-cache-mgr.c(2062): Failed to get path big_file_folder/big500gb.nc for repo aaaaaaaa.
[01/15/20 23:39:02] file-cache-mgr.c(2674): Cache cleaning done.
[01/15/20 23:39:02] file-cache-mgr.c(2676): Removed 0 files, cleaned up 0 MB
[01/15/20 23:43:47] http-tx-mgr.c(1276): Transfer repo 'aaaaaaaa': ('normal', 'data') --> ('normal', 'update-branch')
[01/15/20 23:43:48] http-tx-mgr.c(1276): Transfer repo 'aaaaaaaa': ('normal', 'update-branch') --> ('finished', 'finished')
[01/15/20 23:43:48] sync-mgr.c(2509): removing blocks for repo aaaaaaaa-bbbb-4500-9aea-123456789012
[01/15/20 23:45:34] sync-mgr.c(586): Repo 'test_repo' sync state transition from 'uploading' to 'get sync info'.
[01/15/20 23:45:34] sync-mgr.c(586): Repo 'test_repo' sync state transition from 'get sync info' to 'synchronized'.
[01/15/20 23:45:34] sync-mgr.c(586): Repo 'test_repo' sync state transition from 'synchronized' to 'committing'.
[01/15/20 23:45:34] sync-mgr.c(2727): All operations of repo test_repo(aaaaaaaa) have been processed.
[01/15/20 23:45:34] sync-mgr.c(586): Repo 'test_repo' sync state transition from 'committing' to 'synchronized'.
[01/15/20 23:49:02] file-cache-mgr.c(2645): cache size limit is 20000000000
[01/15/20 23:49:02] file-cache-mgr.c(2651): Cleaning cache space.
[01/15/20 23:49:02] file-cache-mgr.c(2653): 1 files in cache, total size is 482967022096
[01/15/20 23:49:12] file-cache-mgr.c(2674): Cache cleaning done.
[01/15/20 23:49:12] file-cache-mgr.c(2676): Removed 1 files, cleaned up 482967 MB
[01/15/20 23:59:12] file-cache-mgr.c(2645): cache size limit is 20000000000
