Some large uploads failing

Seafile Client
1

I’m trying to use the Linux seaf-cli to upload around 550 GB of files to my Seafile installation. There’s around 1500 files, and the largest files are around 2GB.

I tried it last night and seaf-cli status said “synchronized”, when in reality only ~50 GB of the files had been successfully synced.

I tried it again today after desyncing then resyncing the folder, and seaf-cli just seems to have crashed during the process

There’s a lot of logs like this:

[06/12/21 12:47:21] ../common/fs-mgr.c(896): Failed to chunk file with CDC.
[06/12/21 12:47:21] repo-mgr.c(1220): Failed to index file /data/temp/tempup/some-files/aaa/aaa.
[06/12/21 12:47:21] ../common/fs-mgr.c(607): Failed to write chunk b6859e5f4b1da21ec9875d20fcb6d58e92aca1b4.
[06/12/21 12:47:21] CDC: failed to write chunk.
[06/12/21 12:47:21] ../common/fs-mgr.c(896): Failed to chunk file with CDC.
[06/12/21 12:47:21] repo-mgr.c(1220): Failed to index file /data/temp/tempup/some-files/bbb/bbb.
[06/12/21 12:47:22] ../common/fs-mgr.c(607): Failed to write chunk b44514117b913944141f072543362467700c162f.
[06/12/21 12:47:22] CDC: failed to write chunk.
[06/12/21 12:47:22] ../common/fs-mgr.c(896): Failed to chunk file with CDC.
[06/12/21 12:47:22] repo-mgr.c(1220): Failed to index file /data/temp/tempup/some-files/ccc/ccc.
[06/12/21 12:47:22] start to serve on pipe client
[06/12/21 12:47:22] ../common/fs-mgr.c(607): Failed to write chunk 753f53b3f0e274a51bf64987cd7052f50a3b06dd.
[06/12/21 12:47:22] CDC: failed to write chunk.
[06/12/21 12:47:22] ../common/fs-mgr.c(896): Failed to chunk file with CDC.
[06/12/21 12:47:22] repo-mgr.c(1220): Failed to index file /data/temp/tempup/some-files/ddd/ddd.
[06/12/21 12:47:22] ../common/fs-mgr.c(607): Failed to write chunk 3ce294763fb5690bd38f7707e4ce0c3c2ff5039e.
[06/12/21 12:47:22] CDC: failed to write chunk.
[06/12/21 12:47:22] ../common/fs-mgr.c(896): Failed to chunk file with CDC.
...

And the very end of the log, when it crashed, I see these errors:

[06/12/21 12:52:52] repo-mgr.c(3856): All events are processed for repo 724127b6-715b-42b3-9b54-db5f30cc3fe9.
[06/12/21 12:52:52] sync-mgr.c(574): Repo 'TempLinuxUpload' sync is finished but with error: Error when indexing
[06/12/21 12:52:52] sync-mgr.c(582): Repo 'TempLinuxUpload' sync state transition from 'committing' to 'error'.
[06/12/21 12:52:52] sync-mgr.c(881): Removing blocks for repo TempLinuxUpload(724127b6).
[06/12/21 12:52:52] start to serve on pipe client
[06/12/21 12:52:52] sync-mgr.c(881): Removing blocks for repo TempLinuxUpload(724127b6).
[06/12/21 12:52:52] ../common/block-backend-fs.c(381): Failed to open block dir /home/daniel/seafile-client/seafile-data/storage/blocks/724127b6-715b-42b3-9b54-db5f30cc3fe9/c1.
[06/12/21 12:52:52] sync_error_id_to_str: assertion '(error >= 0 && error < N_SYNC_ERROR_ID)' failed[06/12/21 12:52:52] sync-mgr.c(574): Repo '<E0><ED><EF>^L<C6>U' sync is finished but with error: Unknown error

My setup:

  • The server I’m running seaf-cli on has several terabytes free, so disk space on the client side shouldn’t be an issue
  • seafile-data is located on a storage server with 10 TB disk space. It’s got around 7 TB free. Again, space shouldn’t be an issue there.
  • seafile-data is shared via NFS, which seems to be a valid approach given it’s documented: Setup Seafile cluster with NFS - Seafile admin manual
  • The server running Seafile has 16GB RAM and around 40GB disk space available
  • I have client_max_body_size 0 in the location / block, and proxy_request_buffering off in the location /seafhttp block in my Nginx config, as recommended in the docs
  • I haven’t encountered any issues like this before. My Seafile server has around 200GB of files on it, and all of those synced properly with no errors (many of them uploaded via seaf-cli).

Any ideas?

Edit: I think this may be because the /home partition on the client system only had 5 GB free. I moved /home/daniel/seafile-client to /data/seafile-client as the /data partition has terabytes of free space, and modified /home/daniel/.ccnet/seafile.ini to point to the new path, and it seems to have progressed further now.

Feature request: If that’s the case (it failed due to low disk space), the client should show more useful error messages like “insufficient disk space” rather than just “CDC: failed to write chunk”.

1 Like
2

The upload worked perfectly after relocating the client’s seafile-client directory! A better error message would be appreciated for this issue, so I don’t have to do so much digging if it happens again :slight_smile:

1 Like
3

hi… I found a similar report when upload large (13.6GB) file via NFS. May you test by removing some large files?

forum topic >>> /uploading-large-files-to-one-library-fails/1295

4

It actually worked fine for me once I moved the client-side seafile-client directory to a larger partition. I managed to upload all 550 GB of files with no issues.

The thread you mentioned is talking about storing client-side data on NFS. I’m storing the server-side seafile-data directory on NFS, which is working fine.

1 Like