[9.0.6@Linux] Uploading/rclone issues (one-byte offs and 400 bad requests)

Seafile Server
#1

Dear,

after years using Nextcloud (since V14 now V23) I in the progress to migrate to Seafile.

Setup:

VM1: Devuan Linux Chimera for Seafile 9.0.6
VM2: Devuan Linux Chimera for MySQL

Everything works (Android and iOS clients, HTTPS with Apache proxy, new libraries etc.).
However syncing my data from my Nextcloud VM using rclone 1.56.1 like

rclone sync -i Collection\ -\ Software wolke8:"Collection - Software"

where “Collection - Software” is the local directory holding the SOURCE data on an error-free ext4 filesystem and “Wolke 8” is my new Seafile server gives me reproduceable errors:

2022/07/27 17:44:40 ERROR : 01 Software Archiv/05 Anwendungen/Konz - 1000 legale Steuertricks/Bin/Data/Images/Tip_S/T0263S.jpg: corrupted on transfer: sizes differ 2836 vs 2837
2022/07/27 17:44:48 ERROR : 01 Software Archiv/05 Anwendungen/Konz - 1000 legale Steuertricks/Bin/Data/Images/Tip_S/T0630S.jpg: corrupted on transfer: sizes differ 2836 vs 2837
2022/07/27 17:44:55 ERROR : 01 Software Archiv/05 Anwendungen/Konz - 1000 legale Steuertricks/Bin/Data/Images/Tip_S/T0830S.jpg: corrupted on transfer: sizes differ 2836 vs 2837
2022/07/27 17:45:03 ERROR : 01 Software Archiv/05 Anwendungen/Konz - 1000 legale Steuertricks/Bin/Data/Images/Tip_S/T0886S.jpg: corrupted on transfer: sizes differ 2836 vs 2837

2022/07/27 17:38:25 ERROR : TO SORT/064/ALPHA/PROFEXT.DL_: corrupted on transfer: sizes differ 7001 vs 7002
2022/07/27 17:38:39 ERROR : TO SORT/064/PPC/MDMVAYRS.IN_: corrupted on transfer: sizes differ 2906 vs 2907
2022/07/27 17:38:49 ERROR : TO SORT/064/ALPHA/8514FIXG.FO_: Failed to copy: failed to upload file: HTTP error 400 (400 Bad Request) returned body: “{“error”: “Bad request.\n”}”
2022/07/27 17:38:51 ERROR : TO SORT/064/I386/DISKPERF.SY_: Failed to copy: failed to upload file: HTTP error 400 (400 Bad Request) returned body: “{“error”: “Bad request.\n”}”

I have synced about 2.5 TByte with rclone so far. Large files work great. Many small files not as good (reaaally slow). However these errors like above are the same, regardless if I restart the VM or the services (including Apache and memcached). And it’s a just one-byte bug (which is suspicious).

After like 200 tries some files might be copied. But for most it does not work using rclone. If I try to copy the failed files using the web frontend they will fail with “Network errors” as well - so it’s not based on rclone.

Any ideas/suggestions?

Thanks, -MN

#2

This is the rclone Output for three small files (2 PNGs, 1 HTML):

2022/07/29 08:41:19 NOTICE:
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 22440 / 22440, 100%
Transferred: 0 / 3, 0%
Elapsed time: 9h49m0.1s
Transferring:

  • 23 eXXX/40071 XXXXMO/8…oject_Size.1degree.png: transferring
  • 23 eXXX/40071 XXXXMO/8…oject_Type.1degree.png: transferring
  • Management (Projekte)/…nsmodell/_img/_img.htm: transferring

2022/07/29 08:42:19 NOTICE:
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 22440 / 22440, 100%
Transferred: 0 / 3, 0%
Elapsed time: 9h50m0.1s
Transferring:

  • 23 eXXX/40071 XXXXMO/8…oject_Size.1degree.png: transferring
  • 23 eXXX/40071 XXXXMO/8…oject_Type.1degree.png: transferring
  • Management (Projekte)/…nsmodell/_img/_img.htm: transferring

2022/07/29 08:43:19 NOTICE:
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 22440 / 22440, 100%
Transferred: 0 / 3, 0%
Elapsed time: 9h51m0.1s
Transferring:

  • 23 eXXX/40071 XXXXMO/8…oject_Size.1degree.png: transferring
  • 23 XXXX/40071 XXXXMO/8…oject_Type.1degree.png: transferring
  • Management (Projekte)/…nsmodell/_img/_img.htm: transferring

2022/07/29 08:44:19 NOTICE:
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 22440 / 22440, 100%
Transferred: 0 / 3, 0%
Elapsed time: 9h52m0.1s
Transferring:

  • 23 eXXX/40071 XXXXMO/8…oject_Size.1degree.png: transferring
  • 23 eXXX40071 XXXXMO/8…oject_Type.1degree.png: transferring
  • Management (Projekte)/…nsmodell/_img/_img.htm: transferring

This runs almost 10 hours for about 5 KByte of data.

I am not sure if this increases my feelings about data integrity and stability…

Thanks, -MN

#3

What are you syncing to? WebDAV?

Seafile stores files in blocks. Hence copying them over like that won’t work. But you might know this already.

I’d suggest to use Seafile CLI to sync the data is efficiently. If you have a GUI in your Linux VM you may also use the Seafile Sync client.

You should also think about splitting the data into multiple libraries for performance and convenience with selective data sync.

#4

Having the same issue.

2023/07/18 20:23:10 ERROR : thumbnails/0000315.webp: corrupted on transfer: sizes differ 7126 vs 7127

Always the same files…

Anyone with new hints?

using rclone in a docker

#5

Looks like you can avoid the file size errors with --ignore-size which can occur if the seafile backend has another opinion about the filesize :slight_smile:

But i’m still getting very rare 2023/07/18 20:36:42 ERROR : thumbnails/0003171.webp: Failed to copy: failed to upload file: HTTP error 400 (400 Bad Request) returned body: “{“error”: “Bad request.\n”}”