Large text files upload

Hi,

On Seafile pro 6.2.10, i have some difficulties to upload large text files (*.txt) > 10 MB.
The web upload is at 1.5 MB/s

The web upload process hangs during 2-3 minutes at 50%, then goes to 100 %, but hangs at final saving step.

The files are saved but the process window hangs at 100 % and finally ends in 5 mn

This problem does not occur with XLSX files with the same data in it. I Suppose that the problem comes from text file format.

With the Windows client, the process stops but finnally ends with sucess

Here are the client logs

[03/25/18 11:36:39] http-tx-mgr.c(3423): Upload with HTTP sync protocol version 2.
[03/25/18 11:36:39] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'init') --> ('normal', 'check')
[03/25/18 11:36:39] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'check') --> ('normal', 'commit')
[03/25/18 11:36:39] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'commit') --> ('normal', 'fs')
[03/25/18 11:36:39] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'fs') --> ('normal', 'data')
[03/25/18 11:36:49] http-tx-mgr.c(912): libcurl failed to PUT https://partages.campus-condorcet.fr/seafhttp/repo/edaf2603-4108-47d2-a2ba-4131d1fbac62/block/c24c9bd4d60e4f31ff487e3696127d2c032aa1e5: Failed sending data to the peer.
[03/25/18 11:36:49] http-tx-mgr.c(912): libcurl failed to PUT https://partages.campus-condorcet.fr/seafhttp/repo/edaf2603-4108-47d2-a2ba-4131d1fbac62/block/e43ff23b03e2a53dbc0268090c70a1a86ef641cf: Operation was aborted by an application callback.
[03/25/18 11:36:49] http-tx-mgr.c(912): libcurl failed to PUT https://partages.campus-condorcet.fr/seafhttp/repo/edaf2603-4108-47d2-a2ba-4131d1fbac62/block/bde10d98d8fe50b531176c997be07bfb7f1fd6c9: Operation was aborted by an application callback.
[03/25/18 11:36:49] http-tx-mgr.c(912): libcurl failed to PUT https://partages.campus-condorcet.fr/seafhttp/repo/edaf2603-4108-47d2-a2ba-4131d1fbac62/block/db95878086ad77e24bb16eb9c8b4bf282b4c1f8e: Operation was aborted by an application callback.
[03/25/18 11:36:49] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'data') --> ('error', 'finished')
[03/25/18 11:36:49] sync-mgr.c(764): Repo 'nat2015_txt' sync state transition from uploading to 'error': 'Error occured in upload.'.
[03/25/18 11:37:09] sync-mgr.c(702): Repo 'nat2015_txt' sync state transition from 'synchronized' to 'uploading'.
[03/25/18 11:37:09] http-tx-mgr.c(3423): Upload with HTTP sync protocol version 2.
[03/25/18 11:37:09] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'init') --> ('normal', 'check')
[03/25/18 11:37:09] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'check') --> ('normal', 'commit')
[03/25/18 11:37:09] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'commit') --> ('normal', 'fs')
[03/25/18 11:37:09] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'fs') --> ('normal', 'data')
[03/25/18 11:37:34] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'data') --> ('normal', 'update-branch')
[03/25/18 11:37:35] http-tx-mgr.c(1132): Transfer repo 'edaf2603': ('normal', 'update-branch') --> ('finished', 'finished')
[03/25/18 11:37:35] sync-mgr.c(702): Repo 'nat2015_txt' sync state transition from 'uploading' to 'initializing'.
[03/25/18 11:37:35] sync-mgr.c(1516): Removing blocks for repo nat2015_txt(edaf2603).

Text files are attached here (7 days) : you can decompress and test it

This text file (70 MB) never stops uploadinG…

Regards

Gautier

Both files upload fine on my server. https://home.hoeper.me/f/7ed5261d5dd6496e875b/?raw=1

Thank you for testing.

It’s weird, because this problem occurs on two separated servers (dev, prod) on my side.

My seafile version is :

Pro 6.2.10
+MariaDb
+Apache
+Memcache
+Proxypass WCGI

Restarting Seafile and Apache has no effect.
The process still hangs at 41%

Envoi du fichier en cours … 41% (1.78 MB/s)

I’ll think about for a firewall issue if no one can reproduce the problem.

Regards

I currently use pro 6.2.9

Have you tried creating a new test library and trying to upload them there? If they work, then you’ll at least know the problem is with the library rather than something else.

Hi,
It does not work, unfortunately…
Thank you for the recommandation.

That error is often thrown due to a connection timing out, especially with Apache. Sometimes, it is due to a bug. Did it start after upgrading to 6.2.10? Seafile has had this particular bug in the past, albeit with older versions.

I think, you can check your ‘seahub_settings.py’ to make sure if configure the ‘ENABLE_RESUMABLE_FILEUPLOAD = True’. Sometimes this will make a mistake.

1 Like

HI, i have set this variable to True but it does not change anything about text file upload.

@wthess, I think about a network latency problem on my home connection, i’ll check it at work.

Regards,

Gautier

In your Apache config, what is the proxy read timeout for seafhttp?

Also, what happens if you try uploading the file internally directly to seafile, such as logging into http://192.168.1.X:8000?

Hi,
I can’t test this because internally the server is 127.0.0.1

But, i tried any of these environment variables, with no success

  ProxyTimeout 600
  SetEnv proxy-sendchunked | SetEnv proxy-sendc

  SetEnv force-proxy-request-1.0 1
  SetEnv proxy-nokeepalive 1 

It is not so problematic to me because the issue is only limited to text files.
But i’m aware of it …

regards

127.0.0.1 is the loopback address of the server. It also has to have an IP address accessible from the internal network. You can find it by keying hostname -i in a terminal window.

Hi

I cant’ access to the host from the internal network but only from public IP and apache is configured in HTTPS / PROXY with :

SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
ProxyPass / http://127.0.0.1:8000/
ProxyPassReverse / http://127.0.0.1:8000/

Then if i try any of these

http://public_ip:8080/
http://hostname.domainA.xx:8080/
http://servername.domainB.xx:8080/

I can’t reach seahub (timeout)

regards

I think I’m confused here… You say you have no internal access to the server, but you stated earlier that you can stop and start Seafile. How are you accomplishing that? Do you have access to the server log files?

Sorry to make you confused.
I can access the server via SSH & HTTPS on its public IP.
It is hosted on a Linux VM iwth no browser at all.
I’m not in the same VLAN.

Ok… I understand now… Thanks for clarifying that… Are you behind Apache? If so, take a look at the link below. From what I’m reading, that setting is at 10 MB by default. Changing that value may solve your problem:

@wthess Hi i’m note sure it will help, because i can upload other files larger than 10Gb (XLS, ZIP, etc.)
And i’m quite sure php is not involved in the proxy process.

php is not involved with the proxy process directly. However, it is used to handle file transfers in Seafile. Right now, we aren’t certain whether it’s a proxy problem or API problem. Since you can’t access it internally and try it, there’s no way to tell at this point. If there were a way to bypass Apache and try it, then that would be ideal.

Text files are 7 bit… Binaries are 8 bit… Whole different animal. Therefore, encoding and decoding are involved with text files. Binaries are handled differently than text transfers. I’ve seen this behavior before in many other programs and file transfers over the years for varying reasons.

So, in summary, your problem could be due to several different factors and since we have limited ability to troubleshoot it, then the shotgun approach may be the key. It never hurts to try.

Think you confuse python with PHP here. PHP is not being used at all by Seafile

Yes, you are correct… Had a migraine today… Meant to talk about Python and Django…