While testing performance we noticed that the up- and download is drastically slowed down with http/2 enabled. We tested this on multiple systems in multiple geo locations with different systems and OS versions. Using http/2 is a throughput bootleneck whereas it’s supposed to improve it.
Some example to see the huge difference we are talking about (MB/s, not Mbit/s):
Upload with http/2 enabled: 9 MB/s
Upload with http/2 disabled: 52 MB/s
Download with http/2 enabled: 18 MB/s
Download with http/2 disabled: 120 MB/s
Tested with: CE and PRO 7 + 8 (different versions)
Nginx running on all instances as public proxy.
Important fun fact: When using yet another proxy in front of nginx e.g. haproxy, the problem doesn’t seem to exist??!
[ "Seafile/Seahub" --> "Nginx" with private CA signed cert ] -->
[ haproxy --> public ssl offloading ] --> Internet --> Webbrowser/Client
I found another http/2 related thread that might give some clues.
@daniel.pan@Jonathan Can you please check what might cause the issue here? @Community: Maybe someone else can to do some tests and report their findings.
Also thought about that. Some time in the past this was the way to configure the nginx vhost for Seafile to make it work properly. Maybe it’s time to remove it now. @Jonathan Can you please confirm that proxy_http_version 1.1; is not required anymore!?
Nginx with proxy_request_buffering set to ‘on’: This makes Nginx buffer the entire file before sending to upstream (seaf-server). In this setting, the upload speed is totally decided by Nginx. Result: The upload speed is normal.
Nginx with proxy_request_buffering set to ‘off’: Nginx will forward the data to seaf-server upon receiving it. In this setting, the upload speed is decided by both Nginx and seaf-server. Nginx will convert from http2 to http 1.1. Result: The upload speed is only half of setting 1.
Nginx with proxy_request_buffering set to ‘off’, and replace seaf-server with the new fileserver written in Go language. The result is the same as setting 2.
All tests use Chrome as the client.
We could assume the http handling of Go standard library is quite efficient and conforms to standards. Given that the speed of settings 2 and 3 has no difference, we think the bottleneck is in the conversion from http2 to http 1.1 in Nginx.
One possible solution is to setup as setting 1 in my last reply. Actually it has another benefit that uploads from slow connections will not take up a worker thread in seaf-server for long time. Concurrency is improved.
We’ll have to do more test to get recent results but if I remember correctly we’ve had a lot of trouble with nginx cache enabled in the past. This just doesn’t work with e.g. Webdav uploads and large files. It also fails with many files in a short period of time. Also this doesn’t seem a stable choice with multiple frontend nodes and resumable uploads. One would then also need to put the nginx cache on a shared storage to be available from all frontend nodes, to allow certain features. How large is the nginx cache supposed to be then? Please imagine thousand of connections and a least a couple of hundread users uploading something at the same time.
Did you test your suggestion with large or huge setups? At least multiple nodes and with different use cases?
We haven’t tested with very large deployment ourselves with this option, as we ourselves don’t have such deployment requirement. However from the Nginx documentation: Module ngx_http_proxy_module, this option is enabled by default. If it’s not stable it would not be enabled by default or should be marked as experimental.
Digging deeper into the Nginx documentation, I think you may also need to tune the buffer size and provide enough space for the Nginx temp file directory. Module ngx_http_core_module
This setting is only local to Nginx, it is not related to clustering at all.
Disabling this option doesn’t solve the case that a lot of temp files need to be cached in local disk. It only move the cache from Nginx’s temp directory to Seafile’s temp directory. Since you can easily have relatively large local disk nowadays it should be easy to solve. Also you can add more nodes to avoid having too much concurrent uploads for each node.
In the future when go fileserver is stable enough. A solution would be to directly expose fileserver to the users, without Nginx in the middle. Go fileserver supports http2 natively and has good concurrency. You can deploy a few nodes dedicated to file upload/download and syncing API.