HTTP/2 breaks some functionalities

Hi there,

since Seafile 7 Pro i’m not able to reupload existing files over any browser. I tried it with Chrome, Firefox and Edge.

pic2

Furthermore I could not save any files written/edited with the new markdown editor.

pic1

Both issues show the following error in nginx:

2019/06/21 07:11:11 [error] 22352#22352: *20361 upstream prematurely closed connection while reading upstream, client: XXX.XXX.XXX.XXX,server: XXXXXX, request: “POST /seafhttp/update-api/c0889161-8950-410d-8ea6-9bf6851c6d2c HTTP/2.0”, upstream: “http://127.0.0.1:8082/update-api/c0889161-8950-410d-8ea6-9bf6851c6d2c”, host: “XXXXXXXX”, referrer: “https://xxxxxxxxxx/lib/dc26e32b-71b9-4273-bc64-202af6fb6ba8/file/test.md

Finally I figured it out. It seems there is a problem with HTTP/2 enabled. The borwsers console says:

Failed to load resource: net::ERR_HTTP2_PROTOCOL_ERROR /seafhttp/update-api/…

If I turn off HTTP/2 and validate that it’s turned off, I don’t get these errors and I’m able to save my files (in all browsers).

I’m using NGINX 1.14.2 and Debian Stable so nothing exotic.

I am seeing an issue where saving an MD file in Safari generates an error message that the file cannot be saved (same error you show) when in fact it is. Removing http2 from Nginx config solves the problem.

In general I can upload files but am back to the situation where I cannot drag-and-drop into the file list area in the browser. I have to drop the file in the top button bar or left side bar area to upload it. This is a separate issue and has been reported earlier (Upload problems with Firefox 67.0.2 and 7.0.2 CE).

Nginx 1.14.0, Ubuntu 18.04, Seafile CE 7.0.3.

1 Like

Unfortunately it is not possible to disable HTTP/2 for just one vhost in nginx.
@daniel.pan Are you aware of this issue and will you provide a fix soon?

I’m afraid we don’t have a plan to fix it yet. Since seaf-server is proxied by Nginx, can you config Nginx to use http to talk with seaf-server, while Nginx use http/2 with the clients?

I tried that, but it is still the same. So I took me several hours but I made some debugging with different configurations.

with HTTP/2 ENABLED:

scenario 1 - upload new file via web browser
Upload is working proberly. After sending a file via URL https://<domain>.tld/seafhttp/upload-aj/… as POST
to seaf-server there will be a valid response from seaf-server. With valid I mean the Content-Length-Header of the response actually matches the content-length and the body looks something like this: fc7c9237fe10165fd7d826c85b6951c88118d088

Here is the corresponding output from Chrome’s NetLog-Viewer:

HTTP_TRANSACTION_READ_RESPONSE_HEADERS
→ HTTP/1.1 200
status: 200
server: nginx
date: Tue, 30 Jul 2019 10:27:23 GMT
content-type: application/json; charset=utf-8
content-length: 111
access-control-allow-headers: x-requested-with, content-type, content-range, content-disposition, accept, origin, authorization
access-control-allow-methods: GET, POST, PUT, PATCH, DELETE, OPTIONS
access-control-allow-origin: *
access-control-max-age: 86400
x-xss-protection: 1; mode=block
x-robots-tag: none
strict-transport-security: max-age=63072000; includeSubDomains
-HTTP_TRANSACTION_READ_HEADERS
+NETWORK_DELEGATE_HEADERS_RECEIVED [dt=2]
HTTP2_STREAM_UPDATE_RECV_WINDOW
→ delta = -111
→ stream_id = 53
→ window_size = 6291345
-NETWORK_DELEGATE_HEADERS_RECEIVED

scenario 2 - reupload existing file via web browser and thus replace it
After sending a file via URL https://<domain>.tld/seafhttp/update-api/… as POST any upload failes because the response’s actual content-length and its content-length-header doesn’t match anymore. HTTP/2 implements flow control. Therefore it’s necessary that they match. This time the response body is not readable and much bigger than it’s corresponding header. So replacing files with HTTP/2 is generally working, but malformed POST responses are the problem here. That also explains why saving MD files with HTTP/2 generates error messages like “Could not save file” but in fact it is saved.

Chrome’s NetLog-Viewer says:

HTTP_TRANSACTION_READ_RESPONSE_HEADERS
→ HTTP/1.1 200
status: 200
server: nginx
date: Tue, 30 Jul 2019 09:03:37 GMT
content-type: application/json; charset=utf-8
content-length: 40
access-control-allow-headers: x-requested-with, content-type, content-range, content-disposition, accept, origin, authorization
access-control-allow-methods: GET, POST, PUT, PATCH, DELETE, OPTIONS
access-control-allow-origin: *
access-control-max-age: 86400
x-xss-protection: 1; mode=block
x-robots-tag: none
strict-transport-security: max-age=63072000; includeSubDomains
-HTTP_TRANSACTION_READ_HEADERS
+NETWORK_DELEGATE_HEADERS_RECEIVED [dt=1]
HTTP2_STREAM_UPDATE_RECV_WINDOW
→ delta = -541
→ stream_id = 23
→ window_size = 6290915
HTTP2_STREAM_ERROR
→ description = “Server reset stream.”
→ net_error = “ERR_HTTP2_PROTOCOL_ERROR”
→ stream_id = 23
-NETWORK_DELEGATE_HEADERS_RECEIVED
HTTP_TRANSACTION_READ_BODY [dt=0]
net_error = -337 (ERR_HTTP2_PROTOCOL_ERROR)

Since uploading a new file (using /upload-aj/… ) is working fine via HTTP/2 and replacing it (using /update-api/…) isn’t there could be a bug in how the responses are made.

Further information how HTTP/2 handles malformed requests/responses are available here: https://http2.github.io/http2-spec/#malformed

Information about flow-control are available here:
https://http2.github.io/http2-spec/#WINDOW_UPDATE

Hopefully I could provide some useful information.
@daniel.pan Please make some futher investigation.
For me seafile stands for its reliable and fast file sync service with very modern features like delta-sync. In my opinion it would be sad, if HTTP/2 could not be used because of malformed response bodies.

Since http/2 is working fine in my constellation with Apache instead of nginx, I’m not sure, if seafile is the problem.

best regards,
Markus

1 Like

I don’t know what the issue is, but if there is some issue with HTTP/2 compatibility it really should be fixed.

3 Likes

@MaEh Which seafile version do you use?
Seafile 6 works quite fine which HTTP/2.
At least I’ve been using Seafile 6 PRO for a long time with HTTP/2.

A brief look into seaf-servers code shows that all the magic happens in upload-file.c
Since Seafile 7 they changed the function which is called to create responses due to compatibilty for IE8.
I havn’t enough time to go deeper, yet. But I don’t think that’s my task either. I’m not a C developer.

Hopefully someone from Seafile Ltd. will look futher into this and provide information. I don’t know any reason that justifies why it works in Seafile 6 but not Seafile 7.

I’m using seafile 7.0.4 Community Edition with Apache as Proxy and no Problems with http/2.

best regards,
markus

We will check the problem later.

2 Likes

@daniel.pan I see that there is a fix for HTTP/2 in Seafile 7.1.2 Pro (beta). It this fix related to this forum thread?

Yes, the fix is related to the thread.

1 Like