[BOGUS] Help please: with a HAproxy phenomenon

Hi folks
I’m having problems with HAproxy. Following error.
I think this is HAproxy specific.

[11/13/19 13:25:53] http-tx-mgr.c(808): libcurl failed to GET https://MYDOMAIN/seafhttp/repo/fdd4d58d-e139-47e8-ab8d-037738a8e9e7/block/ca77eaf32891b535929eed39d8474cfa9316192d: Transferred a partial file.

Setup

  • HAproxy on Router
  • Seafile on private DMZIP in DMZ
  • HAproxy does SSL Termination, Seafile runs Nginx on port 80, all else Seafile is on 127.0.0.1:ports

Works OK:

Works NOT OK:

More appearences:

  • WAN Link is 1Gbit, usually around 400Mbit rate peak (lower but stable)
  • DMZ/LAN Link is 10Gbit, highest peak around 1.5Gbit (higher but with described Error)
  • LAN-to-DMZ (http://DMZIP:80) goes up to over 1Gbit and then errors
  • LAN-to-HAproxy-to-DMZ eventually succeeds syncing after the 10x sync time (retransferring again and again)
  • Same thing happens if I do a HAproxy-Seafile-only setup without Nginx.
  • Memory and CPU on HAproxy is plenty, though I suspect some kind of buffer/rate limit problem.
  • The error seems not to appear anywhere on the Seafile server (I may be wrong)
  • I get a lot of OK (Status 200) requests on HAproxy with paths “/seafhttp/repo/SOMEID/block/SOMEOTHERID” so the mapping should be right

The biggest difference between LAN-access-Proxy and WAN-access-Proxy from what I know is that the inbound src-IPs for WAN will be addressed via outbound-NAT at all cases, whereas the former may tempt an app to bypass the proxy. I saw this once with seafile. Would mean I need to force Seafile to not do any bypass tricks?

Can somebody help please?
Thank you very much

Version: seafile-server-7.0.5 CE on Ubuntu 16

Update: Maybe unrelated, as I cannot say it’s absolutely realtime with the Sync Error
/logs/seahub.log says:

2019-11-13 14:18:10,151 [ERROR] django.request:135 handle_uncaught_exception Internal Server Error: /api2/account/info/
Traceback (most recent call last):
  File "/media/seaer/seafile-server-7.0.5/seahub/thirdpart/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/media/seaer/seafile-server-7.0.5/seahub/thirdpart/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/media/seaer/seafile-server-7.0.5/seahub/thirdpart/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/media/seaer/seafile-server-7.0.5/seahub/thirdpart/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/media/seaer/seafile-server-7.0.5/seahub/thirdpart/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/media/seaer/seafile-server-7.0.5/seahub/thirdpart/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/media/seaer/seafile-server-7.0.5/seahub/seahub/api2/base.py", line 23, in dispatch
    response = super(APIView, self).dispatch(*a, **kw)
  File "/media/seaer/seafile-server-7.0.5/seahub/thirdpart/rest_framework/views.py", line 466, in dispatch
    response = self.handle_exception(exc)
  File "/media/seaer/seafile-server-7.0.5/seahub/seahub/api2/base.py", line 20, in handle_exception
    return super(APIView, self).handle_exception(exc)
  File "/media/seaer/seafile-server-7.0.5/seahub/thirdpart/rest_framework/views.py", line 463, in dispatch
    response = handler(request, *args, **kwargs)
  File "/media/seaer/seafile-server-7.0.5/seahub/seahub/api2/views.py", line 340, in get
    return Response(self._get_account_info(request))
  File "/media/seaer/seafile-server-7.0.5/seahub/seahub/api2/views.py", line 333, in _get_account_info
    info['is_inst_admin'] = request.user.inst_admin
AttributeError: 'User' object has no attribute 'inst_admin'

this is the nginx access log:

 ....
10.24.22.16 10.24.62.1 [13/Nov/2019:17:56:16 +0100] "GET /seafhttp/repo/fdd4d58d-e139-47e8-ab8d-037738a8e9e7/block/80916df5d750ff7cccf267b6ad13e44ed5b10f3c HTTP/1.1" 200 10485760 "-" "Seafile/7.0.2 (Apple OS X)" 0.019
10.24.22.16 10.24.62.1 [13/Nov/2019:17:56:16 +0100] "GET /seafhttp/repo/fdd4d58d-e139-47e8-ab8d-037738a8e9e7/block/8f80fa0b4f16fce61031248d1872905d4b9fae51 HTTP/1.1" 200 8236044 "-" "Seafile/7.0.2 (Apple OS X)" 0.006
10.24.22.16 10.24.62.1 [13/Nov/2019:17:56:16 +0100] "GET /seafhttp/repo/fdd4d58d-e139-47e8-ab8d-037738a8e9e7/block/bb0d60bb3d83248a43ea31a43305c4580d2f2077 HTTP/1.1" 200 5436674 "-" "Seafile/7.0.2 (Apple OS X)" 0.019
<connection drop>

10.24.22.16 10.24.62.1 [13/Nov/2019:17:56:20 +0100] "GET /seafhttp/protocol-version HTTP/1.1" 200 14 "-" "Seafile/7.0.2 (Apple OS X)" 0.000
10.24.22.16 10.24.62.1 [13/Nov/2019:17:56:20 +0100] "GET /seafhttp/repo/fdd4d58d-e139-47e8-ab8d-037738a8e9e7/commit/HEAD HTTP/1.1" 200 81 "-" "Seafile/7.0.2 (Apple OS X)" 0.002
10.24.22.16 10.24.62.1 [13/Nov/2019:17:56:21 +0100] "GET /seafhttp/repo/fdd4d58d-e139-47e8-ab8d-037738a8e9e7/permission-check/?op=download&client_id=3194915dee299e918fd7a7e78081272829cc7789&client_name=vm-mpro.local HTTP/1.1" 200 0 "-" "Seafile/7.0.2 (Apple OS X)" 0.087
....

nginx error log is silent
ccnet.log, seafile.log, seahub.log are silent.

A memory issue?

A direct comparison, a bit later (identical requests):

Seafile Client Error Log:
[11/13/19 18:06:44] http-tx-mgr.c(808): libcurl failed to GET https://MYDOMAIN/seafhttp/repo/fdd4d58d-e139-47e8-ab8d-037738a8e9e7/block/8a506f9bee6333832f9a8366f47cc52ff1882077: Transferred a partial file.

HAproxy Log:
haproxy[4899]: {"ip":"10.24.22.16","srcport":"59915","date":"13/Nov/2019:18:06:43.977","timestamp":"1573664803","backserv":"BACK_SEAFILE/hydra_seafile","transport":"gate2INTL_http443~","requestheader":"MYDOMAIN - - -","requesttype":"GET","fulluri":"/seafhttp/repo/fdd4d58d-e139-47e8-ab8d-037738a8e9e7/block/8a506f9bee6333832f9a8366f47cc52ff1882077","status":"200","body":"MYDOMAIN"}

Weird, I have no idea where it even breaks.

Or higher level, something cookie’ish…?

It turned out that using a 10G Link was the problem.
I had to fine-tune my OPNsense Router quite a bit for such a scenario.
WAN probably had no problem because that is a 1G Link.
The problems don’t appear anymore.

How can the 10g be the problem? Virtual or hardware opnsense?

First Virtual (SR-IOV), later Hardware, the same.
I was astonished too.
I also get about double the perf in routing without IPS compared to a vanilla OPNsense install. HBSD 11 that is.

I read something about HBSD 12 handling 10G links differently. Do you know something about this topic?