Nginx - bad gateway after a while

Seafile Server
1

Hi,

I have problems with my seafile configuration since a while. After a while (sometimes a few days but sometimes even weeks or months) when my seafile server is running nonstop I encounter the problem that I get a 502 - bad gateway response from seafile. The easy solution is when I restart my server (raspberry pi 2) then everything works again for a while.

The problem is that I even don´t know where I should start to find the problem. I looked up in the seafile log files but there were no noticeable problems. I also looked up in the nginx error files and found the corresponding error which is written down below. Maybe some of you have an idea where I should look.

Any help would be great.

Versions:
Seafile-server: 6.2.5
Nginx: 1.10.3

Nginx-error-log:

2018/04/08 15:29:37 [error] 690#690: *1775 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.x, server: myserver.net, request: “GET /api2/account/info/ HTTP/1.1”, upstream: “http://127.0.0.1:8000/api2/account/info/”, host: “myserver.net”

Seahub_django_request.log: (there were often errors like this in the time where there was bad gateway but by looking at the timestamp I saw that there was no error exactly when there were the Bad-Gateway-error at nging-error-log (compare timestamp from above)

2018-04-07 22:52:31,412 [ERROR] django.request:256 handle_uncaught_exception Internal Server Error: /api2/events/
Traceback (most recent call last):
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/thirdpart/Django-1.8.18-py2.7.egg/django/core/handlers/base.py”, line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/thirdpart/Django-1.8.18-py2.7.egg/django/views/decorators/csrf.py”, line 58, in wrapped_view
return view_func(*args, **kwargs)
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/thirdpart/Django-1.8.18-py2.7.egg/django/views/generic/base.py”, line 71, in view
return self.dispatch(request, *args, **kwargs)
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/seahub/api2/base.py”, line 23, in dispatch
response = super(APIView, self).dispatch(*a, **kw)
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/thirdpart/djangorestframework-3.3.2-py2.7.egg/rest_framework/views.py”, line 466, in dispatch
response = self.handle_exception(exc)
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/seahub/api2/base.py”, line 20, in handle_exception
return super(APIView, self).handle_exception(exc)
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/thirdpart/djangorestframework-3.3.2-py2.7.egg/rest_framework/views.py”, line 454, in dispatch
self.initial(request, *args, **kwargs)
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/thirdpart/djangorestframework-3.3.2-py2.7.egg/rest_framework/views.py”, line 378, in initial
self.check_throttles(request)
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/thirdpart/djangorestframework-3.3.2-py2.7.egg/rest_framework/views.py”, line 340, in check_throttles
if not throttle.allow_request(request, self):
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/seahub/api2/throttling.py”, line 138, in allow_request
return self.throttle_success()
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/seahub/api2/throttling.py”, line 146, in throttle_success
self.cache.set(self.key, self.history, self.duration)
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/thirdpart/Django-1.8.18-py2.7.egg/django/core/cache/backends/filebased.py”, line 56, in set
expiry = self.get_backend_timeout(timeout)
File “/home/seafile/haiwen/seafile-server-6.2.5/seahub/thirdpart/Django-1.8.18-py2.7.egg/django/core/cache/backends/base.py”, line 84, in get_backend_timeout
if timeout == DEFAULT_TIMEOUT:
IndexError: tuple index out of range

seafile.log contains only this logs but this seems to be only a information message (see Get message "Repo size compute queue size is 0" in seafile.log after update from 6.2.2 to 6.2.3 (every 5 minutes) - #9 by marcusm):

[04/08/18 11:15:16] size-sched.c(96): Repo size compute queue size is 0

No errors at controller.log, ccnet.log and seahub.log.

There is only one cronjob for automatic lets-encrypt certificate update:

@monthly ( date ; /etc/init.d/nginx stop && /opt/letsencrypt/letsencrypt-auto renew && /etc/init.d/nginx start ) >> /var/log/letsencrypt-renew.log 2>&1

I also have configured that my system is automatically updated but there were no logs when these updates happend until now. I configured it as in UnattendedUpgrades - Debian Wiki described.

Nginx-configuration (should be quite the same as the manuals at https://manual.seafile.com/deploy/https_with_nginx.html suggest) but added support for http2 as @wthess
suggested:

server {
listen 80;
server_name myserver.net;
rewrite ^ https://$http_host$request_uri? permanent; # force redirect http to https
}
server {
listen 443 ssl http2;
ssl_certificate /etc/letsencrypt/live/myserver.net/fullchain.pem; # path to your cacert.pem
ssl_certificate_key /etc/letsencrypt/live/myserver.net/privkey.pem; # path to your privkey.pem
server_name myserver.net;
proxy_set_header X-Forwarded-For $remote_addr;

add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
server_tokens off;

location / {
    proxy_pass         http://127.0.0.1:8000;
    proxy_set_header   Host $host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Host $server_name;
    proxy_set_header   X-Forwarded-Proto https;
    proxy_read_timeout  1200s;
    proxy_http_version 1.1;     # if you use http2 or you get errors in nginx like connection refused ... HTTP/1.1

    # used for view/edit office file via Office Online Server
    client_max_body_size 0;

    access_log      /var/log/nginx/seahub.access.log;
    error_log       /var/log/nginx/seahub.error.log;
}
location /seafhttp {
    rewrite ^/seafhttp(.*)$ $1 break;
    proxy_pass http://127.0.0.1:8082;
    client_max_body_size 0;
    proxy_connect_timeout  36000s;
    proxy_read_timeout  36000s;
    proxy_send_timeout  36000s;
    send_timeout  36000s;
    proxy_request_buffering off;
    proxy_http_version 1.1;
}
location /media {
    root /home/seafile/haiwen/seafile-server-latest/seahub;
}

}

2

Have you tried using http2? Also, what is your reboot policy? Do you have a cron job that automatically reboots your server on a regular basis?

Also, can you post your server logs for the time period this last happened?

1 Like
3

I don’t know what the issue could be, but It will be enough to execute sudo nginx -s reload. If this doesn’t helps you can execute sudo systemctl restart nginx.service. Bút first try option 1.

2 Likes
4

Thanks for your reply.

No, I have not tried using http2. I will try it from now on and watch out if the problem occurs again.

For the other information you requested I will provide you on the weekend (I am only at my home town, where the server is, on weekends).

5

Thanks for your reply.

The next time when my problem occurs I will try to fix it with the two commands you suggested.

6

I use now http2 and I edited now my original question with some addtional log files. If you need any other please inform me which one exactly would be interesting.

7

I dug a little deeper… I think your problem is with django… You have an odd error, which is usually due to a programming error… So, I’m baffled as to why it works for a while and then stops working. Unfortunately, I have limited knowledge of django.

Here is the error that I think is odd:

IndexError: tuple index out of range

I’m not certain who here is more knowledgeable about django than I am, but they need to take a look at it. Since you have no errors in your Seafile logs, the error lies elsewhere, and it does not appear to be a problem with NGinx.

1 Like
8

Hi, I have exactly the same problem (random bad gateway error, solved rebooting machine),
on a server based on Ubuntu 16.04.04

Seems to be more frequent after upgrade to 6.3.2, with 6.2.5 it was happening just once a month, now it’s every 1/2 days.

It’s strange that Windows client and WebDav still continue to sync files, I just cannot access from web and Windows client doesn’t show libraries.

9

That’s why SeaFile API is working. I would try to restart every SeaFile process and then execute nginx -s reload.