Corrupt install with no issues in fsck - actually an nginx config issue (solved)

Seafile Server
1

Hi !

Context:
I’ve used seafile for some time on my low-specs home setup: raspberry pi 4 running docker (and the franchetti container for seafile) with an external 2.5" hard drive attached. I had let the default settings of infinite history, and realised recently that one consequence was the hypergrowth of my seafile-data folder, which had ended up taking 380 GB of disk space to store 26GB of data. In fact, just running ‘du -sh seafile-date’ took several hours to return that number. I also found out that my nightly backup had not run successfully for a couple of months (shame on me for not finding out before!), so I took actions to review the entire situation. seafile-fsck did report a few issues it could not fix.

Recent actions:

  • I’ve run an fsck on the ext4 partition used by docker to run seafile; it had a few errors that were fixed
  • I’ve reduced the history to more reasonable values
  • I’ve run seaf-gc.sh to clear old stuff; seafile-data is now “only” 58GB instead of the 380GB.
  • seaf-fsck.sh no longer finds issues in any library
  • I’ve copied the seafile folder and used the copy to replace the original – after reading somewhere that over time, too many small files could cause such fragmentation on ext4 that the traversal of the folder would take much longer, and that a copy would solve this; indeed this specific operation took me from 2h30 to run ‘du -sh’ to 2min30s, on the same 58GB of data. Still rather slow to be honest, but 60 times better.
  • just to be safe, I ran fsck (the filesystem one) again (nothing like taking my server down for 20 hours!); as I expected, no errors (I do not suspect imminent disk failure)

Issue now:
After all this, seafile was running fine, at full speed, and I was now able to back up properly again. Looks like all good… until I tried uploading new files via the web interface. The upload popup shows, and then remains stuck at 0%, with nothing more happening. Intrigued, I tried downloading/previewing a few files from my library. A few can be seen without problem, but others cannot: the preview screen tries to load, but gets stuck (from the Android app, it immediately says “download failed”, without delay). This seems to affect only one library, which is NOT the one that grew crazy big over time and had seaf-fsck issues before the actions I took.
On the logs side, seafile.log shows nothing special, while seahub.log shows some Python errors related to broken pipes. If worth exploring I can post them. Rerunning seaf-fsck does not show any issue, and as we’ve seen the filesystem seems healthy enough now.

I’m considering recreating the library from a recently synchronised device (glad I used a seafile client and not a seadrive one!), but I’m wondering what has happened, and if it’s recoverable. What do you think might be the cause of those issues?

Thanks in advance!

2

My apologies, I was completely misled here. The issue seems to be coming from nginx, the setup of which I slightly amended as well recently, to use the “set variable” pattern in the proxy_pass command. It turns out that doing:

rewrite [...]
proxy_pass http://seafile:8082;

works
but

rewrite [...]
set $upstream_app seafile;
set $upstream_port 8082;
set $upstream_proto http;
proxy_pass $upstream_proto://$upstream_app:$upstream_port;

does not, instead it tries to open “://:” , that is the variables are all empty. Strangely this works on other containers and on the root location in seafile, but this happens specifically for the seafhttp location.

I will keep searching, if anyone has hints in the meantime, don’t hesitate…
(the reason I need this pattern is because without it, after restarts nginx sometimes looks for seafile in the wrong docker IP, as documented in the swag documentation and confirmed by experience)

3

This was solved by changing to

# [...]

server {

# [...]

# 3 next lines added now
    set $upstream_app seafile;
    set $upstream_seafhttp_port 8082;
    set $upstream_proto http;

    location / {
# 3 next lines were already there and working for this location without the lines from above
        set $upstream_app seafile;
        set $upstream_port 8000;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;
        # [...]
    }

    location /seafhttp {
        rewrite ^/seafhttp(.*)$ $1 break;
# next line uses variables set outside location, because setting them inside this location did not work (variables were empty for proxy_pass)
        proxy_pass $upstream_proto://$upstream_app:$upstream_seafhttp_port;
        # [...]

I don’t understand… if anyone can explain I’ll be happy, although admittedly, this is really purely an nginx question now (sorry about this!)