[SOLVED] Duplicate uploads in S3 storage types - Server Version: 6.2.12

Seafile Server
,
1

I have a Seafile server running Ubuntu 18.04 and Server Version: 6.2.12. The storage is Minio, S3.

First issue (double files):
When uploading small files it’s fine. However large files, say over 600MB are uploaded twice. Using the web client, I can see the folder /opt/seafile/seafile-data/httptemp space increase as the file is uploaded. The file then transfers to /opt/seafile/seafile-data/storage as it should. But then, the web interface just stays on “saving”, and it then re-uploads the file itself, without any interaction. The result is a file renamed with a “(1)” in the file name. Both the original and the “(1)” are present in the library.

Sccond issue (Seadrive reports errors):
Seadrive for Linux has the below errors, but it still says finished and it uploads fine. When uploading with Linux Drive client there are some errors in the logs while testing “ubuntu-budgie-18.04-beta2-desktop-amd64.iso” to upload.

logs/index.log

05/13/2018 03:13:04 [ERROR] seafes:119 thread_task: Index Repo Error: 36585256-49fe-441a-b072-684055866957
Traceback (most recent call last):
  File "/opt/seafile/seafile-pro-server-6.2.12/pro/python/seafes/index_local.py", line 105, in thread_task
    self.fileindexupdater.update_repo(repo_id, commit_id)
  File "/opt/seafile/seafile-pro-server-6.2.12/pro/python/seafes/file_index_updater.py", line 67, in update_repo
    self.check_recovery(repo_id)
  File "/opt/seafile/seafile-pro-server-6.2.12/pro/python/seafes/file_index_updater.py", line 63, in check_recovery
    self.update_files_index(repo_id, old, new)
  File "/opt/seafile/seafile-pro-server-6.2.12/pro/python/seafes/file_index_updater.py", line 33, in update_files_index
    new_commit = commit_mgr.load_commit(repo_id, 0, new_commit_id)
  File "/opt/seafile/seafile-pro-server-6.2.12/seahub/thirdpart/seafobj/commits.py", line 31, in load_commit
    data = self.obj_store.read_obj(repo_id, version, obj_id)
  File "/opt/seafile/seafile-pro-server-6.2.12/seahub/thirdpart/seafobj/backends/base.py", line 12, in read_obj
    data = self.read_obj_raw(repo_id, version, obj_id)
  File "/opt/seafile/seafile-pro-server-6.2.12/seahub/thirdpart/seafobj/backends/s3.py", line 66, in read_obj_raw
    data = self.s3_client.read_object_content(real_obj_id)
  File "/opt/seafile/seafile-pro-server-6.2.12/seahub/thirdpart/seafobj/backends/s3.py", line 52, in read_object_content
    self.do_connect()
  File "/opt/seafile/seafile-pro-server-6.2.12/seahub/thirdpart/seafobj/backends/s3.py", line 48, in do_connect
    self.bucket = self.conn.get_bucket(self.conf.bucket_name)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 509, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 542, in head_bucket
    raise err
S3ResponseError: S3ResponseError: 403 Forbidden

seadrive.log

[05/13/18 02:43:06] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'fs') --> ('finished', 'finished')
[05/13/18 02:43:06] sync-mgr.c(561): Repo 'Minio' sync state transition from 'downloading' to 'load repo'.
[05/13/18 02:43:06] sync-mgr.c(561): Repo 'Minio' sync state transition from 'load repo' to 'synchronized'.
[05/13/18 02:44:57] sync-mgr.c(561): Repo 'Minio' sync state transition from 'synchronized' to 'committing'.
[05/13/18 02:44:57] sync-mgr.c(2635): All operations of repo Minio(36585256) have been processed.
[05/13/18 02:44:57] sync-mgr.c(561): Repo 'Minio' sync state transition from 'committing' to 'uploading'.
[05/13/18 02:44:57] http-tx-mgr.c(3878): Upload with HTTP sync protocol version 2.
[05/13/18 02:44:57] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'init') --> ('normal', 'check')
[05/13/18 02:44:57] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'check') --> ('normal', 'commit')
[05/13/18 02:44:57] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'commit') --> ('normal', 'fs')
[05/13/18 02:44:57] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'fs') --> ('normal', 'data')
[05/13/18 02:44:57] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'data') --> ('normal', 'update-branch')
[05/13/18 02:44:58] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'update-branch') --> ('finished', 'finished')
[05/13/18 02:44:58] sync-mgr.c(2261): removing blocks for repo 36585256-49fe-441a-b072-684055866957
[05/13/18 02:44:58] sync-mgr.c(561): Repo 'Minio' sync state transition from 'uploading' to 'get sync info'.
[05/13/18 02:44:58] sync-mgr.c(561): Repo 'Minio' sync state transition from 'get sync info' to 'synchronized'.
[05/13/18 02:46:35] sync-mgr.c(561): Repo 'Minio' sync state transition from 'synchronized' to 'committing'.
[05/13/18 02:46:44] sync-mgr.c(2556): Creating partial commit after adding ubuntu-budgie-18.04-beta2-desktop-amd64.iso in repo Minio(36585256).
[05/13/18 02:46:45] sync-mgr.c(561): Repo 'Minio' sync state transition from 'committing' to 'uploading'.
[05/13/18 02:46:45] http-tx-mgr.c(3878): Upload with HTTP sync protocol version 2.
[05/13/18 02:46:45] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'init') --> ('normal', 'check')
[05/13/18 02:46:45] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'check') --> ('normal', 'commit')
[05/13/18 02:46:45] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'commit') --> ('normal', 'fs')
[05/13/18 02:46:46] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'fs') --> ('normal', 'data')
[05/13/18 03:12:50] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'data') --> ('normal', 'update-branch')
[05/13/18 03:12:50] http-tx-mgr.c(3752): Bad response code for PUT https://<server>/seafhttp/repo/36585256-49fe-441a-b072-684055866957/commit/HEAD/?head=f410beb05edf4a28898e00b8b0f450a7fc0fba99: 500.
[05/13/18 03:12:50] http-tx-mgr.c(3970): Failed to update branch of repo 36585256.
[05/13/18 03:12:50] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'update-branch') --> ('error', 'finished')
[05/13/18 03:12:50] sync-mgr.c(595): Repo 'Minio' sync state transition from uploading to 'error': 'Error occured in upload.'.
[05/13/18 03:12:51] sync-mgr.c(561): Repo 'Minio' sync state transition from 'synchronized' to 'uploading'.
[05/13/18 03:12:51] http-tx-mgr.c(3878): Upload with HTTP sync protocol version 2.
[05/13/18 03:12:51] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'init') --> ('normal', 'check')
[05/13/18 03:12:51] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'check') --> ('normal', 'commit')
[05/13/18 03:12:51] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'commit') --> ('normal', 'fs')
[05/13/18 03:12:51] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'fs') --> ('normal', 'data')
[05/13/18 03:12:51] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'data') --> ('normal', 'update-branch')
[05/13/18 03:12:51] http-tx-mgr.c(1145): Transfer repo '36585256': ('normal', 'update-branch') --> ('finished', 'finished')
[05/13/18 03:12:51] sync-mgr.c(2261): removing blocks for repo 36585256-49fe-441a-b072-684055866957
[05/13/18 03:12:51] sync-mgr.c(561): Repo 'Minio' sync state transition from 'uploading' to 'get sync info'.
[05/13/18 03:12:51] sync-mgr.c(561): Repo 'Minio' sync state transition from 'get sync info' to 'synchronized'.
[05/13/18 03:12:52] sync-mgr.c(561): Repo 'Minio' sync state transition from 'synchronized' to 'committing'.
[05/13/18 03:12:52] sync-mgr.c(2492): All operations of repo Minio(36585256) have been processed.
[05/13/18 03:12:52] sync-mgr.c(561): Repo 'Minio' sync state transition from 'committing' to 'synchronized'.

Any ideas why there are file duplicates with the web UI, and then Bad response code for PUT with Seadrive?

2

In answer to the second question, others have reported problems with the Seadrive client on 18.04. I don’t remember the issues, but I wanted to let you know that there are issues with the new 18.04.

3

Ok, so the errors in index.log are irrelevant to the issue. Disabling the indexing removes those errors. Possibly indexing is not supported on Minio.

I have re-installed Server Version: 6.2.12 on a Ubuntu 16.04 server and still have the same issues uploading with the Web UI, Cloud browser app (same upload method) and Seadrive on 17.10. After the first upload it starts again it later renames the file with a (1) on the end, ending up with 2 files for one upload. The file was around 580MB.

None of the error logs indicate any activity when it starts to upload the second file, so I can’t look for the error at the time it starts the second upload. Looking under ~/.ccnet/logs. Unless I am looking in the wrong place

In saying that, the sync function via the Linux desktop app works fine. The files goes right into Minio and does not double upload.

4

I think seafile.log is where those are recorded. Probably not recorded in controller.log, but it never hurts to check. Can you reproduce the problem using a different browser for the upload? Also, is this affecting multiple PCs or just one?

5

You should use a container. They were invented to prevent such issues. Ok, for Pro you have to build/commit it on your own bu that’s not that worse.

6

I just tested with a trial in Wasabi’s S3 storage. The same issue is there.

Uploading via any browser (chrome or firefox), the android client, linux client’s cloud browser from the desktop app and seadrive. On both a Linux and Windows computer. Anything that’s using the /opt/seafile/seafile-data/httptemp folder has the same issues. After the first upload it stays at “uploading” and then uploads a second time. I can see /opt/seafile/seafile-data/httptemp upload the same data size. It then just renames the file with a (1) on the end.

I believe my config files are in order:

nginx

server {
        listen       80;
        server_name  seafile-test.my-domain.com;
        rewrite ^ https://$http_host$request_uri? permanent;    # force redirect http to https
        server_tokens off;
    }
    server {
	listen 443 http2;
	listen [::]:443 http2;
        ssl on;
        ssl_certificate /etc/letsencrypt/live/seafile-test.my-domain.com/fullchain.pem;        # path to your fullchain.pem
        ssl_certificate_key /etc/letsencrypt/live/seafile-test.my-domain.com/privkey.pem;    # path to your privkey.pem
        server_name seafile-test.my-domain.com;
        ssl_session_timeout 5m;
        ssl_session_cache shared:SSL:5m;

        # Diffie-Hellman parameter for DHE ciphersuites, recommended 2048 bits
        ssl_dhparam /etc/nginx/dhparam.pem;

        # secure settings (A+ at SSL Labs ssltest at time of writing)
        # see https://wiki.mozilla.org/Security/Server_Side_TLS#Nginx
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:HIGH:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS';
        ssl_prefer_server_ciphers on;

        proxy_set_header X-Forwarded-For $remote_addr;

	    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;        
	    server_tokens off;

        location / {
            proxy_pass         http://127.0.0.1:8000;
            proxy_set_header   Host $host;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Host $server_name;
            proxy_set_header   X-Forwarded-Proto https;

            access_log      /var/log/nginx/seahub.access.log;
            error_log       /var/log/nginx/seahub.error.log;

            proxy_read_timeout  1200s;

            client_max_body_size 0;
        }

        location /seafhttp {
            rewrite ^/seafhttp(.*)$ $1 break;
            proxy_pass http://127.0.0.1:8082;
            client_max_body_size 0;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_connect_timeout  36000s;
            proxy_read_timeout  36000s;
            proxy_send_timeout  36000s;
            send_timeout  36000s;
	    proxy_request_buffering off;
        }
        location /media {
            root /opt/seafile/seafile-server-latest/seahub;
        }
    }

ccnet.config

[General]
USER_NAME = seafile-test
ID = <ID>
NAME = seafile-test
SERVICE_URL = https://seafile-test.my-domain.com

[Client]
PORT = 13419

[Database]
ENGINE = mysql
HOST = 127.0.0.1
PORT = 3306
USER = seafile
PASSWD = <password>
DB = ccnet-db
CONNECTION_CHARSET = utf8

seahub_settings.py

# -*- coding: utf-8 -*-
SECRET_KEY = "<key>"
FILE_SERVER_ROOT = 'https://seafile-test.my-domain.com/seafhttp'
SESSION_COOKIE_AGE = 86400
ENABLE_THUMBNAIL = True
THUMBNAIL_IMAGE_SIZE_LIMIT = 10 # MB
ENABLE_VIDEO_THUMBNAIL = False
TIME_ZONE = 'Europe/<city>'
SITE_NAME = 'Seafile'
SITE_TITLE = 'Seafile'

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'seahub-db',
        'USER': 'seafile',
        'PASSWORD': '<password>',
        'HOST': '127.0.0.1',
        'PORT': '3306'
    }
}

CACHES = {
    'default': {
        'BACKEND': 'django_pylibmc.memcached.PyLibMCCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

seafile.conf

[fileserver]
port = 8082
max_upload_size = 5000
max_indexing_threads = 10
fixed_block_size = 2

[database]
type = mysql
host = 127.0.0.1
port = 3306
user = seafile
password = <password>
db_name = seafile-db
connection_charset = utf8

[history]
keep_days = 15

[commit_object_backend]
name = s3
bucket = my-commit-objects
key_id = <key-id>
key = <key>
host = s3.wasabisys.com
path_style_request = true
memcached_options = --SERVER=localhost --POOL-MIN=10 --POOL-MAX=100
use_https = true

[fs_object_backend]
name = s3
bucket = my-fs-objects
key_id = <key-id>
key = <key>
host = s3.wasabisys.com
path_style_request = true
memcached_options = --SERVER=localhost --POOL-MIN=10 --POOL-MAX=100
use_https = true

[block_backend]
name = s3
bucket = my-block-objects
key_id = <key-id>
key = <key>
host = s3.wasabisys.com
path_style_request = true
memcached_options = --SERVER=localhost --POOL-MIN=10 --POOL-MAX=100
use_https = true

Can someone else try to re-produce this with a Wasabi trial?

7

I am using LXC

8

Would it be possible for you to try logging into the web interface locally and trying it, ie bypassing NGinx? Such as http://127.0.0.1:8000 from the server, or even from a machine internally using the IP of the server?

It will, at least, help us narrow it down some.

9

That’s a good idea.

It’s actually sitting on a VPS with LXD. The lxd host forwards 80 and 443 to haproxy and haproxy forwards the seafile subdomain via the sni request to the lxc contaner. I can try to forward 80 and 443 to the lxc container directly, and even try to take out nginx and use 8000. I’ll do some testing tonight and report back.

10

Can you check what the Webserver logs when the put requests fails?

Have you tried increasing timeouts? It much looks like the request times but Seafile finishes indexing and the browser restarts the upload because the request failed.

1 Like
11

And for this I’d look at the minio log to see what’s going wrong.

12

Yes i will check those logs tonight

13

Will check this too

14

I bypassed haproxy and just forwarded 80 and 443 with iptables. And the http uploads worked as normal. No duplicates and the files transferred to Mino with no issues. So it must be the config in haproxy. I’ll debug this tomorrow.

1 Like
15

I suggest to have a look at the timeouts.

16

@philthynz
Good work! So, we’ve narrowed it down to the reverse proxy somehow.

@shoeper
That’s the direction I’m also leaning. Since it’s happening with large files, I’m thinking that the proxy is timing out.

@philthynz
How long does it generally take to upload one of those large files?

17

@wthess I usually get a transfer speed of around 1.3MB/s from home. The upload to Minio is fast, 10MB/s.
@shoeper I guess you want to check the haproxy logs? Seafile has no indication of anything when the duplicate starts to upload.

Getting timeout logs is going to be hard, if not impossible with the way HAProxy is configured. I am using mode tcp to forward 443. So i don’t have to deal with certificates. According to the log manual here the tcp log format will only log when the connection terminates.

“HAProxy will then usually wait for the connection to terminate before logging.”

Here is my HAProxy config, which works fine for everything so far, as a tcp proxy. It only became an issue when i wanted to plug in Minio to seafile.

global
#    log /dev/log    local0 
#    log /dev/log    local1 notice
    log /dev/log    local0 debug
    log /dev/log    local1 debug
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    timeout connect 15s
    timeout client  15s
    timeout server  15s
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

## redirect port 80 to 443
frontend localhost80
    bind *:80
    mode http
    redirect scheme https code 301 if !{ ssl_fc }

## bind 443
frontend localhost443
    bind *:443
    option tcplog
    mode tcp
    acl tls req.ssl_hello_type 1
    tcp-request inspect-delay 5s
    tcp-request content accept if tls

## bind 443 to sni names
    acl is_seafile-test req.ssl_sni -i seafile-test.mydomain.com

## which backend to use for requests
    use_backend seafile-test if is_seafile-test

## backend lxc containers  
backend seafile-test
    mode tcp
    server is_seafile-test seafile-test.lxd:443 check

I added debug to global which gives me errors like:

May 16 16:39:37 haproxy haproxy[8764]: <myip>:42282 [16/May/2018:16:39:22.158] localhost443 seafile-test/is_seafile-test 1/0/15150 3768 cD 1/1/0/0/0 0/0
May 16 16:39:37 haproxy haproxy[8764]: <myip>:42282 [16/May/2018:16:39:22.158] localhost443 seafile-test/is_seafile-test 1/0/15150 3768 cD 1/1/0/0/0 0/0
May 16 16:42:37 haproxy haproxy[8764]: <myip>:43678 [16/May/2018:16:42:22.174] localhost443 seafile-test/is_seafile-test 2/0/15112 486 cD 1/1/0/0/0 0/0

I am glad we are getting further on this. It would be nice to have a lxd with haproxy config that works with backend storage. We can share that with others, i haven’t seen any examples on the net so far.

18

I could try with option http and setup the certs. I was hoping pure tcp proxy would do it. I don’t see why not.

19

I’d first try to increase timeout client and timeout server to an hour.

1 Like
20

Will try that and report back