Problems syncing very large library [to the real-time backup]

I have a very large library (3.4 T) which I cannot sync to the real-time backup. I guess I would get the same error when syncing to a client (but I have no client with that much disk space).

If I manually sync the library on the real-time backup I get:

sudo -u seafile /srv/seafile/seafile-server-latest/ sync e612a24e-02f8-44fa-8700-ea2b4e05e7bf
Failed to sync repo e612a24e-02f8-44fa-8700-ea2b4e05e7bf: Failed to get fs ids from primary.

during this (unsuccessful) run I get many errors like:

../common/obj-cache.c(106): Failed to set e612a24e-02f8-44fa-8700-ea2b4e05e7bf-3e582b74e5ff60dcc9a8ee1e1750f75bf9b33f6c to memcached: ITEM TOO BIG.

but supposedly those are not critical. however; I see proxy errors:

[Thu Jan 25 16:35:18.108642 2018] [proxy_http:error] [pid 28741:tid 140578809427712] (70007)The timeout specified has expired: [client] AH01102: error reading status line from remote server
[Thu Jan 25 16:35:18.108726 2018] [proxy:error] [pid 28741:tid 140578809427712] [client] AH00898: Error reading from remote server returned by /seafhttp/server-sync/repo/e612a24e-02f8-44fa-8700-ea2b4e05e7bf/multi-fs-id-list/

It seems to me, that the proxy gets a timeout while waiting for the response for seafile. How to go about this? Should I increase the timeout in the proxy (apache). Or is this an indication of another problem?


I would try to increase the timeouts. Read timeout seems to be most promising.

I’m not sure what the request does, but if it really lists all fs ids that could take very much time for 3,4T. Depending on the kind of files if could include millions of fs objects.

Why not just use zfs/btrfs snapshot replication for now? The active backup feature does not seem to be out of beta yet judging from this many bugs/issues with it.

I include db dumbs (they are small) with every snapshot and have a post snapshot sync hook running a db import on the backup server. This way I can have a snapshot every 5-15 min depending on network and storage speed.

Using a galera master slave setup for the database might even be better but I did not yet implement this on my personal instance.

of course I could do something like this. however; i use ceph, but I could also mirror my ceph pools to an offsite location. but a mirror is never a backup. I admit, in the case of seafile the distinction is not so clear, because seafile keeps a history. nevertheless, users delete libraries, and the history setting my be too short, to be useful as backup. of course, you could tweak all this settings and never run the garbage collector, then you would have a complete history.

On the other hand, not only users can mess up or delete the data. What happens when the sysadmin (during an update, o just by mistake) breaks things in the database or the backend storage? If you use some mirroring technique for an backup such manipulations will instantly be replicated to the backup, and a lot of data might be lost or inaccessible.

Unless you take additional snapshots of the database and the backend storage.

I agree that the real-time backup is not as stable as it could be. But I like that it is to a large degree independent from the seafile cluster, such that catastrophic events on the cluster cannot have a serious impact on the backup.