Single storage backend separation

I have a question about the backends available for single storage backend. I use s3 compatible backend which works great but the performance is not as good as local storage. So, I decided to separate the fs and commit backend on the local filesystem and leave the blocks storage on the s3 server. Migration went well. It works great so far, however I am not sure if that will continue in the future since I didn’t see this use case documented anywhere. Is splitting the single backend among different storage location advisable or am I going to run into issues in the future?

1 Like

Hey goodnewz,

I don’t see any reasons why this will not work in the future. Some weeks ago Seafile Ltd. recommended this approach to me for a customer with an server that has SSD and HDD-Drives.

Best regards
Christoph

1 Like

Great to hear Christoph. Were you able to specify a location outside the seafile main installation directory for the different portions of the backend? I tried the following in seafile.conf but it didn’t work:

[commit_object_backend]
name=fs
dir=/mnt/seafile/storage

It defaulted to the storage directory inside of the seafile-data directory at the installation location.

Hi goodnewz,

I have no experience with your settings. What I already did is the following:

  1. I used a symlink to put the data-blocks to another device.
    with a symlink seafile finds everything in the folder seafile-data. The seafile folder strukture keeps “normal” but the seafile blocks are stored on another device and only linked with a symlink.

  2. use multiple backends configuration
    create a json file with the description of the desired backend. Here is the explanation.
    https://manual.seafile.com/deploy_pro/multiple_storage_backends.html

Best regards
Christoph

Hi,

it’s definitely possible to specify directories outside the seafile installation directory. In our cluster Seafilein installed in /opt/seafile and the backends are mounted below /fsapp.
Perhaps my storage configuration can serve as an example (there are 10 more backends defined between ›Seafile 001‹ and ›Seafile 012‹ :

[
{
"storage_id": "nfs_001",
"name": "Seafile 001",
"is_default": false,
"fs":      {"backend": "fs", "dir": "/fsapp/seafile_storage/001/seafile.rlp.net"},
"commits": {"backend": "fs", "dir": "/fsapp/seafile_storage/001/seafile.rlp.net"},
"blocks":  {"backend": "fs", "dir": "/fsapp/seafile_storage/001/seafile.rlp.net"}
}
,
}
"storage_id": "nfs_012",
"name": "Seafile 012",
"is_default": true,
"fs":      {"backend": "fs", "dir": "/fsapp/seafile_storage/012/seafile.rlp.net"},
"commits": {"backend": "fs", "dir": "/fsapp/seafile_storage/012/seafile.rlp.net"},
"blocks":  {"backend": "fs", "dir": "/fsapp/seafile_storage/012/seafile.rlp.net"}
}
]

Best regards

Thomas

Hey goodnewz,

I think I was not very precise in my last answer. It is not possible to “just symlink” your blocks-folder to another device. This will result in an error like that:

[08/02/2019 11:29:57 AM] repo-op.c(1112): failed to index blocks[08/02/2019 11:30:19 AM] ../common/block-backend-fs.c(356): [block bend] failed to commit block a7f2a730-75fa-4b97-8165-969cb97b4e06:0fcf6475f0e89829032e2c6c40087471ddb1a785: Invalid cross-device link
[08/02/2019 11:30:19 AM] ../common/fs-mgr.c(559): failed to commit chunk

The blocks-folder has to be on the same device like configure in ccnet/seafile.ini.
BUT it is possible to move the complete seafile-data folder to another device and then add symlinks back to the origin device.

Here is my complete example:

  1. /dev/sda (very fast SSD)
  2. /dev/sdb (slow but big HDD) is mountet on /mnt/storage

In ccnet/seafile.ini you add /mnt/storage/seafile-data as the new folder. But inside /mnt/storage/seafile-data you can symlink the fs and commit folder that are stored on /dev/sda.

Best regards
Christoph

1 Like

Thank you Christoph and Thomas. Very helpful. I feel a bit embarrassed, of course symbolic links! I don’t know why I didn’t think of that.