Multiple seafile-data folders?

It appears Seafile Server supports only one seafile-data folder. It would be wonderful to add support for many seafile-data folders. And the admin should be able to specify which libraries or groups or users are saved in which data folder. This is needed to be able to mount different data folders for different groups and to backup the data folders differently.

We have plan to support more than one storage folders in the future. But the purpose is only to let admins increase storage easily, not to map libraries to storage in a more complex way. Mapping libraries into multiple storage folders would introduce more database accesses. This will slow down Seafile’s syncing.

But your idea is interesting. We’ll keep it in mind.

6 Likes

Multiple backends would require storing it in the database on per library base anyway, wouldn’t it? At least that would be what I’d expect to be the solution with the best performance. Only down I could think of is that once one backend is almost full some libraries would need to be moved to another backend. New libraries could be stored in the default backend (allow setting multiple default backends could also be useful for large setups). Another option would be a flag per backend if library creation is allowed or not and calculate how much libraries we have per backend and choose the one with most GB (total space) per library.

This feature request could be implement on top by adding something like “library policies”, where the policy defines where a library should be stored.

2 Likes

want to know whether this feature added to new 7.0.x version…i also want this feature

Multiple storage backend support was added in Seafile 6.3. Check this out: https://blogs.seafile.com/2018/07/30/seafile-professional-server-6-3-now-production-ready/

It’s a Seafile PE feature though.

Thanks your help…
but mutiple storage backend is different to the requirement, I think.
Actually the author of this topic and myself just want to store the file in local drives. Let’s say in different volumes(/dev/sdb,sdbc) or disks(D drive and E drive). and different users store the fs, commit, blocks file in different seafile-data folder.
No need to integrate with other cloud based storage.

I think this feature is required by most of the small company if they have more than 2 teams, eg, accounting and IT teams

I am using mutiple seafile-data folders which are mounted to different linux disks. However seems only working for what put in side ccnet/seafile.int
or maybe my conf is wrong… hahahaha:rofl:

[
{
“storage_id”: “seafile-data”,
“name”: “seafile-data”,
“is_default”: true,
“for_new_library”: false,
“fs”: {“backend”: “fs”, “dir”: “/seafile/seafile-data”},
“commits”: {“backend”: “fs”, “dir”: “/seafile/seafile-data”},
“blocks”: {“backend”: “fs”, “dir”: “/seafile/seafile-data”}
},

{
“storage_id”: “new_storage”,
“name”: “new_storage”,
“for_new_library”: true,
“is_default”: false,
“fs”: {“backend”: “fs”, “dir”: “/storage/seafile/newseafile-data”},
“commits”: {“backend”: “fs”, “dir”: “/storage/seafile/newseafile-data”},
“blocks”: {“backend”: “fs”, “dir”: “/storage/seafile/newseaflle-data”}
}

]

Do you use Seafile PE or CE? Support for multiple storage backends is a PE feature.

The feature does not care if you use two filesystem-based backends or S3 + Swift or whatever.

I had a quick look at your config. Looks fine to me, but I am not a multi-backend expert. There is one thing though that you might want to change:

seaflle should be seafile.

Do these folders exist?

thanks your help.haha yes, i think that is my typO :stuck_out_tongue: error…but i manually create those folder before I changed seafile.ini content.

I am reinstalling pro version 7.0.5 now and see whether it works…soon update here…heheh
I am curious whether it works as search a lot but no any final solution yet.

I tested again…still failed.
I read both Chinese and English github manual again for this multiple storage backend setup and reinstall new seafile pro 7.0.5 version.
I followed exactly the “user chosen” option:

User Chosen
This policy lets the users to choose which storage class to use when creating a new library. The users can select any storage class that’s been defined in the JSON file.

To use this policy, add following options in seahub_settings.py:

STORAGE_CLASS_MAPPING_POLICY = ‘USER_SELECT’

If you enable storage class support but don’t explicitely set STORAGE_CLASS_MAPPING_POLIICY in seahub_settings.py, this policy is used by default.

and Here is the storage class I defined even though all local file system??? here is my doubt, if I use different seafile-data folder will it support in this mutiple backend storage portion?

[root@app01 conf]# cat seafile_storage_classes.json
[
{
“storage_id”: “seafile-data”,
“name”: “seafile-data”,
“is_default”: true,
“fs”: {“backend”: “fs”, “dir”: “/seafile/seafile-data”},
“commits”: {“backend”: “fs”, “dir”: “/seafile/seafile-data”},
“blocks”: {“backend”: “fs”, “dir”: “/seafile/seafile-data”}
},
{
“storage_id”: “cold_storage”,
“name”: “cold_storage”,
“is_default”: false,
“fs”: {“backend”: “fs”, “dir”: “/storage/seafile/cold_seafile-data”},
“commits”: {“backend”: “fs”, “dir”: “/storage/seafile/cold_seafile-data”},
“blocks”: {“backend”: “fs”, “dir”: “/storage/seafile/cold_seafile-data”}
},
{
“storage_id”: “new_storage”,
“name”: “new_storage”,
“is_default”: false,
“fs”: {“backend”: “fs”, “dir”: “/storage/seafile/new_seafile-data”},
“commits”: {“backend”: “fs”, “dir”: “/storage/seafile/new_seafile-data”},
“blocks”: {“backend”: “fs”, “dir”: “/storage/seafile/new_seafile-data”}
}
]"

cat seahub_settings.py
here is the seahub_setting.py

}
ENABLE_SYS_ADMIN_VIEW_REPO = True
MULTI_INSTITUTION = True
EXTRA_MIDDLEWARE_CLASSES += (
‘seahub.institutions.middleware.InstitutionMiddleware’,
)
ENABLE_STORAGE_CLASSES = True
STORAGE_CLASS_MAPPING_POLICY = ‘USER_SELECT’
#STORAGE_CLASS_MAPPING_POLICY = ‘REPO_ID_MAPPING’
#STORAGE_CLASS_MAPPING_POLICY = ‘ROLE_BASED’
ENABLED_ROLE_PERMISSIONS = {
‘default’: {
‘can_add_repo’: True,
‘can_add_group’: True,
‘can_view_org’: True,
‘can_use_global_address_book’: True,
‘can_generate_share_link’: True,
‘can_generate_upload_link’: True,
‘can_invite_guest’: True,
‘can_connect_with_android_clients’: True,
‘can_connect_with_ios_clients’: True,
‘can_connect_with_desktop_clients’: True,
‘storage_ids’: [ ‘new_storage’, ‘cold_storage’,‘seafile-data’], <---- add the storage_id here
}
}
"
as the “for_new_library”: true, option is only mentioned in libarary ID mapping option so i removed it from my json file.

However I tested, frontend portal when I create new libarary by using new user account, still everything goes to old seafile-data folder. There is no place to “CHOSEN” the storage class.

A quick update here… I am able to make it work by doing some manual work.

i changed policy to repo id mapping
STORAGE_CLASS_MAPPING_POLICY = ‘REPO_ID_MAPPING’
firstly, i create new library from web, and there is a folder created in old seafile-data commit folder.

I manually changed database storage_id for the newly created library. can see the new id is “new_storage” which point to “fs”: {“backend”: “fs”, “dir”: “/storage/seafile/new_seafile-data”},

image
then I copy old commits folder(located at old seafile-data folder,as i didnt upload anything yet,so no folder under “fs” and “bloack”) to this new commit folder(commits": {“backend”: “fs”, “dir”: “/storage/seafile/new_seafile-data”},)
then wola… all working …
I am able to upload files to this new library and at backend all files are stored in new folder. and other libaries no impact.

but it is really not a good way even can make it work.

So my guessing is, need to populate storage_id information before you config the storage class.

If anyone has any finding or can contact the dev team. Please help me to ask for it.

thanks :stuck_out_tongue:

I can confirm that this feature is working as intended now, in version 10 of pro edition (probably earlier). I’m doing a Docker setup, and following the docs exactly does actually work (assuming you don’t mess up a docker volume path string or something like that). And if you do mess that up (like I did) I don’t think it’s always clear what you should/should not do while you’re troubleshooting and fixing it, so I’m planning to open a docs PR soon to help improve it.

For example, if you destroy/recreate any of the containers along the way, you cannot simply copy/paste entire config files from backup and expect it to work after you restart the containers, because there are credentials/keys to the database and stuff in there that get regenerated when you recreate the containers. If you do that, you’ll likely see errors about DB authentication in the logs, and you’ll get stuck on the 502 Bad Gateway screen forever.

This is the test that seems to be working for me:
I added the following config to seafile.conf:

[storage]
enable_storage_classes = true
storage_classes_file = /shared/seafile/conf/seafile_storage_classes.json

[memcached]
memcached_options = --SERVER=memcached --POOL-MIN=10 --POOL-MAX=100

Then defined this storage class config file just for testing (I’m going to replace the backend dirs with the system mount points of multiple external disks now that I have observed this working properly):

[
  {
    "storage_id": "vdisk1",
    "name": "vDisk 1",
    "is_default": false,
    "fs": {
      "backend": "fs",
      "dir": "/shared/vdisk1"
    },
    "commits": {
      "backend": "fs",
      "dir": "/shared/vdisk1"
    },
    "blocks": {
      "backend": "fs",
      "dir": "/shared/vdisk1"
    }
  },
  {
    "storage_id": "disk2",
    "name": "vDisk 2",
    "is_default": false,
    "fs": {
      "backend": "fs",
      "dir": "/shared/vdisk2"
    },
    "commits": {
      "backend": "fs",
      "dir": "/shared/vdisk2"
    },
    "blocks": {
      "backend": "fs",
      "dir": "/shared/vdisk2"
    }
  }
]

Finally I added this to the bottom of seahub_settings.py for the UI:

ENABLE_STORAGE_CLASSES = True
STORAGE_CLASS_MAPPING_POLICY = 'USER_SELECT'

I recreated the seahub docker container ONLY (running in gunicorn mode) to reload the backend and UI apps: docker compose up -d --force-recreate seafile

I wait about 15sec or so for the container to come up again, and I can see that it has created the vdisk1 and vdisk2 directories on my filesystem.

# tree -L 3
// This is the directory mounted to the seafile container @ /shared
. 
├── vdisk1
│   ├── storage
│   │   ├── blocks
│   │   ├── commits
│   │   └── fs
│   └── tmpfiles
└── vdisk2
    ├── storage
    │   ├── blocks
    │   ├── commits
    │   └── fs
    └── tmpfiles

In the UI, when I create a new library, I get this option now:

Screenshot 2023-12-27 at 2.38.33 AM