Huge FS Folder Slowing Down Backup Speed

Hi, I’ve been having problems with extremely slow backup speed for my SF server and I just noticed my fs folder in seafile-data contains about 2 million files whereas it should be around 70,000 according to the blocks folder. This is dragging my backup speed down to a few dozen kilobytes a second. Is there any way I can prompt seafile to re-generate the sf folder? I’m on Seafile Community

I don’t think it’s possible to say how many fs objects there should be looking at the blocks folder. Maybe some files were updated frequently.

Running the garbage collector could help.

I migrated all files into it in one go freshly, no changed files, all set to not keep a version history. Garbage collector doesn’t do anything, apparently Community Version doesn’t have an FS garbage collector.

I’d just want to know if I can delete the FS folder and if Seafile will re-generate it. Because this is such a weird problem that nobody else seems to have…

No, you cannot delete it.

Then do you have any other suggestions? I don’t know why this happens, it happened again on a fresh install after migrating about 80gb of data through the webdav server. It makes running backups impossible and that’s extremely frustrating

How many folders and files do you have? Do you have recursive symlinks?

Additionally, how are backups done and what kind of disk is being used? HDDs cannot handle much IO and combined with little RAM it’ll be slow. Another issue could be the backup method not being capable of handling many files (such as rsync). Using borg backing up more than 3 millions files just takes 30 minutes on my server (which is actually using HDDs).

And then it can be good to know, that objects are immutable - meaning and object written once is never changed.

It’s about 12,000 files at about 90gb total. No symlinks it’s data from personal projects etc.

I migrated that data into Seafile by mounting a Seafdav server to a folder and copying it into that. I’m suspecting that’s the cause of the problem although I don’t know how to circumvent it.
I end up with about 16,000 blocks and 18,000 commits files, the first time ontop of that nearly 2 million FS files the second time “only” about 150,000. Either way I’ve checked everything, read write tests of the HDD and trying to back up each storage folder individually it’s definitely a problem with the files in the FS folder, it’s the only one that tanks performance.

I tried backing up with rsync, scp and tarring it locally, everything had the same issue. Something there is just fishy as even with completely random reads the performance should be about 50 times what I’m seeing which is about 100-200kb/s and I’ve confirmed it to be the FS folder, why I don’t know, nobody online has this problem and none of my colleagues running much larger seafile deployments.

I’m currently migrating the data into seafile using a more conventional client, maybe the webdav thing just messes it up.


Here is a screenshot from a local tar, no compression going at ridiculously slow speeds.


Here IOTOP report

image
Here an Hdparm disk test


And a fio specific random read test

Nothing here lines up

I found a solution.

Apparently transferring the data through WebDav caused Seafile to generate 100-1000 times more FS files than necessary. This seems like a bug as it was totally fine when syncing through a normal client

3 Likes

@daniel.pan Is this the expected behavior? Does WebDAV create an extremely greater number of fs objects?

In the latest version, you can run seaf-gc.sh --rm-fs to remove old fs objects. Seafile GC - Seafile Admin Manual

It is likely to be caused by the WebDAV clients.

2 Likes

Thank you! Sadly FS garbage collection is only available for Pro servers right? I migrated to pro because I use less than 3 users but still.

Thanks for the help, got it all working now!

It should be available in community edition 9.0 too.

1 Like

Oh great, then the documentation is wrong as it only mentions pro server :wink: