Gc.sh --rm-fs causes severe GlusterFS degradation on large production repository

Hi,

We are having problems running gc.sh --rm-fs on a large Seafile Pro 11.0.19 production environment.

Environment:

  • Seafile Pro: 11.0.19
  • Storage: GlusterFS
  • Storage used: ~9.8 TB
  • Blocks: ~15.1 million

The normal GC completed successfully after approximately 60 hours and removed 25,153 blocks.

However, gc.sh --rm-fs progressively degrades the GlusterFS client.

Using strace, we observed:

  • getdents64 initially: ~20 ms
  • after several hours: ~500 ms
  • later: ~1-1.7 s
  • eventually: ~3-5 s

The GlusterFS servers themselves are not CPU or disk saturated and all four bricks remain online.

The same operation worked correctly in PRE, but that environment contains only approximately 40 GB.

Is there a recommended way to run --rm-fs against a repository of this size, for example by repository, in batches, with resumable processing, or with different concurrency settings?

We would also like to know whether newer Seafile Pro versions contain improvements to --rm-fs for very large filesystem repositories.

Thanks.

Hello @Julio ,

Since version 11, GC has not introduced any filesystem performance improvements. At which stage of GC did you start noticing the performance degradation? What are the main messages in the GC logs when the performance drops?

Hi,

The performance degradation starts during the --rm-fs operation, while GC is processing our main repository (Temis).

At that point, the GC log stays at: GC version 1 repo Temis(xxxxxxx-xxxxx-xxxxxx)

There are no additional GC log messages for many hours.

We monitored the seafserv-gc process with strace, and during this stage it continuously performs getdents64() calls, apparently traversing the FS object directories.

The important point is that the latency progressively increases while --rm-fs runs:

  • Initially: ~0.02 seconds per getdents64()
  • After some hours: ~0.5 seconds
  • Later: ~1-1.7 seconds
  • After ~40 hours: ~3-5 seconds

Eventually, the GlusterFS FUSE client becomes almost unresponsive. Simple filesystem operations can take tens of seconds, and Seafile processes end up blocked in uninterruptible I/O wait: D request_wait_answer

The GlusterFS bricks themselves remain online, and CPU/disk utilization on the GlusterFS servers is low. We also observed READDIRP dominating the latency when profiling GlusterFS.

When this happens, stopping seafserv-gc is not enough to immediately restore the service. We had to recover/remount the GlusterFS client and recreate the Seafile container.

This problem is specifically with --rm-fs. Our regular block GC completed successfully, although it took approximately 60 hours.

We also tested --rm-fs successfully in our PRE environment with the same configuration, but PRE contains only around 40 GB of data, while PROD currently has approximately 9.8 TB used.

So it seems that the degradation happens during the FS object traversal phase of --rm-fs, and becomes progressively worse as the traversal continues.

Thanks!

Hello @Julio ,

If the performance does not return to normal after stopping seafserv-gc, it suggests that the issue may be caused by the filesystem itself when GC traverses the FS objects.

Individual FS objects are usually small, but there can be a large number of them. Therefore, your library may contain a large number of FS objects, even though their total disk usage is relatively small.

When GC reads the names of these FS objects from the filesystem, READDIRP may retrieve stat information for every directory entry, potentially introducing additional disk I/O and degrading performance.

You can check the current READDIRP-related configurations. For Seafile, a simple READDIR operation should be sufficient; there is no need to retrieve additional stat information for each directory entry.

Of course, if the FS objects do not consume much disk space, there may be no need to clean them up either.

Hi,

I checked our current GlusterFS configuration and found:

  • performance.force-readdirp = true
  • dht.force-readdirp = on

Based on your explanation about READDIRP retrieving additional stat information, we are planning to change both settings to:

  • performance.force-readdirp = off
  • dht.force-readdirp = off

Our GlusterFS storage consists of 4 nodes, and storage/fs currently consumes approximately 1 TB per node, so we have around 4 TB of FS objects in total.

Before changing this in production, could you please confirm that disabling both performance.force-readdirp and dht.force-readdirp is what you recommend for Seafile --rm-fs?

Thanks.

Hello @Julio ,

disabling READDIRP should be a viable option. Seafile does not need to retrieve additional attributes of dirs or files in the storage directory. Since I don’t have a GlusterFS environment available, I recommend testing this configuration in a test environment before applying it in production to ensure that Seafile continues to work properly.

Hi,

Thanks for the clarification.

We tested disabling READDIRP in our PRE environment and Seafile continued to work correctly. We ran multiple gc.sh --rm-fs tests and did not encounter any functional issues.

The settings tested were:

  • performance.force-readdirp = off
  • dht.force-readdirp = off

From a performance perspective, the results were not conclusive in our PRE environment. The overall GC duration varied between runs and we did not observe a clear or consistent improvement compared to the previous configuration.

However, the important result is that disabling READDIRP did not cause any problems and --rm-fs completed successfully in all tests.

Our PRE environment is relatively small (about 94,000 FS objects), while PROD contains approximately 4 TB of FS objects distributed across 4 GlusterFS nodes, so the impact may only become visible at production scale.

Based on your recommendation, we plan to test disabling READDIRP in PROD during the next maintenance window and monitor whether it helps prevent the progressive slowdown we previously observed during the FS traversal phase.

Thanks for your help.

Hi,

Quick update: we finally seem to have found a configuration that works for us.

Disabling READDIRP on the GlusterFS volume itself was not enough. We still had the same progressive slowdown in PROD.

The change that made the real difference was configuring the Seafile GlusterFS/FUSE client with:

  • --use-readdirp=no
  • --reader-thread-count=4
  • --lru-limit=1000000
  • --invalidate-limit=0

while keeping on gluster config:

  • performance.force-readdirp = off
  • dht.force-readdirp = off

With this setup, gc.sh --rm-fs spent about 54 hours in Populating index on our PROD repository and GlusterFS remained stable. We did not see the progressive degradation/hanging behavior we had before.

There is one important limitation: once GC reaches Scanning and deleting unused blocks, it performs many parallel unlink() operations. At our scale this affects normal uploads and downloads, so we cannot comfortably run that phase during business hours.

We are therefore scheduling the process so that the deletion phase happens over the weekend.

Hopefully this information is useful to anyone dealing with a similar Seafile + GlusterFS setup.

Regards