Seadrive cache cleanup algo

seadrive

#1

Hi everyone. Just start playing with seafile, and I really like it. I have a question on seadrive though, especially on the cache cleaning algo used.
I’m using seadrive on CentOS 7 to expose my music collection to ampache, among other things. The seadrive cache is smaller than the total collection, but that shouldn’t be a problem I guess. I’ve 25GB available, and have limited seadrive cache_size to 10GB, with cache cleaning every 2 minutes

[cache]
size_limit = 10GB
clean_cache_interval = 2

So, everything is working fine, except when accessing quickly to much files. Every night, a cron job scans the whole library for new/removed files. This process should take 2 or 3 hours in total. At the begining, seadrive correctly cleans his cache, but just not quicly enough. Step by step, the cache grows, until it saturates the whole 25GB. See here a part of my log file:

[02/06/18 01:58:29] file-cache-mgr.c(2562): Cleaning cache space.
[02/06/18 01:58:29] file-cache-mgr.c(2564): 1854 files in cache, total size is 10129 MB
[02/06/18 01:58:29] file-cache-mgr.c(2585): Cache cleaning done.
[02/06/18 01:58:29] file-cache-mgr.c(2587): Removed 99 files, cleaned up 3652 MB
[02/06/18 02:10:29] file-cache-mgr.c(2562): Cleaning cache space.
[02/06/18 02:10:29] file-cache-mgr.c(2564): 3232 files in cache, total size is 10603 MB
[02/06/18 02:10:30] file-cache-mgr.c(2585): Cache cleaning done.
[02/06/18 02:10:30] file-cache-mgr.c(2587): Removed 762 files, cleaned up 2607 MB
[02/06/18 02:16:30] file-cache-mgr.c(2562): Cleaning cache space.
[02/06/18 02:16:30] file-cache-mgr.c(2564): 3200 files in cache, total size is 10051 MB
[02/06/18 02:16:30] file-cache-mgr.c(2585): Cache cleaning done.
[02/06/18 02:16:30] file-cache-mgr.c(2587): Removed 177 files, cleaned up 596 MB
[02/06/18 02:18:30] file-cache-mgr.c(2562): Cleaning cache space.
[02/06/18 02:18:30] file-cache-mgr.c(2564): 3263 files in cache, total size is 10108 MB
[02/06/18 02:18:31] file-cache-mgr.c(2585): Cache cleaning done.
[02/06/18 02:18:31] file-cache-mgr.c(2587): Removed 29 files, cleaned up 85 MB
[02/06/18 02:20:31] file-cache-mgr.c(2562): Cleaning cache space.
[02/06/18 02:20:31] file-cache-mgr.c(2564): 3431 files in cache, total size is 10719 MB
[02/06/18 02:20:31] file-cache-mgr.c(2585): Cache cleaning done.
[02/06/18 02:20:31] file-cache-mgr.c(2587): Removed 14 files, cleaned up 60 MB
[02/06/18 02:22:31] file-cache-mgr.c(2562): Cleaning cache space.
[02/06/18 02:22:31] file-cache-mgr.c(2564): 3670 files in cache, total size is 11372 MB
[02/06/18 02:22:31] file-cache-mgr.c(2585): Cache cleaning done.
[02/06/18 02:22:31] file-cache-mgr.c(2587): Removed 82 files, cleaned up 194 MB
[02/06/18 02:24:32] file-cache-mgr.c(2562): Cleaning cache space.
[02/06/18 02:24:32] file-cache-mgr.c(2564): 3813 files in cache, total size is 11865 MB
[02/06/18 02:24:32] file-cache-mgr.c(2585): Cache cleaning done.
[02/06/18 02:24:32] file-cache-mgr.c(2587): Removed 60 files, cleaned up 147 MB
[...]
[02/06/18 02:52:33] file-cache-mgr.c(2585): Cache cleaning done.
[02/06/18 02:52:33] file-cache-mgr.c(2587): Removed 66 files, cleaned up 160 MB
[02/06/18 02:54:33] file-cache-mgr.c(2562): Cleaning cache space.
[02/06/18 02:54:33] file-cache-mgr.c(2564): 6654 files in cache, total size is 20143 MB
[...]
[02/06/18 03:12:35] file-cache-mgr.c(2562): Cleaning cache space.
[02/06/18 03:12:35] file-cache-mgr.c(2564): 8315 files in cache, total size is 24895 MB
[02/06/18 03:12:35] file-cache-mgr.c(2585): Cache cleaning done.
[02/06/18 03:12:35] file-cache-mgr.c(2587): Removed 89 files, cleaned up 254 MB

Shouldn’t the cache cleaning be more agressive when over the size limit ? How does seadrive decide which files to remove from the cache ? Is it a simple LRU ? Unfortunately, as the source is not open (yet ?) I can’t check :slight_smile:


How does seadrive cache cleanup work?
#2

Nobody knows ? For now, I work arround this problem by running an agressive tmpwatch on the cache directory of seadrive. It’s not a very elegant solution, but at least it works.


#3

I don’t know how algo for cache works, but you have to think that if you insert tons of files, it takes some time to calculate changes, create changeset a maily upload files. So files may persist in cache cause they aren’t already fully uploaded. You can try to write your script with some sleep interval to get some time for let SeaDrive upload files and clean cache.


#4

I’m not writing anything on seadrive mount point. Just reading


#5

The algorithm cleans until the total cache size goes down to 70% of limit. It behaves like this at the beginning of your log. But after sometime it may find that some of your files are still being opened. So it cannot delete them from cache. Can you check that? You can check that with ProcessExplorer.


#6

I checked with lsof (running Linux).

[root@seafbridge ~]# lsof | grep '/var/cache/seadrive/media/file-cache/' | wc -l && lsof | grep '/opt/seadrive/htpc/' | wc -l
51
2
[root@seafbridge ~]# 

So, files on the mount point are being closed. A bit more files stays opened by seadrive itself in the cache dir, but not that much. The cache has more than 8000 files in it, And still, it’s not getting cleaned quickly enough. Anything else I can provide to track this issue down ?


#7

Forgot to say, /var/cache/seadrive/media is the storage dir and /opt/seadrive/htpc is the mount point


#8

@dani We’ll look into the issue to determine whether it’s a bug or not.


#9

I have the same problem. Cache size is set to 10GB, but its constantly around 30GB. So obviously the cache has some more settings which decides what to be cleaned. No files are currently open in the file-cache directory. Any advice how to keep the file-cache small?

Related: How does seadrive cache cleanup work?


#10

We found a possible cause for the cache cleaning problem. A fix will be included in 0.9 version. You can have a try.


#11

The cache on Mac is also taking a significant amount of local disk space, under my User profile. This results in my machine running out of space completely on the O/S SSD. I am currently running v0.9.0.

Has this issue been fixed in 0.9.1?

Alternatively, can I move the ~/.seadrive directory to an external drive that has much more space? I have tried this manually, now the the SeaDrive app crashes with message SeaDrive exited unexpectedly as soon as I launch the application.

Finally, how do I delete my running config / temp files / etc? As I cannot launch the application! (See above error message… haha)


#12

Thanks. I can’t try it though, as I’m running seadrive on Linux (Fedora) and 0.9 is not yet available.