[solved] Cannot backup Seafile using Rsnapshot anymore

Seafile Server
1

EDIT: Found the issue! ClamAV was the culprit. I had it enabled by accident. I’m leaving my post and this resolution for others to see in the future if they experience slowness without realizing that the AV was responsible.

I’m running OpenMediaVault and use Rsnapshot to backup my main ZFS array to another set of drives. Lately my backups have stopped completing. After a lot of testing it would appear that the backup halts at some point when it gets to my Seafile folder, or simply crawls to an unbelievable slow rate. My Seafile folder is ~400GB in size and it always has trouble around 100GB. It takes about 12hours to get to 100GB for a backup drive that benches 130MB/s. Watching it backup it would appear it takes a lot longer to backup this folder than any other.

I pulled up some logs and this is what I see:

/var/log/kern.log:Jan 10 22:23:09 omv kernel: [80357.464384] rsync D 0 20319 20312 0x00000000
/var/log/kern.log:Jan 10 22:25:10 omv kernel: [80478.301118] INFO: task rsync:20319 blocked for more than 120 seconds.
/var/log/kern.log:Jan 10 22:25:10 omv kernel: [80478.301176] rsync D 0 20319 20312 0x00000000
/var/log/kern.log:Jan 11 19:48:21 omv kernel: [29001.803557] INFO: task rsync:15870 blocked for more than 120 seconds.
/var/log/kern.log:Jan 11 19:48:21 omv kernel: [29001.803608] rsync D 0 15870 13936 0x00000000
/var/log/kern.log:Jan 12 11:39:39 omv kernel: [50752.564377] INFO: task rsync:21916 blocked for more than 120 seconds.

Now considering Rsync is the recommended backup method for Seafile and Rsnapshot uses Rsync I’m not sure what is going on. It wasn’t like this before so I’m not sure what changed. Any suggestions how to troubleshoot this?

2

Tried running Rsync manually, different options and it continues. Additionally I tried running it 15hours over the weekend and found an interesting result. The array hangs, I am no longer able to copy or run any files. I think the progress gets slower and slower to the point it hangs. Again this is only when copying the Seafile folder and memory and cpu are low. I’m unsure what causes this as it’s very unusual. Any suggestions would be highly appreciated. I may re-install the OS to see if that helps.

3

I’m curious. Have you run fsck on your server and do you see any correlation with Seafile data and the points at which rsync slows or stops?