I’m trying to re-upload all the several GBs of files I have on Nextcloud. Frankly, it’s impossible.
Both the Seafile client and the browser are incredibly slow uploading (the client especially), but that’s not the worse of it:
I left the browser uploading all night and in the morning it was more or less where i had left it - no progress whatsoever.
It seems the uploading process gets stuck on its own after uploading a few many files, or big files, I’m not sure; like this is 2010 or something.
You’d think you could do the initial first massive upload using SFTP or some other method (I remember doing this with Nextcloud), but that’s not possible either, since Seafile chops all the files in chunks.
How am I going to migrate all my files? To think that the files are right there, on the same VPS, just some folders away!
There should be, at the very least, a few commands or a migration script that can grab those files, for Seafile to use.
It’s hard to tell what’s going on without having proper information. Seafile has been running flawless for years on my side and maxing out the internet connection when uploading hundreds of GB has never been a problem.
If it’s just a few GB I would probably install Seafile Drive Client on the VPS and the copy the files over.
Basically the drive client provides a virtual filesystem (Fuse mount) that gives you access to all your files without the need to save them on disk (except for the cache). If you do have enough storage space on your VPS you can install the seafile-client instead.
Question: you mentioned having enough space. Do you mean to say that the mount would virtually double the space taken by the user files on seafile?
If the answer is yes, could I move the nextcloud files onto the seafile drive mount at first, then remove/disable the mount so as to regain drive space?
Seadrive CLI and Seafile CLI are 2 different programs. Seadrive uses a virtual filesystem and doesn’t require any storage space except for the cache. Seafile physically saves the files on the hard disk and thus requires storage space.
With Seafile client you basically have 2 options:
Create a library and setup syncing with your existing Nextcloud data directory. I think the library name and the directory name need to be the same for this to work, but I’m not sure as I’ve never done with Seafile CLI before. A backup of your data is definitely advisable for this option.
Create a library in Seafile server, create a new directory and setup syncing for both. Then copy your data from Nextcloud to your newly synced directory. This option requires more storage space but might be easier and saver.
Not sure I understand. Seafile/Seadrive CLI aren’t needed once the transfer is complete unless you want to keep them for different reasons.
Im not sure whats happening on your end, I had no issues Syncing a 800 gig library with 400,000 files, it was done within a day. This was using Seafile Server 6.1
It seems that there is some kind of bottleneck on one of your machines.
I’ve been using Seafile Server since version 3.x and every time I had an issue where my transfer speeds were bad there was some kind of ressource issue.
Reading your inital post there could be the following issues:
a.) Your drives are at load (not storage!) capacity. If the vps is a KVM you could try using iotop to monitor the I/O of your virtual disks. This will not work if your VPS is some kind of container (LXC, Docker, …)
b.) Your CPU is at capacity. Monitor cpu load on your VPS
c.) There is a misconfiguration in your reverse proxy (if you have any) - double check if everything checks out here
d.) Check your log files, maybe there is some kind of error you are missing? Logs of Seafile as well as for your reverse proxy.
e.) If you have trouble up- or downloading stuff to / from your seafile, especially with large files, try setting the parameter “web_token_expire_time” to a higher time. I did have an issue with a customers instance where they had slow uploads and went into timeout because the file was to large.
I cannot include links here but if you google "seafile times out with large files you can get the issues.
Currently I have three different instances of Seafile running, accumulating approx. 4TB in stored data and tens of TB every month in up- and download.
The average transfer speed in LAN is about 200MBits which could probably be higher but the storage are still rotating disks.
Are you completely up to date on the client? My Seafile system has been working flawlessly for several years but last week I was unable to upload lots of new files - photos. Kept getting the “Error” and exclamation mark in the client. Upgrading the client (I was one version behind) resolved this.
Are you uploading across the internet? If so, what’s your upload speed?
Thank you for the suggestions and sorry for the late answer. I haven’t been monitoring this thread, since I long solved this initial problem by using SeaDrive for my first upload. It worked great.
Now upload and download speeds seem to be normal for my location. I haven’t bench-marked it, but it may just well be Seafile is being faster than Nextcloud used to be. As it should.
Of all the points you make, I think point e) might be relevant in my case - it was surprising how the upload queue would just hang, not proceeding at all. I will look into it.