Seadrive cache and big files


#1

I have problems with big files. The other involves Adobe Indesign files (100Mb-1500Mb) and big Word files (50-200Mb). When I try to open these file types, Word often gives an error about corrupted file and Indesign crashes every time. Only way to open these files is to download these files first to cache and open them only after file is in the cache. But this is not practical as users want to just click and open these files.

Problem seems to be about cache. I can see that problematic docx files are not in their full size in the cache. If the file is 125Mb in size, I see the same file to be only 15Mb in the cache. So it seems that the file does not load to cache completely before assosiated software tries to open the file.

This behaviour is with PC and Mac clients (version 0.9.5). Is this a known limitation and is there a solution to get the files open without downloading them to the cache first?


#2

It may be due to the slow network. The client tries to download the file when it’s being read. But it cannot catch up with the read speed if the network is not fast enough.


#3

But then it hopefully blocks until the data has been received and doesn’t return a wrong result …


#4

Yes, this is desired situation and how it should work. As people are using Seadrive at office and out of office, network speeds vary a lot. But even fast 1Gbit network at office is not fast enough to load big files to cache before software is trying to access it.

Seadrive client should serve the file from cache only after it is fully cached. I do not know how this is handled on OneDrive and other solutions.


#5

It seems that the cache problem is not only resctricted to a software, I’m also unable to copy big files from Seadrive (v1.0.0) to the local drive (using robocopy or explorer). In my example, the file is 855Mb and copying stops to a random percentage. This problem happens with many different files and with multiple computers, so it is not file related nor computer related.

I tried to upload a screen capture of the problem, but the forum says that I’m not able to upload files to a post… :confused:


#6

Well, we don’t want to block a system call for that long. Sometimes it can render the system unresponsive. That’s why there is a timeout.

This problem doesn’t seem to be easy to solve. Dropbox smart sync also blocks until the entire file is cached locally. Before that the program cannot even open the file. It’s effectively the same as manually downloading the file before opening it.


#7

I think you can. The issue with not doing is that the file is corrupt - at least that is what users report.

Maybe this would be the best for SeaDrive as well.

Effectively, yes. But I think for the user it is much easier and a big help. It is well integrated in the OS, he can just open the file as any other file, he doesn’t have to upload the file manually after changing it and he doesn’t have to manage the cache. So even, if it effectively is the same as downloading the file from a technical view, from a users view it is much better.


#8

Just wanted to chime in that I agree with Shoeper on how it should function.


#9

I’m using the seadrive client on linux, and it’s great. The FUSE mount allows random access to large files (we’re putting the solution in place for a neuroscience research project). I’ve been able to copy files in the 5-10GB range just fine with a fast connection (actual throughput around 8-10MB/s) as well as read byte ranges directly just fine. Most of the access we have for now is through fast connections, although that will change in the coming months.

We definitely don’t want the drive client blocking until the whole file is available, some of the files we handle are larger than the cache. I haven’t had any issues with read errors though, but if that happened, I assume that would cause real issues. IO errors are usually not handled very robustly in applications. The read operation timeout could perhaps be extended rather than errorring out.

In the case of programs that expect full access to the file and don’t tolerate IO errors, the sync client does this, eg, copy the whole file locally.

For us, the drive client’s random read access is the differentiating factor compared to solutions like NextCloud and such which download the whole file via webdav.


#10

Try it out on a slow connection and it will be a pain.


#11

For me, I have been testing Seafile specifically for inDesign files.

As you may be aware, Adobe doesn’t officially support working on files over network storage. They only support copying the files to your local computer and working from there.

I was hoping that Seafile/drive could be a good solution so that it would seem to the users that they were accessing files on a file share as usual but in the background it would be downloading the files first so that they would in effect be working locally.

But in my testing I am also running into issues that seem to stem from the fact that Seadrive doesn’t cache the whole file if it’s large…this feature makes Seafile useless to me. Which sucks because I really really really like Seafile until I found out about this…I guess I will try NextCloud now.