Can I use Seafile in conjunction with NextCloud?


#1

Okay So I’ve spent the last couple of weeks trying to set up a personal cloud server, and I’ve made some changes overtime to my approach, but here’s the gist of what I’ve been trying to do currently:

image

Originally my plan was to use NextCloud for everything. I would have my work files in my /home/david/work folder synced with my work files in my /users/david/work folder on windows. I would have these files made accessible from the web via the NextCloud web gui, and I could easily create links and share files with people as I chose.

This has turned out to be a bad solution because NextCloud does not like you directly manipulating files on the machine that is running the server. It prefers you to do stuff through the web gui or on a separate machine that is running as a client. As I am using my desktop as a server AND as a client, whenever I change a file on the desktop, or I move something around using the File Manager, it gets wonky and out of sync with the nextcloud system as a whole. Basically the intended use for Nextcloud, as I understand it, is to run the server as a server, and leave it alone, while using any of the files or manipulating them etc exclusively through client machines.

So then I decided to try using Syncthing. My plan is as the above picture shows, to use Syncthing to sync the laptop and the desktop, while having nextcloud run on the desktop and make files available through the web interface for sharing etc. Syncthing is having a super hard time syncing these folders though as projects range from a few folders with extremely large binary files (Hi poly 3D scans of landscape using photogrammetry), to many hundreds of thousands of small folders with tiny text files ( such as config folders for VScode that have node module dependencies.)

So, I’m taking a step back and trying to do this process over again as a clean slate, and since that’s what I’m doing it makes sense to look and see if there are perhaps better technologies to use.

I do want NextCloud. I know that. Seafile has a lot of dope features but I def want to have the web gui, user permissions, ability to stream media, install apps, cloud edit documents, etc.

But Seafile seems more efficient for syncing than NextCloud or Syncthing, from what I’ve heard. It has delta syncing at least, and that seems that should help some with this process. And the ability to mount my large Archive drives by the Seafile seadrive function rather than using WebDav or SFTP sounds enticing.

But I’m also not clear if Seafile is the actual solution I’m looking for. As I understand it, Seafile files on the server are in a special format that can’t be manipulated or messed with. In that case I’m basically back to square one with the problem I had with NextCloud right? If I wanted to use SeaFile for syncing I would not be able to directly interact with those files on my Desktop server, correct?

I could hypothetically, I guess, run the seafile server off of the desktop and then mirror that the home folders of the desktop using the client app, but that would require using twice as much hard drive space for every file, correct? Because I’d basically be maintaining a server copy and a client copy locally.

Is my understanding of Seafile etc and how I could use it with NextCloud correct?


#2

To address a couple things first:
nextcloud/owncloud is intended to function as an appliance/service – just like dropbox.
Yes, you can in theory run them both, but it will be a cumbersome setup where you have the owncloud/seafile clients each monitoring the same folders. I have thought about doing this, but it sounds very brittle and error prone. Doing so would require at /least/ double the space (including file changes, etc)

I also found nextcloud/owncloud slow to update, and other issues with mobile uses made me switch to seafile.
I also found syncthing unacceptable for my purposes ranging from many small files to very large files. Project files were one of the main culprits for me and syncthing, the same as you.

Yes, seafile does support delta changes, and is one of the core reasons I made the switch as well.

I think first you should define your requirements.
Do you need to run the server on your desktop/workstation? If so, why?
Do the features you lack from owncloud really impact your usage? If so, you may be better off with owncloud/nextcloud.

Seafile stores all content in 1MB (I think) blocks, and does not allow direct file access. This was a fact that gave me pause, as in owncloud if my service took a dump I could at least access the OS filesystem on the server and directly access my files. Not so in seafile. This is a limitation that I have come to peace with. I understand that on the seafile server you should have a way to mount the seafile DB locally as a r/o filesystem and access it that way in the case of a failure. I have not tested this so YMMV.

If all you’re doing is keeping 2 folders in sync (or trees), you could do something like a periodic rsync, etc etc. But this wouldn’t provide you the linking, etc.

However, to address the core of your question, I would do 1 of the following:
If you NEED access to both technologies at the same time for different uses, I don’t know how to help you. There be dragons.
If you need access to the base OS filesystem, seafile is not the solution you’re looking for.

Assuming you’re running your main desktop on linux/bsd I would consider setting up a chroot/jail enviornment and picking whichever server suits you best. I chose seafile because of the android background synching works where owncloud/nextcloud did not a couple years ago. Also the delta sync contributed.


#3

The ability to easily share files or folders via links through the web etc, without download etc, is a pretty big use case need for me as I want to be able to easily send files to co workers or clients. I’m also using this server as a media server, and the ability to actually stream video from the web client without having to download anything is a pretty big plus for me.

I’m not sure you are correct when you say " you have the owncloud/seafile clients each monitoring the same folders" because for the desktop I would have nextcloud monitoring the folders via cron job, but then on the laptop I wouldn’t even bother installing the NextCloud Client at all. I would just install seafile to sync.

And I’m also not sure if you are right or maybe that we are not understanding each other, when you say “Doing so would require at /least/ double the space (including file changes, etc)”. You might be right in that using Seafile and wanting to be able to host and also access the files would require me to have the files on the desktop twice, but I don’t think that is true for the laptop at all, and it’s not so much the combination of NextCloud and Seafile as it is the particular way that I want to be able to use Seafile on my desktop.

As for the question of if I really need to have direct file access on the desktop server, the answer is I think, yes, because I’m also using it as a media server for my living room, and also because I’ll be using it as a rendering server for some neural network art projects that I’ve been working on, and I want to be able to have those image outputs automatically saved to the work folders in which I am going to be manipulating etc them on using my laptop.

If I’m understanding the situation though, it sounds like Seafile isn’t really the tool for what I’m looking for, mostly because whether or not it can be run in conjunction with NextCloud, it can’t be used on the desktop in such a way in which the files also are available for direct access from the desktop. I would really need a separate machine to run solely as a server, which I don’t have the money or space for right now.

Would you say that’s an accurate estimation?


#4

It’s not very clear what you mean by “direct access to files on the server”, but you could run Seadrive on the server if you need rw access to your libraries, that won’t be the most efficient way to do it considering that it will still need to fetch files to the cache using the network stack, but I don’t think you have many options.


#5

By direct access I mean that on my desktop, which is also running the server, I can open up a text file in /home/david/work that is being synced by seafile, and edit it, and save it. From my understanding, seafile puts all the files into the server database as blocks or whatever, and therefore you can’t actually effect those files directly from the machine that the server is running on (unless you also install the seafile client on the desktop and sync the seafile server to another part of the desktops harddrive using the client. Which means having essentailly a 100mb file take up 200mb of space, 100mb taken up as part of the seafile database, and the 100mb taken up as a client file).


#6

If you want sync and you aren’t focused on the best speed that you could access your files at, give seadrive a try, you can mount seadrive on /home/david/work and have all your libraries “synched” on the same machine you host the server, it’s similar to mounting a FTP or SSHFS filesystem using FUSE.


#7

Oooh yeah, I forgot about how seadrive can mount drives from the server. That would be a solution.

As I’m primarily using my laptop etc, and on’y really need the desktop as a render farm, personal cloud, file sync, media center etc, speed woulldn’t really be an issue. Okay cool, I think I might give this a shot. Still copying over files to my external, but this sounds like it could actually work!


#8

btw if you are working with big files, try to set seadrive’s cache to a reasonable size considering that workload.


#9

Thinking it over, would NextCloud still be able to look at the mounted virtual Seafile drive and make those files accessible through the cloud to the web interface? Or would using Seafile in this fashion essentially mean having to give up on having stuff accessible in the NextCloud environment etc?


#10

Jackalope,

I’m not sure you are correct when you say " you have the owncloud/seafile clients each monitoring the same folders" because for the desktop I would have nextcloud monitoring the folders via cron job, but then on the laptop I wouldn’t even bother installing the NextCloud Client at all. I would just install seafile to sync.

By this I meant that you would have an agent from each server watching the same directories. Which, I believe, having a cron job periodically check would count. An interesting solution, as I always have operated with the condition that I want all changes in sync between my various machines ASAP once I make a file change.

And I’m also not sure if you are right or maybe that we are not understanding each other, when you say “Doing so would require at /least/ double the space (including file changes, etc)”.

By this I meant that (If I remember owncloud/nextcloud correctly) nextcloud will keep it’s own file DB as well as histories. Seafile will also do the same, so each synced location (assuming having seafile watch the nextcloud server db location doesn’t work correctly) would store a copy with history as the server process. So a 1MB file would live as 1 seafile block, 1 nextcloud file, and wherever the client locations are. So my perspective was on total storage, not necessarily on one particular workstation where you may only have one copy locally.

I would really need a separate machine to run solely as a server, which I don’t have the money or space for right now.
Would you say that’s an accurate estimation?

Yesterday I would have said yes, but the seadrive options may be a solution for you.
Honestly, the easiest solution may be to go get or re-purpose a hard drive to use an the server’s datastore. This could be a free solution if you have a spare drive with enough capacity laying around (I always seem to have 3 or 4 old but working drives laying around).

Another solution might be that since running seafile would require at least double the datastore space on the desktop, why not spin up a lightweight nextfile/owncloud VM and host the files there? If you’re concerned about direct server access, you could repartition your desktop and allow the VM to access the partition raw. This would give you a way to mount the partition while the VM is down if needed, and you wouldn’t have to worry about VMDK or whatever the virtual disk of choice would be.

If you want sync and you aren’t focused on the best speed that you could access your files at, give seadrive a try

Great suggestion, I completely forgot about seadrive after I couldn’t get it to work properly last year or the year before.

would NextCloud still be able to look at the mounted virtual Seafile drive and make those files accessible through the cloud to the web interface? Or would using Seafile in this fashion essentially mean having to give up on having stuff accessible in the NextCloud environment etc?

This is an excellent question. My first thought would be that it could work as seadrive mounts the file as a drive in windows, and caches certain files on demand when accessed (from how I understand it).
My concern, and what I would suggest you test various modes of failure on, is that what happens if seadrive crashes or otherwise becomes unavilable, and the nextcloud client sees an empty dataset as your watched directory. I would hope that nextcloud’s client would see that the drive doesn’t exist, but there’s a small possibility (and I sincerely hope I’m incorrect here) that the nextcloud clent may think the watch directory has been emptied, and potentially delete all files in the watch directory on the nextcloud server.


#11

Cool thank you for the response!

I’m still copying over these dang files :confused: BUT it has been made easier by me finding some windows and linux programs that do it faster than the default shell applications.

Thank you for the help!


#12

Any time.

What direction have you decided to go with? Also, what programs are you using that are faster than the shell applications?

When you find a solution that works for you, please be kind enough to post the details here.

While your setup isn’t a usual one for me, interesting and non-standard solutions are some of the best ones to have on hand for outlier issues and use cases.


#13

rsync is the fastest in conclusion with netcat And it’s made for shell. So this statement is not true.


#14

I’d always heard that on initial use, rsync was quite a bit slower than other solutions since there are no file changes to be incrementally updated.

I has assumed the windows side of the situation would be where most of the performance gains came from – my guess would be faster than single threaded SMB.


#15

On windows I’m using FastCopy (https://fastcopy.jp/en/ ) and on Ubuntu I’m planning on using UltraCopier, which I don’t know if it’s faster but it’s supposed to have some extra features to help make it less error prone. https://ultracopier.first-world.info/ It’s also cross platform so if it performs well then I plan on using it on the windows machine too.

For set up, I’m unsure but I think I’m going to just get NC working again and see if I can get the default sync there to work properly. Seafile seems like a good solution in the longterm, but the fundamental issue I’m having with confusing NC through direct manipulation seems to be present no matter what solution I decide to use. But if this fresh copy and sync doesn’t work then I’ll probably move forward on getting Seafile set up. I just want to make sure the problems I was having are indeed something related to the syncing and not due to corrupted files or permissions issues.


#16

Seafile’s frontend is not the best, you might want to consider a media server as an alternative to NextCloud