This is very interesting! Can you tell me more about that client that’s working fine? Which client version exactly is it? On what operating system and which detailed version?
All machines run Arch Linux, fully up-to-date. The working client is version 6.1.4. The non-working clients are 6.1.6. Unfortunately the AUR doesn’t seem to retain old versions, so I can’t easily try 6.1.4 on the non-working machines. Server is 6.2.5 but I’d be very surprised if it’s a server issue.
I’ve tried a whole bunch of things - uninstalling/reinstalling the client, deleting my config and .seafile-data, disabling 2FA… all to no avail. The logs aren’t useful; they just say the same thing (“Failed to add download task: Argument can’t be NULL”).
I’m a software engineer / devops kind of person so I’m happy to try anything (except hacking the code, my C is super rusty).
I’m running 6.1.6 client and have no issues with any of my libraries. I’m on Debian behind NGinx. We went through the code and found that the error is thrown by the rpc shell script in Seafile Server. It happens when all the arguments passed are non-existent. Now, whether the client is not sending it, or there is a version mismatch somewhere, we have not yet identified.
Yeah there’s gotta be something different. I was able to install 6.1.4 on a machine on which Seafile never worked, and interestingly, the problem persists, so it doesn’t seem like the issue is only present in specific versions.
I compared the packages on the working vs non-working machines; there are many more packages installed on the working machine, but all the common libraries to all machines (working and not working) are on the same (most recent) versions.
I set SEAFILE_DEBUG=all, but the logs didn’t include any extra information.
I don’t think it’s necessarily a Seafile issue. There could be a corrupt file, a wrong version, or even a permissions problem with something. As quickly as it happens, almost instantaneously, something is not passing the arguments from the clients to the server. Unfortunately, I don’t know enough about how the client interacts with the server and the dependencies the information runs through in order to give an objective answer. I got to thinking about the django logs, but I doubt that would have anything to do with it. You may want to check that log.
The server logs are not helpful in this case; I guess the client doesn’t get far enough for much to show on the server. Anyway since one client works fine, it doesn’t seem likely that it’s a server issue.
Even if the root cause is not Seafile itself, Seafile’s entirely unhelpful error message makes diagnosing this problem pretty much impossible.
Update:
I ended up installing every single package that is present on the working client to a not-working client; cleared out Seafile configs and reinstalled the client, but unfortunately I still get the same message.
It won’t hurt to open it as a bug, but this morning I updated to 6.2.5 from 6.2.4. I don’t have the issue. I have clients using 6.0.X and two or three using 6.1.6, on Win 7, Win 8, and Win 10, and it’s working fine on all of them. I am on Debian 9 with NGinx. Also, remember that when I tried it on your server, I got the same error you did on one of my clients, and that client is working fine with my setup. That client was 6.1.6, btw.
I wish I knew more about what happens between the client and the server, and the processes that communicate between the two, but I don’t.
Ok, I’ve gone through the code to see how the client and server communicate with each other. Python is involved with handling cloning and downloading. Being that I’m not a Python programmer, and the clone process spans a lot of the code, the best that I can tell is that there is a problem between Python and the server. Something there isn’t passing any of the parameters that the server expects in order to do a sync.
That being said, go ahead and report it as a bug. It’s certainly something that the programmers will have to look at.
It’s something with .seafile-data. I synced .seafile-data from the working machine to a non-working machine (and also synced a library), and that library is syncing perfectly. (I don’t know if this was a very good idea, but I’m rather desperate to get things working…)
However, I still cannot sync any libraries not already present on the working machine, and I cannot create new libraries, so this isn’t a solution or even a decent workaround.
Looks like most of the useful data in .seafile-data are SQLite databases, so while not impossible to compare, definitely not easy.
Also, remember that when I tried it on your server, I got the same error you did on one of my clients, and that client is working fine with my setup.
Note that I’m not deryo; s/he’s the the person to whose server you connected. But that’s interesting that your client had the same problem on deryo’s server. (Actually I thought you were a Seafile developer which is why I asked if I should open a bug. ) Anyway - an issue with .seafile-data could explain the behavior you guys experienced too.
Correct. I’m not a Seafile developer, but I’ve done my share of scripting and programming over the years. Arguments that don’t pass are a common thing in the developer world, and usually due to a version incompatibility or bad coding. Sometimes, it can be due to file corruption. In this case, bad coding is not the issue since it is working for other people.
As for Deryo’s problem, libraries would create. They just wouldn’t sync. Even creating the library from the client would work, but would bomb out while trying to sync.
BTW, I’m not certain what you mean when you say you synced seafile-data from the working machine to a non-working machine. Are you saying from a working server to the non-working server? Also, did you sync seafile-data folder on the client or the server?
I can dig further into the code to see how Seafile handles the sync process. I looked into it some and I think the issue is with the “clone” section of the seafile script. I’ll dig further into it as time permits.
Oh, and BTW… Here is the list of arguments that Seafile expects to be passed to the rpc… One or all of those is null, which is what is causing the problem. It’s just a matter of tracking down which program is failing to do it:
Can you look at the logs on the server and the client and let me know the errors you are getting in each one? That will help me narrow down which sections of the code to track. The rpc server is handled by a C++ program, and it’s quite lengthy.
Are you saying from a working server to the non-working server? Also, did you sync seafile-data folder on the client or the server?
Sorry I wasn’t clear before - I didn’t do anything on the server. I entirely replaced the non-working client’s .seafile-data with the working client’s. But like I said, this only let me sync already-existing libraries that were copied to the new computer; the new machine still can’t sync other libraries or create new libraries.
Just because it’s working for other people doesn’t mean it’s not a bug. As software engineers, I suspect we both know this all too well.
So the problem I’m experiencing is a little different from deryo’s. Still the same error message, so it seems likely they’re still in some way related.
Deryo, if you want to try what I did, I’d be very curious so know the outcome. Make backups first…
Bad code falls under the category of bugs, but not all bugs are due to bad code. In this case, my statement was that it wasn’t bad code. It could still be a bug due to corruption, a version mismatch, etc.
Deryo’s problem… We tried installing Seafile client to a machine that had never had it before. Same problem. He created me an account to log into his server, which I did successfully. I was able to download and upload from the WebUI.
However, from a machine that works fine in my environment and syncs perfectly with my server, I got the same issue as did he when I tried to sync to his, and that’s even trying to create a new folder and sync it as a new library.
What happens with Deryo is it creates the library perfectly. However, when it then checks for files to sync, the error is thrown, even if the folder is completely empty.
So, the problem appears to lie within the clone or download section of Seafile. The Seafile client code slings one error while the server logs another. So, if I had those error logs from the server and the client, I may be able to track down the issue at hand by looking at the code.
Oh, also in Deryo’s case, he stated that he did upgrades and switched over to NGinx. So, it’s really hard to track it down, but I’m almost certain it’s not NGinx.
So, did you do upgrades or anything before it started happening to you?
Oh, and if I’m understanding you correctly, you copied the seafile-data folder from a working client that syncs with another server, and it works syncing with the working server and the seafile data you copied which is compatible with the working server.
If that’s the case, it certainly sounds like a versioning issue, possibly with the commits/repos, or even one of the db files on the server that needs to be upgraded.
And, the only thing I have as far as errors go when I tried it is the applet.log on my client computer. Unfortunately, I don’t have other log files, such as the system log, seafile logs, etc in order to track it down. So, when mine failed on Deryo’s server, here is what I got: (I changed the name of his domain to protect it from the public eye)
[03/18/18 11:22:19]ServerStatusService: ignore request for host “seafile.deryosserver.com”
[03/18/18 11:24:32]Failed to add download task: Argument can’t be NULL
[03/18/18 11:29:26]Failed to add download task: Argument can’t be NULL
[03/18/18 11:37:00]Failed to add download task: Argument can’t be NULL
By the way, thanks for continuing to think about this weird issue. I’m glad someone is interested!
The server has gone through two or three upgrades since this error has occurred. In the past I was too busy to actively try to diagnose this issue and was able to make do with just one client, but now I really need multiple clients.
This problem exists on (at least) client versions 6.1.4 and 6.1.6 on the “bad” machines. The “good” machine is still on 6.1.4, and I’m not especially really inclined to upgrade, lest it break. The “good” machine has also had its client upgraded several times without incident since I started using Seafile a few years ago.
And here’s where it gets weirder…
I decided to run seaf-fsck.sh. One library had three or four damaged files, so I repaired that library.
Block 5786389b78bf829710272aac438a91f6887c8c6b is damaged, remove it.
File /path/to/some/file(62273884) is damaged, recreate an empty file.
After the repair, the good client now can no longer sync that library; it said to resync it. So I rescyned it, and guess what? Our favorite “argument can’t be NULL” message!
Other libraries on the good client continue to sync correctly. This is kind of driving me nuts but at this point it’s in for a penny, in for a pound.
My next idea, and I’m bummed it’s come to this, is to use seaf-fsck.sh to export all my libraries (~2 TB) and delete and reinstall a fresh Seafile server. Then re-import with seaf-import.sh. Finally, on the clients, sync their existing directories with the re-imported libraries on the server. Anyone have any thoughts about whether this is a bad idea?
If you could provide some error logs from the server, it would be helpful. So far, all I have is error logs from the client when I tried on Deryo’s machine. I’ve been through the code the best I could, and I know that the error is generated from the server side based on its location in the code.
The error logs I’m most interested in are the typical Seafile logs, but also any other logs that may be helpful, such as the systemd log.
I have noticed that during the upgrade path, especially when you are supposed to run several scripts during a major upgrade, that something will sometimes fail to update/install, and if you aren’t watching it during the upgrade, you could miss it. Back in the days of the 4.X to 5.X conversion, I did one, and accidentally missed one of the upgrade scripts. As a result, my seafile failed to sync, though the webui worked fine.
Also, as time goes one, some settings inside the config files are deprecated. Most people just leave them there, but eventually, they have caused problems for me in the past. Each time I do an upgrade, I check the manual to see what’s changed with the config files, and then change them accordingly.
One quick note - when seaf-fsck.sh exports, the files lose their original mtime/ctime and just have the date they were exported. So that’s pretty terrible (although I could script something to correct the file dates).
Another issue is seaf-import.sh is only shipped with the Pro edition.
Yeah, that pretty much happens to any exported file from any program. It’s an annoyance, for certain. My backup restores are that way. The date changes to the date I restore, rather than the date they were originally created. I also have that problem whenever I do a fresh install of Windows on my PC. The file dates change on the files I copy over.
) Anyway - an issue with .seafile-data could explain the behavior you guys experienced too.