Files getting randomly deleted

I’ve had this issue occasionally before, but never this bad. Once, years ago, it was blamed on a bug in one of the Mac client versions. Even after removing the bugged client from the environment, it still occasionally crops up.

Basically what happens is a big swath of a library will disappear, and it will show up in the history:

This user didn’t even do anything to the library, at least, they didn’t delete a ton of random files.

When this happens I have to restore the last good snapshot, then manually re-add all the files from history up to the present day.

  1. Why is this happening? It’s been going on for a long time, and it’s become too much to bare!

  2. Is there a better way to fix my libraries? I could just restore all the deleted items but viewing trash never works, I just get a 500 error, guessing because there is too much trash. I could try and identify when the accidental delete happened in the history and restore the snapshot right before, but then I have to MANUALLY merge in all changes since the present day. Currently I have a large library where the accidental deletion seems to have happened months ago, and there’s been tons of changes since then! Basically I will have to spend TONS of hours manually reassembling this library, this isn’t acceptable! Also, didn’t I used to be able to download a history snapshot as ZIP? Now the only option I have is to restore the whole snapshot!

Someone please throw me a bone!

We haven’t received reports about such kind of problems for a long time.

How often does it occur in your server? Can you find which client caused the problem?

Yesterday something happened to my installation, that at first glance did look quite like the subject of this thread says.

Closer inspection showed: A user connected from a PC that has not been used for a while, where the local cache was outdated. The client there synced the old files to the server and undid all the new stuff here.

The Library-Restore from the WebUI went both quick and smooth and everything is back to order. Later today I will let you know the bad client’s release information.

The client, that behaved badly, was revision 5.0.6.0, system clock was correct, the server is 5.1.4 - I did not dare to let the old client try another time (this is a live system, not a staging area), instead installed the current client (windows) and this time the system synced to the server, not the other way around. I got lucky, because the mischief was noticed early, seafile was up to the task, nothing much to aid in helping prevent such things happening though from my part.

Hello,

I am using Seafile Pro 6.2.4 and Seafile Client 6.1.1.
Recently I’ve had the “random file deletion” problem on different folders in a library.
One of the folder has been deleted and restored 2 to 3 times.
We also had the exact same problem for a customer.
And probably more that we didn’t discover yet…

This is a very critical matter for us.

Does anyone have a solution or a lead that we could follow?

Thanks

Can you post the screenshots of the library history at the server side that contain the record of the file deletion?

Hello Daniel,

Sure !
Is it this one that you want ?

I mean record like this:

It should tell you which client delete the files. And records around the record that deleting the files can also help.

OK, I logged into Seahub, went to the library and then displayed the history using the clock icon.
Here is what I’ve got :


As you can see, there are not much details : the deletion happened in the “Détails” event.
I am supposed to be the user responsible for all thoses events…

When I click on details, I have all the modifications listed :

  • Two subfolder and their files marked as new (which they are not)
  • A lot of deleted files, which I didn’t delete
  • A modified file that I don’t believe to have modified at this time of the day
  • 5 subfolders deleted, which I’m sure I didn’t delete

Only two accounts have access to this encrypted library and the other person didn’t make any change at this time.

This morning, a third machine with the other account and Seafile Client 6.1.3, deleted the content of a folder.


There was no user on this machine at this time…

Are there anything unusual at the client log (seafile.log)?

In the Seafile.log we have recurrent errors like this, but not at the time of file deletion :
[01/08/18 14:03:05] http-tx-mgr.c(771): libcurl failed to GET https://[server]/seafhttp/repo/bd5aa9fe-6839-480a-a483-7a5a30454d14/commit/HEAD: Couldn’t resolve host name.
[01/08/18 14:03:05] http-tx-mgr.c(771): libcurl failed to GET https://[server]/seafhttp/repo/59911a3e-51f7-40d8-ab45-96391deb2f08/commit/HEAD: Couldn’t resolve host name.
[01/08/18 14:03:05] http-tx-mgr.c(771): libcurl failed to GET https://[server]/seafhttp/repo/1393e514-8139-4f2d-8e84-6620ff4e80ac/commit/HEAD: Couldn’t resolve host name.
[01/08/18 14:03:05] http-tx-mgr.c(771): libcurl failed to GET https://[server]/seafhttp/repo/c2537294-c8a6-4e23-a334-08576857c1b6/commit/HEAD: Couldn’t resolve host name.
[01/08/18 14:03:05] http-tx-mgr.c(1023): libcurl failed to POST https://[server]/seafhttp/repo/locked-files: Couldn’t resolve host name.
[01/08/18 14:03:05] http-tx-mgr.c(1023): libcurl failed to POST https://[server]/seafhttp/repo/folder-perm: Couldn’t resolve host name.

At the precise time of the file deletion problem (18:24:42), here is what we have :
[01/08/18 18:24:42] sync-mgr.c(702): Repo ‘Direction’ sync state transition from ‘committing’ to ‘uploading’.
[01/08/18 18:24:42] http-tx-mgr.c(3423): Upload with HTTP sync protocol version 2.
[01/08/18 18:24:42] http-tx-mgr.c(1132): Transfer repo ‘59911a3e’: (‘normal’, ‘init’) --> (‘normal’, ‘check’)
[01/08/18 18:24:42] sync-mgr.c(702): Repo ‘Partages publics’ sync state transition from ‘synchronized’ to ‘committing’.
[01/08/18 18:24:42] http-tx-mgr.c(1132): Transfer repo ‘59911a3e’: (‘normal’, ‘check’) --> (‘normal’, ‘commit’)
[01/08/18 18:24:42] http-tx-mgr.c(1132): Transfer repo ‘59911a3e’: (‘normal’, ‘commit’) --> (‘normal’, ‘fs’)
[01/08/18 18:24:42] http-tx-mgr.c(1132): Transfer repo ‘59911a3e’: (‘normal’, ‘fs’) --> (‘normal’, ‘data’)
[01/08/18 18:24:42] http-tx-mgr.c(1132): Transfer repo ‘59911a3e’: (‘normal’, ‘data’) --> (‘normal’, ‘update-branch’)
[01/08/18 18:24:43] http-tx-mgr.c(1132): Transfer repo ‘59911a3e’: (‘normal’, ‘update-branch’) --> (‘finished’, ‘finished’)
[01/08/18 18:24:43] sync-mgr.c(702): Repo ‘Direction’ sync state transition from ‘uploading’ to ‘initializing’.
[01/08/18 18:24:43] repo-mgr.c(3739): All events are processed for repo a49df0be-275c-41c3-8739-8e1e8031b90b.
[01/08/18 18:24:43] sync-mgr.c(702): Repo ‘Partages publics’ sync state transition from ‘committing’ to ‘initializing’.
[01/08/18 18:24:43] sync-mgr.c(702): Repo ‘Actif’ sync state transition from ‘synchronized’ to ‘committing’.
[01/08/18 18:24:45] repo-mgr.c(3739): All events are processed for repo 7d3ef64b-3a41-4e98-b64a-dac6a652293b.
[01/08/18 18:24:46] sync-mgr.c(702): Repo ‘Actif’ sync state transition from ‘committing’ to ‘initializing’.
[01/08/18 18:24:46] sync-mgr.c(702): Repo ‘Commun’ sync state transition from ‘synchronized’ to ‘committing’.
[01/08/18 18:24:46] repo-mgr.c(3739): All events are processed for repo 805d8f88-fb11-4d73-bf51-6e6cdc5eb7f4.
[01/08/18 18:24:46] sync-mgr.c(702): Repo ‘Commun’ sync state transition from ‘committing’ to ‘initializing’.
[01/08/18 18:24:47] sync-mgr.c(702): Repo ‘Direction’ sync state transition from ‘synchronized’ to ‘committing’.
[01/08/18 18:24:47] repo-mgr.c(3596): All events are processed for repo 59911a3e-51f7-40d8-ab45-96391deb2f08.
[01/08/18 18:24:47] sync-mgr.c(702): Repo ‘Direction’ sync state transition from ‘committing’ to ‘synchronized’.
So no error at this point in time…

Don’t have much information from the log.

How often does this problem occur? Does it happen only in one of the library (and probably another on your customer)?

The problem occurs quite frequently, I’m afraid.
For instance, we’ve had two files and a folder deleted again yesterday (by myself, supposedly).
I don’t know if it happens on other libraries, at least we didn’t notice it yet.

I will run a seaf-fsck on both libraries, but they are both recent ones, so I would be surprised to find any corruption.

Seaf-fsck went well without finding any corruption.

What do you think about the “libcurl failed to GET” errors? Is there anything to worry about?

Any other idea?
Is there a theoretical file limit in Seafile?

Those (the libcurl errors) did likely occur due to missing network connectivity (e.g. after wakeup, after boot, while there were some disconnect (wifi) or connection outage (DSL, VDSL, …) and don’t have any impact.

This is not related.

We are hosting and managing Seafile for customers in a professionnal environment and this situation is very bad for us.

Does anyone have any lead at all ?

I sincerely hope this is not the case. The seafile client(s) should be able to be smart enough to not think a network outage equates to “delete all data”.

It was about the libcurl lines. As I see now it wasn’t that clear.

I’ve edited my last comment to better reflect that.

Hi @RomainC

In the seahub history interface, can you give a screenshot of the change description for the deletion event, and a screenshot of the details for that event? That may give us some clue.

Can you also set an environment variable on Windows to let the client print more useful messages in the log? Please set env variable ‘SEAFILE_DEBUG’ to ‘watch’. That would ask the client to print the file system change events in seafile.log. When the deletion happen again, you may find useful information in the log.