Seafile/Docker, Failed to write fs object, "not a directory"

Docker
, ,
#1

I am receiving a strange error in my seafile.log, clients can’t upload data for now. The Windows client just says “server error”.

This is the log output:

2023-02-06 20:42:05 …/common/obj-backend-fs.c(227): [obj backend] Failed to open tmp file /opt/seafile/seafile-data/storage/fs/2807eb26-c8d9-4921-8119-ac1fe2691b8c/f6/2b4b3c18d1403a63a3032c2bad056284c4763a.BG6ZZ1: Not a directory.
2023-02-06 20:42:05 …/common/obj-backend-fs.c(309): [obj backend] Failed to write obj 2807eb26-c8d9-4921-8119-ac1fe2691b8c:f62b4b3c18d1403a63a3032c2bad056284c4763a.
2023-02-06 20:42:05 http-server.c(2242): Failed to write fs object f62b4b3c to disk.

These 3 lines repeat, always with a different string just before the “not a directory” message, so i.e. BG6ZZ1, LNKA01, N0E0Z1, etc.

I am running Docker on Windows Server, the data directory is mounted to a dedicated drive in Windows. The drive has enough space on it. It has worked really well for about a week and suddenly this.

EDIT: See last post about investigation results.

#2

I noticed the log contains line references, so I’ve checked the source files for this. It appears that Seafile simply fails to create a temp file on disk. That sounded weird, so I ran ls -l over the directory in which the file was supposed to be, and I couldn’t cd into the f6 directory that is part of the path in the log. Turns out that somehow, f6 is not a directory at all:

image

Everything else is a dir, except f6. Furthermore, since the container is running in Docker for Windows (Windows Server), I’m able to navigate to the directory manually and here, f6 is a directory:

image

… and I can totally navigate into it. This seems like the file system somehow managed to corrupt itself, since it’s a Windows<->Linux mount.

#3

So after doing some research, I don’t know why this data corruption has occurred. NTFS should be supported in Ubuntu. I figured I should just use something that is native to Linux, so I’m going to be migrating to NFSv3 share and I’m gonna have to figure out if that is even supported by docker, especially when running on Windows.

#4

Unfortunately I wasn’t able to save this instance’s data at all. I use TrueNAS to host everything and when copying the data from DockerVM to a NFS share, the ownership/permissions for both Seafile and MariaDB got all wrong and Seafile would never recover. I spent about a week trying to recover from this and eventually gave up today.

#5

For anyone eventually digging into the same issue - I found out that the root of the problem was corrupted MariaDB instance. I connected to it from phpmyadmin, it shows all of the tables, but when I tried accessing them, it would say the table doesn’t exist. That is what Seafile kept complaining about. When investigating, I found that the problem was created by increasing the size of the volume in TrueNAS. The volume was expanded successfully, Windows expanded it’s partition also successfully, but when MariaDB was restarted, it corrupted it’s own ibdata1 file, which apparently is crucial and the engine can’t recover from this. Even commands like TABLE CHECK and TABLE REPAIR can’t fix this. My current Seafile instance is mounted through NFS shares directly to a TrueNAS dataset, which is pain to do, but works better and snapshots/recovery are easier.