Restarting server (Docker) correctly after failure

Seafile Server
1

Hello there!
I should preface this by saying that I’m a Linux newbie and I’m just having my luck googling around for solutions and instructions, so I have no idea how most of the commands work. I’m also on Debian 8 (Jessie).

I’ve had my server crash overnight due to some issues connecting to MySQL, and woke up to:

Page unavailable
Sorry, but the requested page is unavailable due to a server hiccup.
Our engineers have been notified, so check back later.

It apparently crashed around 5-6 hours into the uploads and the log shows MySQL stopped being “unreachable” around 6 hours after that, with a couple “missing or corrupted repo” errors at the start.
I tried restarting the container in Docker but to no avail - the URL that was previously working on custom port is not reachable anymore.
I’ve had it work more or less fine for about a month, syncing, sharing and uploading folders and files through both browser and client, and only today I had to sync a massive quantity of files, including many small ones and a few really big ones (video files and such).
Not sure if that’s the incorrect way to restart it, but I just did sudo docker restart seafile for that.

First time install and setup I did:

sudo docker run
-d --name seafile
-e SEAFILE_SERVER_HOSTNAME=seafile.​myactualdomain.​com
-v /opt/seafile-data:/shared
-p 8083:80 seafileltd/seafile:latest

And then inside my control panel (browser), changed my url and locations to respect the domain (subdomains don’t work on my server and I never set them up, so I just accessed my Seafile through custom port (8083) on normal domain - and port 80 is already in use, so I used 8083:80 in the command).

I have read about the FSCK script that might help in case the problem was due to corrupted data, but I cannot find that anywhere on my server. In fact, I have no idea where the installation folder of Seafile is; I was lucky enough to find the logs, conf folders and data inside the /opt directory, but this “cd seafile-server-latest ./seaf-fsck.sh”, as well as any other mentioned script/folders I can find in the docs, eludes me. I have read the installation path depends on where I first installed, but I really only set up Docker and left all the settings by default or by the docs, and it just worked.

Any suggestion?

2

Hey @Banderi

first a few docker commands, with docker container ls you get an overview of the running containers. The restart command should be docker container restart <container name or id from the ls command>"

Sometimes the restart doesn’t work, then check if the container is running with docker container ls then stop the container with docker container stop <container name or id from the ls command> and start it with docker container start <container name or id from the ls command>

To search for broken files (SeafFSCK) you need to run seaf-fsck.sh. This file is not located in the Docker Volume under /opt/seafile-data because only the files that are permanently needed are stored in this volume.

To jump into the container you have to start a bash in the container. The command would be docker exec -it <container name> /bin/bash. Then you jump into the container where you can see where you are with the command ls. Then jump to the seafile-server-latest folder cd seafile-server-latest where you will find the seaf-fsck.sh you can run as described in the manual.

3

Hello there @D_FR!
Thanks so much for the inputs, I’m trying them now.

I have checked the containers status with ls and oddly nothing appeared in the list, was perplexed as I distinctly remember I started the container earlier;
I have started it again using docker start seafile and immediately checked the list again - it appears to be running for a couple of seconds, then it disappears. I have also tried docker container start seafile and even the container id instead of the name, but the exact same happens (also I will note I need to prepend sudo to the commands most of the time, if that changes anything).
Also testing the stop command I now notice that regardless of the command working successfully or not, it will always echo back the container’s name/id used, which I previously thought it was only on success.

I’ve tried docker logs on the container, and it printed out some repeating errors, in particular it looped through these:

mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!

And on shutdown:

*** Shutting down /scripts/start.py (PID 38)...
*** Shutting down runit daemon (PID 37)...
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown...
Mar 26 08:35:41 4ab99bbb00e1 syslog-ng[28]: syslog-ng shutting down; version='3.13.2'
*** Init system aborted.
*** Killing all processes...
*** Not all processes have exited in time. Forcing them to exit.

*** Running /etc/my_init.d/01_create_data_links.sh...
*** Running /etc/my_init.d/10_syslog-ng.init...
Mar 26 12:28:35 4ab99bbb00e1 syslog-ng[27]: syslog-ng starting up; version='3.13.2'
*** Running /etc/my_init.d/99_mysql_setup.sh...
*** Booting runit daemon...
*** Runit started as PID 36
*** Running /scripts/start.py...
Mar 26 12:28:36 4ab99bbb00e1 cron[45]: (CRON) INFO (pidfile fd = 3)
Mar 26 12:28:36 4ab99bbb00e1 cron[45]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
[2019-03-26 12:28:36] Skip running setup-seafile-mysql.py because there is existing seafile-data folder.
[03/26/2019 12:28:36][upgrade]: The container was recreated, running minor-upgrade.sh to fix the media symlinks
[03/26/2019 12:28:36][upgrade]: Running script /opt/seafile/seafile-server-6.3.4/upgrade/minor-upgrade.sh

-------------------------------------------------------------
This script would do the minor upgrade for you.
Press [ENTER] to contiune
-------------------------------------------------------------


------------------------------
migrating avatars ...


DONE
------------------------------


updating seafile-server-latest symbolic link to /opt/seafile/seafile-server-6.3.4 ...

DONE
------------------------------


failed to run "ccnet-server -t"
[03/26/19 12:28:37] ../common/session.c(132): using config file /opt/seafile/conf/ccnet.conf
[03/26/19 12:28:37] ../common/ccnet-db.c(124): Failed to get database connection: Failed to connect to MySQL: Can't connect to MySQL server on '127.0.0.1' (111).
[03/26/19 12:28:37] user-mgr.c(769): Failed to create user db tables.
Traceback (most recent call last):
  File "/scripts/start.py", line 86, in <module>
    main()
  File "/scripts/start.py", line 71, in main
    call('{} start'.format(get_script('seafile.sh')))
  File "/scripts/utils/__init__.py", line 68, in call
    return subprocess.check_call(*a, **kw)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/opt/seafile/seafile-server-6.3.4/seafile.sh start' returned non-zero exit status 1
*** /scripts/start.py exited with status 1.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
*** Shutting down runit daemon (PID 36)...
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown...
Mar 26 12:28:37 4ab99bbb00e1 syslog-ng[27]: syslog-ng shutting down; version='3.13.2'
*** Killing all processes...

I’ve read I can start a bash session immediately to avoid startup crashes, should I use that to look into the .sh script or find out why MySQL is not behaving first?

4

Hey,
first of all, a brief explanation. Each time the Docker Container is started, it checks whether a volume exists. So if there is already a database and seafile-data. If so, docker uses them and all files from /opt/seafile-data/

Can you tell me what you have in /opt/seafile-data/seafile/conf/ccnet.conf under [Database]? I guess the host will be “localhost” with you. I have rebuilt this on my PC, if I enter “localhost” instead of 127.0.0.1 I get the same error as you.

Correct should be there:

[Database]
ENGINE = mysql
HOST = 127.0.0.1
PORT = 3306
USER = seafile
PASSWD = 01a989de52f72da-xxxxxxxxxxx
DB = ccnet_db
CONNECTION_CHARSET = utf8

Change the ccnet.conf and then the container should start without problems.

5

Hey there,
my current config is set as following, which seems to be the same as suggested:

[Database]
ENGINE = mysql
HOST = 127.0.0.1
PORT = 3306
USER = seafile
PASSWD = xxxxxxxxxxxxxxxx
DB = ccnet_db
CONNECTION_CHARSET = utf8

However, I had noticed the url on one of the above settings didn’t match my configurations and I tried to change it… unfortunately, due to conflicting file permission and FTP session, my setting file is now gone. Hurray!

Do you know by any chance if there’s a way to set it up again or recover it? I could maybe get a sample default file and just change the few lines that seemed important, since it didn’t seem like a very long file.

I’m seeing these lines in the docs regarding the file, which I don’t know if would be impossible for me to recover…

# Used internally. Don't delete.
ID=eb812fd276432eff33bcdde7506f896eb4769da0

# Used internally. Don't delete.
NAME=example
6

There’s a way to get back to your data.

However, you have to be very careful not to lose your data. Copy only do not move

You create a new container with a different name and a different volume assigned to it.

sudo docker run
-d --name seafile-recover
-e SEAFILE_SERVER_HOSTNAME=seafile.​myactualdomain.​com
-v /opt/seafile-data-recover:/shared
-p 8083:80 seafileltd/seafile:latest

After this new container is started and reachable, you stop it again.
Now go to the new folder under /opt/seafile-data-recover/ and delete the following directories:

/opt/seafile-data-recover/db/ccnet_db
/opt/seafile-data-recover/db/seafile_db
/opt/seafile-data-recover/db/seahub_db
/opt/seafile-data-recover/seafile/seafile-data
/opt/seafile-data-recover/seafile/seahub-data

You then get these folders from the old directory and copy them into the new one.
So copy /opt/seafile-data/db/ccnet_db to /opt/seafile-data-recover/db/ccnet_db. The same for all folders deleted above.

Finally you restart the seafile-recover container and your data should be there again.

7

Hey there,
great idea! I saw that suggestion from another thread as well and am trying it out; I’m now attempting to get the second instance running, unfortunately the url now gives me a 502 (Bad Gateway) error, which I’m trying to understand… I think it might be related to my prior MySQL errors since I don’t remember this happening on my original installation, though that’s a blind guess.

EDIT: The error pages talks about Nginx but I don’t have it installed. In fact, the docs said either Nginx or Apache worked and so it worked fine last time. But the container gets somehow stuck before it can even generate the config files - I have no ccnet, seahub/seafile logs and settings folders, nothing.
If I try to run seahub.sh it’s complaining:

Error: there is no ccnet config directory.
Have you run setup-seafile.sh before this?

The docs say to just wait a few minute after deploying with Docker and the initialization will be done automatically, which is exactly what I did last time. Something it’s preventing the initialization from succeeding or even starting, it seems.
Is there a way to check the initialization scripts’ logs, maybe?

EDIT2: Ok, I’ve attempted again to make a new setup a couple times; the setup now runs a bit longer and the error is not a 502 anymore, rather it refuses connection entirely. docker logs displays this:

*** Running /etc/my_init.d/01_create_data_links.sh...
*** Running /etc/my_init.d/10_syslog-ng.init...
Mar 27 12:11:22 9ffb76325b88 syslog-ng[23]: syslog-ng starting up; version='3.13.2'
*** Running /etc/my_init.d/99_mysql_setup.sh...
Rebuilding mysql data dir
Starting mysqld
Waiting for mysqld to come online
Mar 27 12:11:24 9ffb76325b88 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] /usr/sbin/mysqld (mysqld 10.1.34-MariaDB-0ubuntu0.18.04.1) starting as process 216 ...
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [ERROR] mysqld: Got error 'Size of control file is smaller than expected' when trying to use aria control file '/var/lib/mysql/aria_log_control'
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [ERROR] Plugin 'Aria' init function returned error.
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
Mar 27 12:11:24 9ffb76325b88 mysqld:
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] InnoDB: The InnoDB memory heap is disabled
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] InnoDB: Compressed tables use zlib 1.2.11
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] InnoDB: Using Linux native AIO
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] InnoDB: Using SSE crc32 instructions
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] InnoDB: Initializing buffer pool, size = 128.0M
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] InnoDB: Completed initialization of buffer pool
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [ERROR] InnoDB: auto-extending data file ./ibdata1 is of a different size 0 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages!
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [ERROR] Plugin 'InnoDB' init function returned error.
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [Note] Plugin 'FEEDBACK' is disabled.
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [ERROR] Unknown/unsupported storage engine: InnoDB
Mar 27 12:11:24 9ffb76325b88 mysqld: 2019-03-27 12:11:24 139740625689728 [ERROR] Aborting
Mar 27 12:11:24 9ffb76325b88 mysqld:
Mar 27 12:11:24 9ffb76325b88 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended

EDIT3: Found the issue… and I see what happened in the end.

banderi@face:/opt/seafile-data$ sudo du seafile -s -h
62G     seafile

While I was transfering my files during the night, apparently the partition got full. Mysql crashed and could not get back up due to that:

banderi@face:~$ sudo systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: failed (Result: exit-code) since Thu 2019-03-28 16:16:40 PDT; 20s ago
  Process: 2109 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
  Process: 2248 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)

Mar 28 16:16:40 face systemd[1]: Starting LSB: Start and stop the mysql database server daemon...
Mar 28 16:16:40 face mysql[2248]: /etc/init.d/mysql: ERROR: The partition with /var/lib/mysql is too full! ... failed!
Mar 28 16:16:40 face /etc/init.d/mysql[2268]: ERROR: The partition with /var/lib/mysql is too full!
Mar 28 16:16:40 face systemd[1]: mysql.service: control process exited, code=exited status=1
Mar 28 16:16:40 face systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
Mar 28 16:16:40 face systemd[1]: Unit mysql.service entered failed state.

I have now cleared up space and restored my Seafile installation, everything is working perfectly fine again!