Docker client to sync files with containers

Multiple libraries are now supported. Amongst other things :dizzy:

Thanks a lot for this - looks like a very nice solution.
I will definitely use this in my setup to tidy up my container mess!

It looks like there is something wrong with entrypoint.py that it always fails during the first run. I have to change the command line to something like sh -c "/home/seafile/entrypoint.py; /home/seafile/entrypoint.py" to make it work. It looks like rpc server does not start correctly during the first run.

Log attached.

seafile-client-1  | 2024-05-06 05:08:06,233 - INFO - Initializing `seaf-cli`.
seafile-client-1  | 2024-05-06 05:08:06,233 - INFO - Seafile .ini file not found, running `seaf-cli init`
seafile-client-1  | Writen seafile data directory /seafile/seafile-data to /home/seafile/.ccnet/seafile.ini
seafile-client-1  | 2024-05-06 05:08:06,287 - INFO - Starting `seaf-cli`.
seafile-client-1  | Starting seafile daemon ...
seafile-client-1  | Started: seafile daemon ...
seafile-client-1  | Traceback (most recent call last):
seafile-client-1  |   File "/usr/lib/python3/dist-packages/pysearpc/named_pipe.py", line 85, in _get_transport
seafile-client-1  |     transport = self._pool.get(False)
seafile-client-1  |                 ^^^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3.11/queue.py", line 168, in get
seafile-client-1  |     raise Empty
seafile-client-1  | _queue.Empty
seafile-client-1  |
seafile-client-1  | During handling of the above exception, another exception occurred:
seafile-client-1  |
seafile-client-1  | Traceback (most recent call last):
seafile-client-1  |   File "/home/seafile/entrypoint.py", line 307, in <module>
seafile-client-1  |     client.synchronize()
seafile-client-1  |   File "/home/seafile/entrypoint.py", line 163, in synchronize
seafile-client-1  |     repository = self.rpc.get_repo(uuid)
seafile-client-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3/dist-packages/pysearpc/client.py", line 125, in newfunc
seafile-client-1  |     ret_str = self.call_remote_func_sync(fcall_str)
seafile-client-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3/dist-packages/pysearpc/named_pipe.py", line 97, in call_remote_func_sync
seafile-client-1  |     transport = self._get_transport()
seafile-client-1  |                 ^^^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3/dist-packages/pysearpc/named_pipe.py", line 87, in _get_transport
seafile-client-1  |     transport = self._create_transport()
seafile-client-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3/dist-packages/pysearpc/named_pipe.py", line 80, in _create_transport
seafile-client-1  |     transport.connect()
seafile-client-1  |   File "/usr/lib/python3/dist-packages/pysearpc/named_pipe.py", line 44, in connect
seafile-client-1  |     self.pipe.connect(self.socket_path)
seafile-client-1  | ConnectionRefusedError: [Errno 111] Connection refused
seafile-client-1  | 2024-05-06 05:08:06,397 - INFO - Initializing `seaf-cli`.
seafile-client-1  | 2024-05-06 05:08:06,397 - INFO - Starting `seaf-cli`.
seafile-client-1  | Starting seafile daemon ...
seafile-client-1  | Started: seafile daemon ...
seafile-client-1  | Starting to download ...
seafile-client-1  | ==> /home/seafile/.ccnet/logs/seafile.log <==
seafile-client-1  | [05/06/24 05:08:06] seaf-daemon.c(553): rpc server started.
seafile-client-1  | [05/06/24 05:08:06] repo-mgr.c(6272): Removing store /seafile/seafile-data/deleted_store/commits/d228697a-d99a-4a8b-b0d8-91fb5c1a0

Would you mind sharing more details ? What are your environment variables ? How did you override the command ? You say it only fails on the first run, not on the following runs ?

From what your logs says, the [Errno 111] Connection refused suggest an issue that’s network related.

Environment variables are fairly standard. I have one volume mapped to /library, and a SEAF_LIBRARY_doc variable pointing to a doc library.

As for command line overriding, this line in a compose file would do:

  command: sh -c "/home/seafile/entrypoint.py; /home/seafile/entrypoint.py"

Unfortunately I got rid of seafile after it blowed up my files. I dumped a mutt maildir containing around 60000 emails files into it and this made seafile unhappy. A lot of errors like too many db connections and then it stopped syncing. This makes me asking myself if I could trust seafile to have my files. Though I restored my files from backup, I decided to continue using icloud for a bit longer.

I’d say Seafile can be trusted. Performance wise, it’s probably the best FOSS plug and play cloud storage provider. Nextcloud is looking quite good with caching strategies, I’m reading.

Don’t lose hope.

In the meantime, thank you for notifying the problem, I’ve logged the issue you’ve met on the Docker client, I’ll be adding an explicit and better managed error message for such situation.

I think I got the log, first run and second run:

[+] Running 1/0
 ✔ Container seafile-client-seafile-client-1  Recreated                                                                                                               0.1s 
Attaching to seafile-client-1
seafile-client-1  | 2024-05-31 19:46:03,908 - INFO - Single library detected. Multiple libraries will be ignored.
seafile-client-1  | 2024-05-31 19:46:03,909 - INFO - Initializing `seaf-cli`.
seafile-client-1  | 2024-05-31 19:46:03,909 - INFO - Seafile .ini file not found, running `seaf-cli init`
seafile-client-1  | Writen seafile data directory /seafile/seafile-data to /home/seafile/.ccnet/seafile.ini
seafile-client-1  | 2024-05-31 19:46:03,994 - INFO - Starting `seaf-cli`.
seafile-client-1  | New device id created
seafile-client-1  | Starting seafile daemon ...
seafile-client-1  | Started: seafile daemon ...
seafile-client-1  | Traceback (most recent call last):
seafile-client-1  |   File "/home/seafile/entrypoint.py", line 307, in <module>
seafile-client-1  |     client.synchronize()
seafile-client-1  |   File "/home/seafile/entrypoint.py", line 163, in synchronize
seafile-client-1  |     repository = self.rpc.get_repo(uuid)
seafile-client-1  |                  ^^^^^^^^
seafile-client-1  | AttributeError: 'Client' object has no attribute 'rpc'
seafile-client-1 exited with code 1
[+] Running 1/0
 ✔ Container seafile-client-seafile-client-1  Created                                                                                                                 0.0s 
Attaching to seafile-client-1
seafile-client-1  | 2024-05-31 19:54:41,000 - INFO - Single library detected. Multiple libraries will be ignored.
seafile-client-1  | 2024-05-31 19:54:41,000 - INFO - Initializing `seaf-cli`.
seafile-client-1  | 2024-05-31 19:54:41,001 - INFO - Starting `seaf-cli`.
seafile-client-1  | Starting seafile daemon ...
seafile-client-1  | Started: seafile daemon ...
seafile-client-1  | Traceback (most recent call last):
seafile-client-1  |   File "/usr/bin/seaf-cli", line 1023, in <module>
seafile-client-1  |     main()
seafile-client-1  |   File "/usr/bin/seaf-cli", line 1019, in main
seafile-client-1  |     args.func(args)
seafile-client-1  |   File "/usr/bin/seaf-cli", line 675, in seaf_sync
seafile-client-1  |     token = get_token(url, username, password, tfa, conf_dir)
seafile-client-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/bin/seaf-cli", line 301, in get_token
seafile-client-1  |     token_json = urlopen("%s/api2/auth-token/" % url, data=data, headers=headers)
seafile-client-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/bin/seaf-cli", line 234, in urlopen
seafile-client-1  |     resp = urllib.request.urlopen(req)
seafile-client-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3.11/urllib/request.py", line 216, in urlopen
seafile-client-1  |     return opener.open(url, data, timeout)
seafile-client-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3.11/urllib/request.py", line 525, in open
seafile-client-1  |     response = meth(req, response)
seafile-client-1  |                ^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response
seafile-client-1  |     response = self.parent.error(
seafile-client-1  |                ^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3.11/urllib/request.py", line 563, in error
seafile-client-1  |     return self._call_chain(*args)
seafile-client-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
seafile-client-1  |     result = func(*args)
seafile-client-1  |              ^^^^^^^^^^^
seafile-client-1  |   File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default
seafile-client-1  |     raise HTTPError(req.full_url, code, msg, hdrs, fp)
seafile-client-1  | urllib.error.HTTPError: HTTP Error 403: Forbidden
seafile-client-1  | ==> /home/seafile/.ccnet/logs/seafile.log <==
seafile-client-1  | [05/31/24 19:46:04] seaf-daemon.c(525): starting seafile client 9.0.4
seafile-client-1  | [05/31/24 19:46:04] seafile-session.c(390): client id = cf339026...e5d2a56, client_name = unknown
seafile-client-1  | [05/31/24 19:46:04] seaf-daemon.c(553): rpc server started.
seafile-client-1  | [05/31/24 19:54:41] seaf-daemon.c(525): starting seafile client 9.0.4
seafile-client-1  | [05/31/24 19:54:41] seafile-session.c(390): client id = cf339026fb...d2a56, client_name = unknown
seafile-client-1  | [05/31/24 19:54:41] socket file exists, delete it anyway
seafile-client-1  | [05/31/24 19:54:41] seaf-daemon.c(553): rpc server started.
seafile-client-1  | [05/31/24 19:54:41] start to serve on pipe client

Now I always get the 2nd logged error, even with @roylez’s override command. I don’t understand why it is 403 forbidden. My envs are:

  SEAF_SERVER_URL: "https://.../"
  SEAF_USERNAME: "..."
  SEAF_PASSWORD: "..."
  SEAF_LIBRARY: "..."

Also, while I’m here:

When I use UID and GID in compose file, the container immediately exit with code 4. No logs anywhere. I removed UID and GID, then I saw logs mentioned above.

Alright, found the issue on this one. Thank you for the logs. Will fix soon.

Would you have logs for this as well ?

Sorry, I tried to find logs everywhere, there are none. Only info is exit with code 4.

What’s the UID and GID your are trying to use ? The exit code 4 match a usermod/groupmod exit code.

I’m using UID 1000 and GID 100, as that is what I see with id command. I’m using NixOS.

I’ve just published a new version with a fix for the first run issue. I’ll check out the UID/GID issue next week.

Thank you!

And that’s fixed as well. The GID 100 was already taken by the group users. Since the GID 100 is used quite often, I’ve reassigned the group users to the GID 90.

Hello,
Nice work, I really like it. I still have a question : let’s say one day, I want to sync a library and later, I want to add another one. Should I destroy the container and delete the “client” volume ?
I already tried to add an environment variable for the new library but it doesn’t start the sync for this last.

Hey, thank you, glad to hear that.

It should work without the need to delete the client volume. But you should be using named libraries as soon as you wish to use multiple libraries. How have you defined your libraries ?

Hello,
I had 2 named library and I wanted to add a third. Adding environment variable didn’t start the sync for the new library.
I connect to the container itself as seafile user and use the command line to add the new library. It worked.
I just wait for tomorrow to see if the container restart still sync files.

I did reproduce the issue and deleting the client volume solves it.

So, that’s a bug, and it’s tracked: Added libraries on existing clients are not synced. (#71) · Issues · Open source software / Docker Seafile Client · GitLab.