Metadata server constant error

Hello,

Several weeks ago, my server where Seafile is hosted from crashed. I believe that this marked when I started to have problems with my metadata server. Since I was not too reliant on it of course, it was not too big of a deal, but I would like to get it fixed, if possible, as it does not seem to work at all anymore. I am not sure of the right way to maybe reset it to its base state, if that might help, or if there is just something going on with my the containers accessing one another causing issues.

When I go to my seafile on the browser, upon clicking the library, I see “Internal Server Error”. The network inspector shows me this is an error from sending a GET request to the endpoint /api/v2.1/repos/<my-repo-id>/metadata/.

Here is some logs that appear around the time this error happens.

seaf-md-server.log
[2025-11-23 16:36:21] [INFO] starting seaf-md-server
[2025-11-23 16:37:04] [ERROR] error: failed to apply operation 88768 at 0: unknown option "89287" on column course
	from oplog.convertValueToDatum() at seafileltd.com/seaf-md-server/pkg/oplog/utils.go:77
	from oplog.UpdateRowsOp.Update() at seafileltd.com/seaf-md-server/pkg/oplog/row.go:99
	from store.applyOpLogs() at seafileltd.com/seaf-md-server/pkg/store/base.go:245
	from store.open() at seafileltd.com/seaf-md-server/pkg/store/base.go:202
	from store.Open() at seafileltd.com/seaf-md-server/pkg/store/base.go:159
	from server.queryRowsAPI() at seafileltd.com/seaf-md-server/pkg/server/query.go:29
	from server.APIHandler[...].handle() at seafileltd.com/seaf-md-server/pkg/server/api.go:120
	from server.APIHandler[...].ServeHTTP() at seafileltd.com/seaf-md-server/pkg/server/api.go:101
	from http.(*ServeMux).ServeHTTP() at net/http/server.go:2747
	from http.serverHandler.ServeHTTP() at net/http/server.go:3210
	from http.(*conn).serve() at net/http/server.go:2092
	from runtime.goexit() at runtime/asm_amd64.s:1700
seahub.log
[2025-11-23 13:52:07] [WARNING] django.request:253 log_response Not Found: /graphql
[2025-11-23 13:52:08] [WARNING] django.request:253 log_response Not Found: /api
[2025-11-23 13:52:09] [WARNING] django.request:253 log_response Not Found: /api/graphql
[2025-11-23 13:52:10] [WARNING] django.request:253 log_response Not Found: /graphql/api
[2025-11-23 13:52:10] [WARNING] django.request:253 log_response Not Found: /api/gql
[2025-11-23 13:52:10] [WARNING] django.request:253 log_response Not Found: /swagger-ui.html
[2025-11-23 13:52:11] [WARNING] django.request:253 log_response Not Found: /swagger/index.html
[2025-11-23 13:52:11] [WARNING] django.request:253 log_response Not Found: /swagger/swagger-ui.html
[2025-11-23 13:52:12] [WARNING] django.request:253 log_response Not Found: /webjars/swagger-ui/index.html
[2025-11-23 13:52:13] [WARNING] django.request:253 log_response Not Found: /swagger.json
[2025-11-23 13:52:13] [WARNING] django.request:253 log_response Not Found: /swagger/v1/swagger.json
[2025-11-23 13:52:13] [WARNING] django.request:253 log_response Not Found: /v2/api-docs
[2025-11-23 13:52:14] [WARNING] django.request:253 log_response Not Found: /v3/api-docs
[2025-11-23 13:52:14] [WARNING] django.request:253 log_response Not Found: /api-docs/swagger.json
[2025-11-23 13:52:14] [WARNING] django.request:253 log_response Not Found: /api/swagger.json
[2025-11-23 13:52:14] [WARNING] django.request:253 log_response Not Found: /@vite/env
[2025-11-23 13:52:15] [WARNING] django.request:253 log_response Not Found: /actuator/env
[2025-11-23 13:52:15] [WARNING] django.request:253 log_response Not Found: /server
[2025-11-23 13:52:15] [WARNING] django.request:253 log_response Not Found: /.vscode/sftp.json
[2025-11-23 13:52:16] [WARNING] django.request:253 log_response Not Found: /about
[2025-11-23 13:52:16] [WARNING] django.request:253 log_response Not Found: /debug/default/view
[2025-11-23 13:52:16] [WARNING] django.request:253 log_response Not Found: /v2/_catalog
[2025-11-23 13:52:16] [WARNING] django.request:253 log_response Not Found: /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application
[2025-11-23 13:52:17] [WARNING] django.request:253 log_response Not Found: /server-status
[2025-11-23 13:52:17] [WARNING] django.request:253 log_response Not Found: /login.action
[2025-11-23 13:52:17] [WARNING] django.request:253 log_response Not Found: /_all_dbs
[2025-11-23 13:52:18] [WARNING] django.request:253 log_response Not Found: /.DS_Store
[2025-11-23 13:52:18] [WARNING] django.request:253 log_response Not Found: /.env
[2025-11-23 13:52:18] [WARNING] django.request:253 log_response Not Found: /.git/config
[2025-11-23 13:52:18] [WARNING] django.request:253 log_response Not Found: /s/230313e29313e2938313e2234313/_/;/META-INF/maven/com.atlassian.jira/jira-webapp-dist/pom.properties
[2025-11-23 13:52:19] [WARNING] django.request:253 log_response Not Found: /config.json
[2025-11-23 13:52:19] [WARNING] django.request:253 log_response Not Found: /telescope/requests
[2025-11-23 13:52:19] [WARNING] django.request:253 log_response Not Found: /info.php
[2025-11-23 14:18:30] [WARNING] django.request:253 log_response Not Found: /favicon.ico
[2025-11-23 14:18:30] [WARNING] django.request:253 log_response Not Found: /favicon.png
[2025-11-23 14:19:12] [WARNING] django.request:253 log_response Not Found: /favicon.ico
[2025-11-23 16:17:31] [WARNING] django.request:253 log_response Not Found: /notification/ping
[2025-11-23 16:27:16] [ERROR] seahub.repo_metadata.apis:99 get HTTPConnectionPool(host='seafile-md-server', port=8084): Max retries exceeded with url: /api/v1/base/c8152443-28ff-4756-a469-f2f8c4ee6439/query (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f9162f890a0>: Failed to resolve 'seafile-md-server' ([Errno -2] Name or service not known)"))
[2025-11-23 16:27:16] [ERROR] django.request:253 log_response Internal Server Error: /api/v2.1/repos/c8152443-28ff-4756-a469-f2f8c4ee6439/metadata/
[2025-11-23 16:37:04] [ERROR] seahub.repo_metadata.apis:99 get [Errno 500] {"error":"internal server error"}
[2025-11-23 16:37:04] [ERROR] django.request:253 log_response Internal Server Error: /api/v2.1/repos/c8152443-28ff-4756-a469-f2f8c4ee6439/metadata/
[2025-11-23 16:40:20] [ERROR] seahub.repo_metadata.apis:99 get [Errno 500] {"error":"internal server error"}
[2025-11-23 16:40:20] [ERROR] django.request:253 log_response Internal Server Error: /api/v2.1/repos/c8152443-28ff-4756-a469-f2f8c4ee6439/metadata/
[2025-11-23 16:40:28] [ERROR] seahub.repo_metadata.apis:99 get [Errno 500] {"error":"internal server error"}
[2025-11-23 16:40:28] [ERROR] django.request:253 log_response Internal Server Error: /api/v2.1/repos/c8152443-28ff-4756-a469-f2f8c4ee6439/metadata/
[2025-11-23 16:41:36] [ERROR] seahub.repo_metadata.apis:99 get [Errno 500] {"error":"internal server error"}
[2025-11-23 16:41:36] [ERROR] django.request:253 log_response Internal Server Error: /api/v2.1/repos/c8152443-28ff-4756-a469-f2f8c4ee6439/metadata/

These two definitely seem off.

docker compose logs seafile-md-server
docker compose logs seafile-md-server
seafile-md-server  | [md-server] [2025-11-23 16:36:11] [INFO] Created data links
seafile-md-server  | [md-server] [2025-11-23 16:36:14] [INFO] Database initialization completed
seafile-md-server  | [md-server] [2025-11-23 16:36:14] [INFO] Starting Metadata server
docker compose logs redis
docker compose logs redis
seafile-redis  | 10:C 23 Nov 2025 16:36:04.485 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
seafile-redis  | 10:C 23 Nov 2025 16:36:04.485 * Redis version=8.2.3, bits=64, commit=00000000, modified=1, pid=10, just started
seafile-redis  | 10:C 23 Nov 2025 16:36:04.485 * Configuration loaded
seafile-redis  | 10:M 23 Nov 2025 16:36:04.485 * monotonic clock: POSIX clock_gettime
seafile-redis  | 10:M 23 Nov 2025 16:36:04.536 * Running mode=standalone, port=6379.
seafile-redis  | 10:M 23 Nov 2025 16:36:04.710 * Server initialized
seafile-redis  | 10:M 23 Nov 2025 16:36:04.744 * Ready to accept connections tcp
seafile-redis  | 10:M 23 Nov 2025 16:41:05.081 * 100 changes in 300 seconds. Saving...
seafile-redis  | 10:M 23 Nov 2025 16:41:05.142 * Background saving started by pid 16
seafile-redis  | 16:C 23 Nov 2025 16:41:05.227 * DB saved on disk
seafile-redis  | 16:C 23 Nov 2025 16:41:05.227 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
seafile-redis  | 10:M 23 Nov 2025 16:41:05.242 * Background saving terminated with success
seafile-redis  | 10:M 23 Nov 2025 16:46:06.009 * 100 changes in 300 seconds. Saving...
seafile-redis  | 10:M 23 Nov 2025 16:46:06.009 * Background saving started by pid 17
seafile-redis  | 17:C 23 Nov 2025 16:46:06.061 * DB saved on disk
seafile-redis  | 17:C 23 Nov 2025 16:46:06.061 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
seafile-redis  | 10:M 23 Nov 2025 16:46:06.109 * Background saving terminated with success

These two look normal to me.

(part 2 of OP)

docker compose logs seafile
docker compose logs seafile
seafile  | *** Running /etc/my_init.d/01_create_data_links.sh...
seafile  | *** Booting runit daemon...
seafile  | *** Runit started as PID 17
seafile  | *** Running /scripts/enterpoint.sh...
seafile  | 2025-11-23 18:46:17 Waiting Nginx 
seafile  | 2025-11-23 18:46:17 Nginx ready 
seafile  | 2025-11-23 18:46:18 This is an idle script (infinite loop) to keep container running. 
seafile  | nginx: [warn] conflicting server name "" on 0.0.0.0:80, ignored
seafile  | nginx: [warn] conflicting server name "" on 0.0.0.0:80, ignored
seafile  | [2025-11-23 18:46:20] Skip running setup-seafile-mysql.py because there is existing seafile-data folder.
seafile  | [11/23/2025 18:46:20][upgrade]: The container was recreated, start fix the media symlinks
seafile  | mv: not replacing '/shared/seafile/seahub-data/avatars/default-non-register.jpg'
seafile  | mv: not replacing '/shared/seafile/seahub-data/avatars/default.png'
seafile  | mv: not replacing '/shared/seafile/seahub-data/avatars/groups'
seafile  | [11/23/2025 18:46:20][upgrade]: Done
seafile  | 
seafile  | Starting seafile server, please wait ...
seafile  | Seafile server started
seafile  | 
seafile  | Done.
seafile  | 
seafile  | Starting seahub at port 8000 ...
seafile  | 
seafile  | Seahub is started
seafile  | 
seafile  | Done.

I can’t remember if the ‘conflicting server name’ is new or not..

seafevents.log
[2025-11-23 16:35:24] [ERROR] seafevents.app.mq_handler:110 Failed to get event: Failed to read from socket: [Errno 104] Connection reset by peer
[2025-11-23 16:35:24] [ERROR] seafevents.app.mq_handler:110 Failed to get event: Failed to read from socket: [Errno 104] Connection reset by peer
[2025-11-23 16:35:24] [ERROR] seafevents.app.mq_handler:110 Failed to get event: Failed to read from socket: [Errno 104] Connection reset by peer
[2025-11-23 16:35:24] [ERROR] seafevents.app.mq_handler:110 Failed to get event: Failed to read from socket: [Errno 104] Connection reset by peer
[2025-11-23 16:36:45] [INFO] root:83 LDAP is not set, disable ldap sync.
[2025-11-23 16:36:45] [INFO] virus_scan:51 [virus_scan] scan_command option is not found in seafile.conf, disable virus scan.
[2025-11-23 16:36:45] [INFO] seafevents.app.mq_handler:127 Subscribe to channels: {'seaf_server.event', 'seahub.audit', 'seahub.stats', 'seaf_server.stats'}
[2025-11-23 16:36:45] [INFO] root:534 Start counting user activity info..
[2025-11-23 16:36:45] [INFO] root:240 Start counting traffic info..
[2025-11-23 16:36:45] [INFO] root:547 [UserActivityCounter] update 0 items.
[2025-11-23 16:36:45] [INFO] root:268 Traffic counter finished, total time: 0.0006120204925537109 seconds.
[2025-11-23 16:36:45] [INFO] root:23 Start file updates sender, interval = 300 sec
[2025-11-23 16:36:45] [WARNING] root:57 Can not start work weixin notice sender: it is not enabled!
[2025-11-23 16:36:45] [INFO] root:133 search indexer is started, interval = 600 sec
[2025-11-23 16:36:45] [INFO] root:56 seahub email sender is started, interval = 1800 sec
[2025-11-23 16:36:45] [WARNING] root:17 Can not start ldap syncer: it is not enabled!
[2025-11-23 16:36:45] [WARNING] root:18 Can not start virus scanner: it is not enabled!
[2025-11-23 16:36:45] [INFO] waitress:449 Serving on http://127.0.0.1:8889
[2025-11-23 16:36:45] [INFO] root:35 Start data statistics..
[2025-11-23 16:36:45] [WARNING] root:40 Can not start content scanner: it is not enabled!
[2025-11-23 16:36:45] [INFO] root:182 Start counting total storage..
[2025-11-23 16:36:45] [WARNING] root:46 Can not scan repo old files auto del days: it is not enabled!
[2025-11-23 16:36:45] [INFO] root:87 Start counting file operations..
[2025-11-23 16:36:45] [INFO] root:403 Start counting monthly traffic info..
[2025-11-23 16:36:45] [WARNING] root:33 Can not start face cluster updater: it is not enabled!
[2025-11-23 16:36:45] [INFO] seafevents.repo_metadata.index_master:32 metadata master event receive thread started
[2025-11-23 16:36:45] [INFO] seafevents.repo_metadata.slow_task_handler:53 slow_task_handler_thread_0 starting update metadata work
[2025-11-23 16:36:45] [INFO] seafevents.repo_metadata.slow_task_handler:53 slow_task_handler_thread_1 starting update metadata work
[2025-11-23 16:36:45] [INFO] seafevents.repo_metadata.slow_task_handler:53 slow_task_handler_thread_2 starting update metadata work
[2025-11-23 16:36:45] [INFO] seafevents.repo_metadata.index_master:48 metadata master starting listen
[2025-11-23 16:36:45] [WARNING] root:18 Can not start face cluster timer: please check you config!
[2025-11-23 16:36:45] [INFO] root:111 Start metric collect, interval = 15 sec
[2025-11-23 16:36:45] [INFO] root:115 Starting metric handler
[2025-11-23 16:36:45] [INFO] root:47 Starting handle redis channel
[2025-11-23 16:36:45] [WARNING] root:86 Can not start seasearch file index updater: it is not enabled!
[2025-11-23 16:36:45] [WARNING] root:87 Can not start seasearch wiki index updater: it is not enabled!
[2025-11-23 16:36:45] [INFO] root:115 search wiki indexer is started, interval = 600 sec
[2025-11-23 16:36:45] [INFO] root:213 Starting quota usage saver timer, interval = 30 sec
[2025-11-23 16:36:45] [INFO] root:217 Starting repo change collector
[2025-11-23 16:36:45] [INFO] root:162 Starting handle redis channel
[2025-11-23 16:36:45] [INFO] root:22 seahub quota alert email sender is started, interval = 86400 sec
[2025-11-23 16:36:45] [INFO] root:169 [FileOpsCounter] Finish counting file operations in 0.042609453201293945 seconds, 0 added, 0 deleted, 0 visited, 0 modified
[2025-11-23 16:36:45] [INFO] root:491 Monthly traffic counter finished, update 2 user items, 1 org items, total time: 0.05659151077270508 seconds.
[2025-11-23 16:36:45] [INFO] root:225 [TotalStorageCounter] Finish counting total storage in 0.10908389091491699 seconds.
[2025-11-23 16:46:45] [INFO] root:141 starts to index wiki files
[2025-11-23 16:46:45] [INFO] root:160 starts to index files
seafhttp.error.log
2025/11/23 16:36:15 [error] 64#64: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.8, server: , request: "GET /seafhttp/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439 HTTP/1.1", upstream: "http://127.0.0.1:8082/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439", host: "seafile.<MYDOMAINHERE>"
2025/11/23 16:36:15 [error] 64#64: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.8, server: , request: "POST /seafhttp/repo/locked-files HTTP/1.1", upstream: "http://127.0.0.1:8082/repo/locked-files", host: "seafile.<MYDOMAINHERE>"
2025/11/23 16:36:16 [error] 64#64: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.8, server: , request: "GET /seafhttp/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439 HTTP/1.1", upstream: "http://127.0.0.1:8082/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439", host: "seafile.<MYDOMAINHERE>"
2025/11/23 16:36:17 [error] 64#64: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.8, server: , request: "GET /seafhttp/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439 HTTP/1.1", upstream: "http://127.0.0.1:8082/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439", host: "seafile.<MYDOMAINHERE>"
2025/11/23 16:36:18 [error] 64#64: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.8, server: , request: "GET /seafhttp/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439 HTTP/1.1", upstream: "http://127.0.0.1:8082/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439", host: "seafile.<MYDOMAINHERE>"
2025/11/23 16:36:19 [error] 64#64: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.8, server: , request: "GET /seafhttp/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439 HTTP/1.1", upstream: "http://127.0.0.1:8082/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439", host: "seafile.<MYDOMAINHERE>"
2025/11/23 16:36:20 [error] 64#64: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.8, server: , request: "GET /seafhttp/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439 HTTP/1.1", upstream: "http://127.0.0.1:8082/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439", host: "seafile.<MYDOMAINHERE>"
2025/11/23 16:36:21 [error] 64#64: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.8, server: , request: "GET /seafhttp/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439 HTTP/1.1", upstream: "http://127.0.0.1:8082/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439", host: "seafile.<MYDOMAINHERE>"
2025/11/23 16:36:22 [error] 64#64: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.8, server: , request: "GET /seafhttp/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439 HTTP/1.1", upstream: "http://127.0.0.1:8082/accessible-repos/?repo_id=c8152443-28ff-4756-a469-f2f8c4ee6439", host: "seafile.<MYDOMAINHERE>"

I suppose there’s maybe something wrong with my redis cache, but there is nothing in any of the manual on how to debug such issues. There is more logs too, but I don’t know which ones are the most relevant, so let me know what else would be best to add.

When loading a base, the md-server tries to recover latest data from the last snapshot and the operation logs. It seems, due to server crash, these two set of data become inconsistent with each other. That causes the error in the log.

To temporarily skip this problem, you can set this environment variable in .env:

MD_OPLOG_IGNORE_ERRS = true

This option ask md-server to ignore read/write errors for the operation log. Note that turning on this option may cause some data lose (which is expected due to server crash). It’s recommended to turn it off again after the data succeeds to load again.