OVH object storage - error when uploading

Romain_P · October 29, 2017, 3:21pm

Hi everyone,

I am trying to set up a server on a OVH VPS, using OVH object storage as storage backend.
Here is my seafile.conf:
[fileserver]
port = 8082[database]
type = mysql
host = 127.0.0.1
port = 3306
user = seafile
password = *
db_name = seafile_db
connection_charset = utf8

[block_backend]
name = swift
tenant = *
user_name = *
password = *
container = seafile-blocks
auth_host = auth.cloud.ovh.net
auth_ver = v2.0
region = GRA3
use_https = true

[commit_object_backend]
name = swift
tenant = *
user_name = *
password = *
container = seafile-commits
auth_host = auth.cloud.ovh.net
auth_ver = v2.0
region = GRA3
use_https = true

[fs_object_backend]
name = swift
tenant = *
user_name = *
password = *
container = seafile-fs
auth_host = auth.cloud.ovh.net
auth_ver = v2.0
region = GRA3
use_https = true

I can connect to the web interface, create libraries, create new files. The containers seafile-fs and seafile-commits contain some objects. Yet, I cannot upload anything to the server (Internal Server error)…
Here is the tail of my seafile.log
[10/29/2017 02:52:47 PM] ../common/fs-mgr.c(554): failed to commit chunk 29d10aba005db15ba4567cb4134496a8a27aa886. [10/29/2017 02:52:47 PM] repo-op.c(1100): failed to index blocks[10/29/2017 02:53:02 PM] ../common/block-backend-swift.c(496): Put block 29d10aba005db15ba4567cb4134496a8a27aa886 error 404. Response: ``<html><h1>Not Found</h1><p>The resource could not be found.</p></html>`` [10/29/2017 02:53:02 PM] ../common/fs-mgr.c(554): failed to commit chunk 29d10aba005db15ba4567cb4134496a8a27aa886. [10/29/2017 02:53:02 PM] repo-op.c(1100): failed to index blocks[10/29/2017 03:30:21 PM] filelock-mgr.c(917): Cleaning expired file locks.

Would you have any idea of what I am doing wrong?
Many thanks,
Romain

omascia · October 30, 2017, 5:23am

If this “Put block error” from a source file named ‘block-backend-swift’ is what I suspect it is (https://developer.openstack.org/api-ref/object-store/index.html#create-or-replace-object) then a 404 error probably means the container ‘Seafile-blocks’ configured in the seafile.conf file was not found.

omascia · October 30, 2017, 11:38am

I since have installed a similar setup, and reproduce exactly the same issue. Simple install. Don’t start it. Update the seafile.conf for swift. ./seafile.sh start. Check logs/seafile.log. There are already similar errors to yours. And these are the very first lines in the log file. This is the auto-creation of the default content of the “My Library” for the admin user which fails.

[10/30/17 11:33:32] http-server.c(194): fileserver: worker_threads = 10
[10/30/17 11:33:32] http-server.c(209): fileserver: fixed_block_size = 8388608
[10/30/17 11:33:32] http-server.c(224): fileserver: web_token_expire_time = 3600
[10/30/17 11:33:32] http-server.c(239): fileserver: max_upload_size = -1
[10/30/17 11:33:32] http-server.c(254): fileserver: max_download_dir_size = 104857600
[10/30/17 11:33:32] http-server.c(269): fileserver: max_indexing_threads = 1
[10/30/2017 11:33:32 AM] ../common/mq-mgr.c(61): [mq client] mq cilent is started
[10/30/2017 11:33:32 AM] filelock-mgr.c(924): Cleaning expired file locks.
[10/30/2017 11:33:33 AM] size-sched.c(103): Repo size compute queue size is 0
[10/30/2017 11:33:35 AM] ../common/block-backend-swift.c(496): Put block 8663a70ef30a5987b440a621483af2044bae1e0a error 404. Response:
<html><h1>Not Found</h1><p>The resource could not be found.</p></html>
[10/30/2017 11:33:35 AM] ../common/fs-mgr.c(554): failed to commit chunk 8663a70ef30a5987b440a621483af2044bae1e0a.
[10/30/2017 11:33:35 AM] repo-op.c(705): failed to index blocks[10/30/2017 11:33:35 AM] seafile-session.c(448): Failed to add template file /opt/seafile/seafile-data/library-template/seafile-tutorial.doc.

Romain_P · October 30, 2017, 12:17pm

Well thank you for having tried! So, do you believe the problem is independent of my setup? Do you have any idea of where the problem is located and what could be a solution? I admit to be quite a beginner…
Many thanks again.

omascia · October 30, 2017, 12:23pm

Well… I have no insider experience with Seafile (especially on OpenStack Swift). What I can say is that it probably is not an authentication issue : I have set it up with 3 containers as recommend and after my test I do have one object stored in the commits container. But nothing in the blocks container.

A this stage, I have no clue on what goes wrong with the blocks container. Or if this is read error while attempting to re-read the commit that got stored in the commits container.

I don’t think this feature (Swift support) is built in the community edition, so it isn’t a path to review that part of the source code and get a better understanding of what’s going on.

I have a paid commercial license for the Pro edition, I will contact Seafile support with a pointer to this forum thread.

Jonathan · November 1, 2017, 3:20am

@Romain_P I noticed that you configured “region” option. Would this affect the resource location?

Marc_Cizeron · November 6, 2017, 9:05am

Hi,

I encounter exactly the same issue. Did you find a fix ?

Jonathan · November 6, 2017, 1:44pm

Nope. It looks like a configuration issue. The code works with other Swift installations.

Marc_Cizeron · November 8, 2017, 2:50pm

Hi, thank’s for your answer. So i check all my configuration, i think i havn’t got any problems with the credential, since i am authenticate.

Here my seafile.conf file:

[block_backend]
name = swift
tenant = *
user_name = *
password = *
container = seafile-blocks
auth_host = auth.cloud.ovh.net
auth_ver = v2.0
region = GRA1
memcached_options = --SERVER=127.0.0.1:11211 --POOL-MIN=10 --POOL-MAX=100
use_https = true

[commit_object_backend]
name = swift
tenant = *
user_name = *
password = *
container = seafile-commits
auth_host = auth.cloud.ovh.net
auth_ver = v2.0
region = GRA1
memcached_options = --SERVER=127.0.0.1:11211 --POOL-MIN=10 --POOL-MAX=100
use_https = true

[fs_object_backend]
name = swift
tenant = *
user_name = *
password = *
container = /seafile-fs/
auth_host = auth.cloud.ovh.net
auth_ver = v2.0
region = GRA1
memcached_options = --SERVER=127.0.0.1:11211 --POOL-MIN=10 --POOL-MAX=100
use_https = true

And then i have this error :

[11/08/2017 02:48:52 PM] …/common/obj-backend-swift.c(319): Put object
553a62ec237003b92872e4f5a811c54ef7d0fc38 error 404. Response: < html>< h1>Not Found< /h1>< p>The resource could not be found.< /p>< /html>

Wich log file should i inspect ? I there an obvious mistake in my configuration ?

Thank you very much for any help.

Jonathan · November 9, 2017, 9:56am

Have you tried to remove the region option?

Marc_Cizeron · November 9, 2017, 3:07pm

Yes, I did. I ran into the same error a little bit defferent :

[11/09/2017 03:04:46 PM] …/common/obj-backend-swift.c(319): Put object e8b030de1c83ff3d000a3a6bcef2ac690bbdbf37 >error 404. Response:
< html>< h1>Not Found< /h1>< p>The resource could not be found.< /p>< /html>
[11/09/2017 03:04:46 PM] repo-mgr.c(5890): Failed to add commit.
[11/09/2017 03:04:46 PM] seafile-session.c(502): Failed to create system default repo.
[11/09/2017 03:05:01 PM] …/common/fs-mgr.c(1893): [fs mgr] Failed to read dir >73fe86f8a264a6bd87f4c059be6d260fdb454f42.

I am totaly lost. I think OVH have change something, because this configuration works well until 18/09/2017, then after this date I encountered this issue. During this period, i didn’t chnage anything on my server (debian 8, seafile server pro for one user, or package updates).

Thank you very munch for helping.

GNUletik · November 9, 2017, 3:46pm

Same problem here on OVH Object Storage. I had a working seafile but since a few days, I got a 404 error from the swift library so all my files are not available anymore.
I tried to do a ./seaf-fsck --repair but all my files were replaced with 0 bytes files.

I will investigate more and give any updates here.

shoeper · November 9, 2017, 9:41pm

Did you get in touch with ovh to find out what changed and to let them know it causes issues?

GNUletik · November 10, 2017, 1:40pm

Hey there,

I was in contact with the team in charge of Object Storage and we realized that Seafile was trying to reach different regions even if the region is set in the seafile.conf file. This error then sometimes result in 404 errors as the container is not found.

Also, this error appeared recently because they added a new region in the catalog which change its order.
If no region is set, the client takes the first region in the catalog.

So it seems that a some point the region variable is not used/set in Seafile.

As the version using swift object storage is not open source, I can’t fix it.
I’ve contacted Seafile and will let you know what’s up.

shoeper · November 10, 2017, 8:47pm

Great. I think this will definitely be helpful to solve the issue.

GNUletik · November 11, 2017, 12:16am

Here is a screenshot from mitmproxy of what’s happenning when uploading a file with a fresh install.

For the seafile-fs and seafile-commits containers the right region is used but not for seafile-blocks.
Even if the region is set to the same value for the three containers in the seafile.conf file.

Jonathan · November 11, 2017, 2:33am

Thanks for the information. We’ll check the code and get back to this.

GNUletik · November 16, 2017, 9:06am

@Jonathan Any news on this ?

omascia · November 19, 2017, 8:19am

Any news on this ?

Marc_Cizeron · November 22, 2017, 7:38am

@Jonathan Any news on this ?