AWS v4 signatures not working with S3 compatible backend (wasabi)

Hi!

I’ve been troubleshooting this for hours now and can’t figure out why this isn’t working with wasabi s3 backend. The same config works just fine with AWS S3

I have the following three buckets in wasabi in the region eu-central-1 (note, at wasabi this is in Amsterdam, not Frankfurt):

  • xie8do-commit-objects
  • xie8do-fs-objects
  • xie8do-block-objects

I want to use AWS signature version 4, but when setting this

[commit_object_backend]
name= s3
bucket = xie8do-commit-objects
key_id = my-key-id
key = my-secret-key
host = s3.eu-central-1.wasabisys.com
use_v4_signature = true
aws_region = eu-central-1
path_style_request = true
memcached_options = --SERVER=memcached --POOL-MIN=10 --POOL-MAX=100
use_https = true

[fs_object_backend]
same as above except fs bucket name

[block_backend]
same as above except block bucket name

And also do in ~/.boto

[s3]
use-sigv4 = True

the log shows

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>JAHDSALGFLJAREDACTED</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256&#xA;&#xA;20200212/eu-central-1/s3/aws4_request&#xA;45d0ec0b950c9ac55a8b42369a44dcef392ba18ea56da56923ece407a5a96c76</StringToSign><SignatureProvided>b8a516cb34937033ce224e7a32e51c12ab7f823baba9f52f2174940dd5ef53fd</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 a a 32 30 32 30 30 32 31 32 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 a 34 35 64 30 65 63 30 62 39 35 30 63 39 61 63 35 35 61 38 62 34 32 33 36 39 61 34 34 64 63 65 66 33 39 32 62 61 31 38 65 61 35 36 64 61 35 36 39 32 33 65 63 65 34 30 37 61 35 61 39 36 63 37 36</StringToSignBytes><CanonicalRequest>PUT&#xA;/034b94fb-2fb6-4bbc-b86d-590d8b9e0786/1160a2cfa2bfad637db0e26a9ed718828e016fdc&#xA;&#xA;host:xie8do-commit-objects.s3.eu-central-1.wasabisys.com&#xA;x-amz-content-sha256:1d5e9fe4dad7f0d131574e979eca1472e215aa43795f0ece39ab72edab57128f&#xA;&#xA;host;x-amz-content-sha256&#xA;1d5e9fe4dad7f0d131574e979eca1472e215aa43795f0ece39ab72edab57128f</CanonicalRequest><CanonicalRequestBytes>50 55 54 a 2f 30 33 34 62 39 34 66 62 2d 32 66 62 36 2d 34 62 62 63 2d 62 38 36 64 2d 35 39 30 64 38 62 39 65 30 37 38 36 2f 31 31 36 30 61 32 63 66 61 32 62 66 61 64 36 33 37 64 62 30 65 32 36 61 39 65 64 37 31 38 38 32 38 65 30 31 36 66 64 63 a a 68 6f 73 74 3a 63 68 6f 6f 34 69 78 6f 6f 39 6f 6f 2d 63 6f 6d 6d 69 74 2d 6f 62 6a 65 63 74 73 2e 73 33 2e 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2e 77 61 73 61 62 69 73 79 73 2e 63 6f 6d a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 31 64 35 65 39 66 65 34 64 61 64 37 66 30 64 31 33 31 35 37 34 65 39 37 39 65 63 61 31 34 37 32 65 32 31 35 61 61 34 33 37 39 35 66 30 65 63 65 33 39 61 62 37 32 65 64 61 62 35 37 31 32 38 66 a a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 a 31 64 35 65 39 66 65 34 64 61 64 37 66 30 64 31 33 31 35 37 34 65 39 37 39 65 63 61 31 34 37 32 65 32 31 35 61 61 34 33 37 39 35 66 30 65 63 65 33 39 61 62 37 32 65 64 61 62 35 37 31 32 38 66</CanonicalRequestBytes><RequestId>5D92B42A52124773</RequestId><HostId>nmG9t6bD75KoJSb2Ddkum01+O/F0v/TXznfkXaQwU28L64roIfLHCgKNdLsuUiXhK9zVAJEIqVXY</HostId></Error>

If I change in seafile.conf

use_v4_signature = false

and in ~/.boto

[s3]
use-sigv4 = False

it works fine.

taking out path_style_request = true did not make a difference.

I could successfully put files into the buckets using v4 signatures by following instructions found in the wasabi knowledge base. I would post the link to that article, but it looks like I can’t. Just google/duckduckgo How do I use AWS Signature Version 4 with Wasabi? and you’ll find it.

So from wasabi’s side this should work.

I’m wondering where and how does the v4 signature get calculated in seafile pro? is it possible that seafile is getting confused with the wasabi region eu-central-1 not being the same as eu-central-1 in AWS? Is it possible that seafile is calculating the signature based on the wrong region?

I tried taking out aws_region from seafile.conf but then seafile server can’t be started.

Any help would be appreciated!

edit:
Fresh/New setup of Seafile Pro 7.0.13

@daniel.pan @Jonathan I’m still stuck with this. Would be great to hear if this is a known problem or confirm that this is due to a bug or something else:-)

I’m afraid we don’t have resource to look at the problem yet. If more people need to work with wasabi using v4 signatures, we will investigate the problem in the future.

Hi @daniel.pan, any chance your team could look into this? Is there anything I can do to help make it easier for you to tackle this?

Hello,
I have the same issue with Seafile Pro 7.1.5 and Wasabi - I also tried Scaleway S3-compatible storage object with the same error “SignatureDoesNotMatch”. I verified their implementation with s3cmd and have been able to connect, list and upload data to buckets.

I did some digging and as far as I have been able to find out you are not always sending x-amz-date signature element. Excerpt from the seafile.log file:

../common/s3-client.c(624): Authorization: AWS4-HMAC-SHA256 Credential=XXXREPLACEDXXX/20200719/nl-ams/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256,Signature=08b0225fa37367e7197b6d47b80fc3febff9XXXREPLACEDbecb3ac25718b03a5d876
../common/s3-client.c(624): x-amz-content-sha256: ba34b37f9eef973dbeea0XXREPLACEDXX56e2a7c3ce230903e0650ff70

The document at Amazon says it is required: https://docs.aws.amazon.com/general/latest/gr/sigv4_elements.html

Also, the headers (or signature elements) must be sorted alphabetically (by lowercase) - this is how the Amazon calculates signature.

Attn: @daniel.pan