Migrate from local Storage to S3 using migration script

Seafile Server
1

Hi there,

Server Version 6.0.12 Pro

i wanted to move my storage backend from local storage to S3
i copied the config file and i added the following data there:

[commit_object_backend]
name = s3
use_v4_signature = true
aws_region = eu-west-1
use_https = true
bucket = sf-commit-objects
key_id = XXX
key = XXX
memcached_options = --SERVER=localhost --POOL-MIN=10 --POOL-MAX=100

[fs_object_backend]
name = s3
use_v4_signature = true
aws_region = eu-west-1
use_https = true
bucket = sf-fs-objects
key_id = XXX
key = XXX
memcached_options = --SERVER=localhost --POOL-MIN=10 --POOL-MAX=100

[block_backend]
name = s3
use_v4_signature = true
aws_region = eu-west-1
use_https = true
bucket = sf-block-objects
key_id = XXX
key = XXX
memcached_options = --SERVER=localhost --POOL-MIN=10 --POOL-MAX=100

all good. i used the migration script and i ran: migrate.sh config/seafile-s3.conf
the script ran successfully (completed blocks, commits and fs). what i have to do next ? i thought the script is moving the data to the S3 buckets. I have to do this manually ?

Thanks !

Best, Andre

2

@Jonathan - can you help here ?

Thanks ! Best, Andre

3

The script should actually move the data to s3.

4

But this was not working then, the response from the migration script was successful (or completed)
The script ran ~15 min. for currently 700GB of Data. My assumption is that the script is just updating the database or ā€œpreparesā€ the filesystem and i have to move the data to S3, but would be nice if someone can confirm this, because the sync of the folder ā€œdata/storage/fsā€ takes ages to complete. (due to the thousands of subfolders)

5

After running the script and using the new config file, you should be able to see your data in your s3 buckets and have Seafile running again. Whatā€™s the problem you see?

6

Pretty much looks like the script aborts after 15 mins in his case saying it has finished (which is unlikely with 700 GB).

7

Right, it took ~15 min. and the script said ā€œcompletedā€ for commits, fs and blocks. But the data wasnā€™t uploaded to S3. So not sure what i should do now.
should i sync the folders manually to the s3 buckets and then use the new config file ?

8

ok, uploading the folder to the s3 buckets is not really working :slight_smile:

That is the result:

Bildschirmfoto 2017-05-08 um 19.28.16.png3502Ɨ1752 604 KB

1 Like
9

Pretty much looks like there has been some error case that wasnā€™t handled and thus not all data has been uploaded to s3.

10

i uploaded (synced) all files (checked multiple times) from the folder

  • data/storage/
    ā€“ blocks
    ā€“ commits
    ā€“ fs

to the respective S3 buckets and changed the config file. So all files should be there. Do i have to do sth. else ?
iā€™m afraid to start the server with the old config after the migration script confirmed that the migration is completed. I guess the script did sth. in the 15 min. :wink:

11

Afaik naming of the files is being done different for s3. So just uploading them wonā€™t work.

12

Just wondering if there has been any update on this issue. Iā€™ve tried migrating my ~200 GB of data to S3 using the above script - it completes in about 30 seconds with no errors but no data is moved to S3.

13

Weā€™ve tested the script internally with S3. But we actually havenā€™t officially released the script, thus no manual is available for it right now. Can you tell how do you run the script?

14

Has there been any progress with this issue? Iā€™m running in to the same issue and not seeing any glaring problems that would cause the migration to fail. Iā€™m seeing a significant amount of network traffic on the system while the script is running, though at the end (mine is only running for 2 minutes, with a 520G repo, though at the end (with no returned errors) the buckets are empty.

Iā€™ve been in the seafile-server-latest running ./migrate.sh seafile.s3.conf
Iā€™ve also tried the above, but specifying the full path to the conf file.

Iā€™ve also changed to the data folder and then executed the full path of migrate.sh (with and without the full path to the conf file).

Results are always the same, ~2 minutes of execution, no errors returned, and empty S3 buckets.

15

Is your config file named ā€˜seafile-s3.confā€™? you should rename it to seafile.conf under config/ , then run ./migrate.sh config , give it a try.

16

Official document is added for this migration script: https://manual.seafile.com/deploy_pro/migrate.html

1 Like
17

Sorry for necropost, but this issue nearly drives me mad. :frowning:

I follow the official manual to migrate data from local storage to a S3-compatible Object Storage.

When I use the dir path of the new seafile.conf, the output is:
image

When I use the full path of the new seafile.conf, the output is:

image
image755Ɨ246 6.5 KB

In both situation the script runs well with no errors, but the buckets are always empty.

Many thanks for any reply !

18

I would add your temporary seafile.conf in /opt directory (just like their docs show) since Seafile migration script has hardcoded path based on my last experience.

19

Hey. Working actually with version 11.0.6 and getting the same behaviour when iĀ“m trying to migrate data to S3.
S3 config is imho working. IĀ“ve setup a test instance and without data i can use the S3 config and create new libs and save data.
But the migrate.sh script does not copy the data.
Already tried to save it in /opt/ and run it with folder and complete path.
It is a docker setup.

Any hints?

20

Can anybody answer it?
I have the same problem. Docker installation needs to migrate to S3 API compatible storage. Seafile v11.0.17, the script takes just 5 seconds to output ā€œDone.ā€ and thatā€™s it. Object storage is empty. Using S3 v2 API without e2e encryption.