Will an AWS/S3 cluster deployment increase upload speed of large files?


#1

Hello everyone,

Simple question: will deploying on a cluster increase maximum upload speed for large files? We currently have an S3 backed SeaFile server and the best upload I can achieve is 80MB/S (this is actually the same as when I tested nextcloud, so not unique to SeaFile). The place I uploaded from, however, is a university institute with up to 500MB/S upload speed (tested using google speed test).
If instead we deployed a SeaFile cluster in AWS using S3 as the back end, would it be possible to have the block uploads be split between different front end servers to further parallelize the upload?

The diagram in the below page implies that no, it won’t, but I thought it would be worth asking!
https://manual.seafile.com/deploy_pro/deploy_in_a_cluster.html

EDIT: forgot to state, that this is to upload multi-gigabyte files.

Kind regards,
Pedro Almada


#2

I think it should be possible when the load balancer does not redirect all requests from the same client to the same frontend node.

Another question is whether the client is fast enough


#3

Cheers, that’s promising. However, is that true for very large (>4GB) files?


#4

The client splits the file up into blocks and uploads these. They are a few MiB each and uploaded in parallel (think 3 at a time).


#5

I see, so each client will talk to several front ends simultaneously for one large file upload?


#6

It depends on how your load balancing works. When your load balancer sends each request to the next server it should communicate with up to three front end nodes simultaneously.


#7

Cheers @shoeper, you’ve convinced me to try the cluster installation!