On-the-fly creation of zip files (streaming instead of precomputing)

Hi,

As far as I know Seafile first creates an on-disk zip file before transmitting it. Depending on the number and size of the chosen files and of course the performance of the server this can be very time-consuming and may even lead to timeouts on slower servers. Here’s my wish: implement on-the-fly creation of zipfiles without writing anything or at least start streaming immediately.

I’d even say that most users don’t particularly care about compression, zip download is mostly a means to download multiple files in one go. So if this feature were easier to implement without compression it could still be an interesting feature!

4 Likes

+1 It would be nice to see a solution here! Many customers of us complain about this!

The main problem now is that when the zip takes a lot of time, the web UI does not response.

In version 6.0, the zip task will be performed in the background and the Web UI will show a progress. Google Drive works as this too. It should satisfy most users.

6 Likes

+1 for direct download without compression of any directory. I just tried to click the zip-Button multiple times - each time a new compression task is started and a new temporary file is generated without cancelling the running task or even better show the status of the running task an reuse it. Looks like a huge waste of ressources.
Please just use the zip as a container without any compression and serve it on-the-fly. I think this will be faster, especially on low end hardware.

2 Likes

This is currently the issue. Afaik there is a bug (which is hard to solve) in a used library which prevents usage of streaming a zip on the fly.

I just want to make some noise and revive this feature request. Don’t know if serving zip archives on-the-fly is still buggy and hard to implement but it would be great if zipping files could be done with multiple threads (see pigz as a parallel gzip implementation for example). Right now it’s still done by one thread.

2 Likes

To the developers:
I cannot imagine why it is hard to solve… You need to create a zip stream with no compression that is not seekable. Then the zip header should be written (directly to the browsers stream) and after that one file after another is written with its headers to the stream. I implemented this already some time ago in dotnet core. It’s not about streaming the files itself directly from the library but read one file from the library and put the contents to the zip stream that is transported to the browser client.
Does this help? https://pypi.org/project/zipstream-new/

This is on the roadmap for Seafile 9 (https://www.seafile.com/en/roadmap/). Looks like the plan is to rewrite a component in Go and to also implement zip streaming in that pass. I’d expect a positive outcome regarding security as well.

3 Likes