Downloading a folder as one zip without holding it in memory

Press Download on a folder in Nemi and the browser starts saving a zip at once, whether the folder holds three photos or two hundred gigabytes of video. There is no wait while an archive is built somewhere first. The zip is built while you download it, one file at a time, and no part of it is ever stored.

This post explains how that works, why the server's memory stays flat no matter how big the folder is, what happens when storage hiccups in the middle of a ten gigabyte file, and the two honest costs of doing it this way.

Why not build the zip first and then send it?

Building first is the obvious design, and it has three problems that all grow with the folder. You wait for the whole archive before the first byte arrives. Somewhere has to hold the finished archive, in memory or on disk, and for a large folder that is the same size as the folder. And that space is spent again for every person who downloads it.

A zip file does not need to be built that way. Each file in it is written as a small header followed by its contents, one after another, and the table of contents comes at the very end. So an archive can be produced front to back, as a stream, and sent out as it is produced. Nemi reads one file from storage, pushes it through the zip writer and out to your browser, then moves to the next. Only one file is open at a time.

The streamed process used 95 MiB at its peak for a 256 MiB folder and 96 MiB for a 4 GiB one, sixteen times bigger. About 68 MiB of that is the runtime before the first file is read; the zip adds a small, roughly fixed amount on top, whatever the size of the folder.

What keeps the memory flat?

Streaming alone is not enough. If storage delivers a file faster than your connection can take it, the difference has to wait somewhere, and on a slow line “somewhere” would quietly become the whole folder again. So the pipe is driven from the browser's end. When the outgoing buffer is full, the zip writer is paused, which stops it pulling from the file, which stops the read from storage. When your browser asks for more, everything resumes. The speed of the slowest link sets the speed of the whole chain, and nothing piles up in between.

The same wiring runs the other way when you cancel. Closing the download aborts the pipeline, which stops the read from storage, so a cancelled download does not keep a server busy fetching files nobody will receive.

Why the browser's download manager, not the page?

A web page can download a file itself, by fetching it and saving the result, but then the page has to hold what it fetched, and a tab cannot hold two hundred gigabytes. The browser's own download manager has no such problem: it writes to disk as bytes arrive, shows progress in the place you expect, and keeps going if you navigate away from Nemi.

The catch is that a download the browser starts is a plain link, and a plain link cannot carry a list of three hundred files. So the page sends the selection first, Nemi checks access and resolves every folder into its files, and gives back a short ticket. The link handed to the browser carries only that ticket. Because the list is already resolved, the zip starts streaming the instant the browser asks. A ticket stays valid for 15 minutes and works only for the person who asked for it.

What happens when storage hiccups halfway through a file?

A folder download can be hundreds of separate reads from storage in a row, and over enough reads, one of them will fail. If a failed read ended the download, a large folder would be a gamble. Instead, Nemi remembers exactly how many bytes of the current file it has already written into the zip. When a read breaks, it waits a moment and asks storage for the same file again, starting from that byte. The zip never notices: the bytes after the break follow the bytes before it, and your browser sees nothing but a short pause.

The retry counter only gives up when the retries stop getting anywhere. Every time a resumed read delivers new bytes, the count starts again, so a long file on a shaky day can survive many hiccups, as long as each one is followed by progress.

And when a file really cannot be read, the download still does not fail. If it cannot be opened at all after 3 attempts, it is left out. If storage goes away in the middle of it, the file is closed where it stopped so the archive stays valid. Either way a small text file named after it, ending in .[ERROR].txt, goes into the zip saying it could not be fully downloaded, and the rest of the folder follows. One bad file never costs you the other two hundred.

Storage is allowed to stumble. Your download should never have to know.

Does the zip make my files smaller?

Usually not by much, and that is by design. Most of what people keep in a folder, photos, video, PDFs, other archives, is already compressed, and compressing it again spends processor time to save almost nothing. Nemi therefore uses the lightest setting the format has, deflate level 1. Our converter follows the same reasoning when it packs photos and videos into an archive. A folder of text files would come out somewhat bigger than with a heavier setting, and in exchange the server spends little time per byte, which keeps a big download moving at the speed of your connection.

What are the costs of streaming a zip?

Your browser cannot show a percentage. A zip built on the fly does not know its final size when it starts, so the browser counts bytes received instead of drawing a bar. Knowing the size up front would mean building the archive first, which is everything this design avoids.

Your own download cannot be resumed. The resume described above happens between Nemi and storage. Between Nemi and you there is one continuous stream, and if your connection drops or your laptop goes to sleep, the download has to start again. For very large folders, keep the machine awake and plugged in, or download the biggest files on their own: a single file is served directly from storage like any other download. Uploads are the other way round: they go up in parts, so an interrupted upload can carry on where it stopped.

The same streaming is behind the Download all button on a share link, which is how the person you sent a folder to gets it in one go; see Files for the rest of that side, and our guide to sending large files securely for the settings on the link. The customer version of this page is Downloading files and zips in the help centre.