How a 10 GB upload survives a closed tab

Sending a big file through a browser has one failure that everybody has met. The bar is at ninety percent, the wifi drops for a second, and the upload starts again from zero. For a product whose whole job is moving large files, that is the most expensive thing that can go wrong, so Nemi does not upload a large file in one piece.

This post is about what it does instead: how a 10 GB file is cut into parts, why a dropped connection costs only the parts in flight rather than the whole file, how an upload picks up where it stopped after you close the tab, and what happens to the parts of an upload that is never finished. Every number below is a constant in the uploader.

Why does a large upload start again from zero?

Because an ordinary upload is a single HTTP request. The browser opens one connection and streams the file down it, and the server either receives the whole body or it receives nothing it can use. There is no standard way to say “carry on from byte 9,123,456,789” on a plain request, so when the connection breaks, the only recovery is to send it all again. On a fast office line that is annoying. On hotel wifi or a train it means a large file may never arrive at all.

Storage services solved this a long time ago with multipart uploads. You announce an upload, send the file as numbered parts in any order, and then ask storage to stitch them into one object. Each part is its own small request. If one fails, you send that one again. Nemi uses exactly this, from the browser, straight to storage.

How does Nemi split a large file?

Anything from 32 MB upwards is sent in parts of 16 MB. Below that a single request is simply better: one request, one signature, nothing to stitch. A 10 GB file becomes 640 parts. The browser keeps 3 of them in flight at once, and each one goes directly to storage with its own signed address, so the bytes never queue behind a server of ours.

Sixteen megabytes is a compromise between two costs. Storage requires every part except the last to be at least 5 MB, so parts cannot be tiny. But the bigger a part, the more a failure throws away, and on a bad line you want a failure to be a cheap retry. There is one more constraint at the other end: storage accepts at most 10,000 parts per upload. Business allows a single file of 200 GB, which at 16 MB would need 12,800 parts, so for files that large the part size grows instead (21 MB for a 200 GB file, 9,753 parts). The server works out the part size and tells the browser, so the two can never disagree about where part seven starts.

What happens when you close the tab?

The transfer stops, because a page that is closed cannot send anything. But the parts that already arrived stay on storage, and before the first part went out, the browser wrote a small note on your device: which upload this file belongs to, and how it was cut up. That note is kept in the browser's own storage for 5 hours.

A browser cannot reopen a file on its own; it has no path to it, only the file you hand it. So resuming is something you do: drag the same file into the same folder again. Nemi recognises it by its name, its size and the time it was last modified, finds the note, and carries on with the same upload rather than starting a new one. That works after a reload, a closed tab, or a browser restart.

Why does Nemi ask storage, not the browser?

The obvious way to resume is for the browser to remember which parts it sent and skip them next time. Nemi deliberately does not trust that list. A tab that was killed in the middle of a request cannot know whether its last part landed. If its memory were wrong in the optimistic direction, the missing part would never be sent, and the finished file would have a hole in it that nothing would notice: the part count would look right and so would the total.

The note on your device only says which upload to continue. Storage says what is in it.

So every attempt, including the first retry after a blink, starts by asking storage which parts exist, and sends everything else. There is one weak point in recognising a file by name, size and date: two different files that match on all three would look like the same file. Such a match is rare, which is one reason the note expires after 5 hours.

How much does a resume actually save?

At most, an interruption costs the parts that were in flight at that moment: 3 parts of 16 MB, or 48 MB. A single request loses everything it had sent. The chart is that arithmetic for an upload interrupted at ninety percent, which is where it hurts most.

We have not put a speed figure on any of this, and that is deliberate. Sending three parts at once can help on some connections and makes no difference on others, and a number measured on our own line would tell you nothing about yours. What multipart buys for certain is that progress you have made stays made.

What happens to an upload you never finish?

This is the part most people never think about, and it is the one that costs real money. The parts of an unfinished multipart upload do not show up as a file. They are invisible in a normal listing of the bucket and cannot be reached by name, but storage keeps them, and bills for them, until somebody explicitly aborts the upload. Lose track of one and you pay for it forever.

So every large upload is recorded on our side the moment it starts, together with its upload id, and that record expires after 6 hours. A cleanup job runs every hour, finds the records that expired without being finished, and aborts each upload, which throws its parts away. The browser's note expires an hour earlier, at 5 hours, so it can never offer to resume an upload the server has already forgotten. If you cancel an upload yourself, Nemi aborts it straight away rather than waiting, and the sweep catches anything that call misses.

None of those parts ever count against your storage, and nothing appears in your workspace until the whole file has arrived and been checked. Outside a Vault, files up to 500 MB are then scanned for malware; larger files are not, as our security page sets out.

The honest limits

Resuming needs you. Within one visit the uploader retries a file by itself, up to 3 attempts in all, each one carrying on from what storage has; after that, or if you close the tab, it waits for you to drop the file again. If you never do, the upload does not finish by itself, and after 5 hours it starts from scratch. It also needs the same browser on the same device, because that is where the note lives; a private window that refuses to keep it still uploads normally, it simply cannot resume after a reload. And the file has to go back into the same folder of the same workspace, because that is part of what the upload was approved for.

Within those limits, a big upload on a bad connection is no longer a gamble. A 10 GB file fits on Creator and up, and the largest single file Nemi takes is 200 GB on Business; see pricing for the rest. The step by step version for customers is in the help centre under Uploading files, and how the files then travel on to somebody else is on the Files page and in our guide to sending large files securely. Downloads have the same problem in reverse, and a folder zip that streams is how Nemi handles it.