We rebuilt our file converter. Here is every benchmark we ran

Every time you convert a photo, pull a frame out of a video or pack files into an archive, a service we run does the work. In September we rewrote it from top to bottom. This post measures what that changed, properly, including the parts that are not flattering and the two things measuring it made us fix.

We built the version from before the rewrite, the rewrite itself and today's version, side by side on the same machine. Then we ran the same twelve jobs through each, the kind people actually do: re-encoding a phone photo, shrinking a camera file, turning a recording into an MP3, packing a folder. Five timed runs each, the middle one counts.

All twelve jobs, before and today

Here is every result on one axis, comparing the version from before the rewrite with the one running today. It is logarithmic, because the gains range from a few percent to fifty times, and anything slower would sit to the left of the line.

Three groups stand out. Images got dramatically faster. Archives got faster where media was involved. And the jobs that hand straight over to a well known encoder, video, audio and documents, barely moved, because the encoder was never the problem. We will take them in that order.

Watch any of them run

A ratio is easy to read past. So here are the measured times played side by side. Pick a job and press start; "Real time" plays the medians exactly as they were measured.

Why did one JPEG take seven seconds?

Re-encoding a 12 megapixel JPEG at quality 85 took 7.08 s before the rewrite and takes 173 ms today. The encoder is the same library at the same setting, and the two files differ by fewer than a hundred bytes. The difference is entirely in how the result was written to disk.

The old service handed the encoder a plain file. A JPEG encoder writes its output in very small pieces, and without a buffer in between, each of those pieces becomes its own request to the operating system. Millions of tiny writes, each one cheap, add up to seconds. The rewrite puts a buffer between the encoder and the file, so the same bytes reach the disk in a few hundred large writes.

Fifty images is the same story fifty times. Converting fifty 1080p PNGs to JPEG took 36.8 s before and 716 ms after, because every one of them went through that same JPEG writer. That makes it the biggest win in the table, at 51 times.

The slowest part of the old converter was not converting. It was writing the answer down.

A still from a video, and a resize

Taking one frame from 45 seconds into a minute long video went from 1.79 s to 213 ms. The old service asked the video tool to decode from the start and throw frames away until it reached 45 seconds. The rewrite asks it to jump to 45 seconds first and decode from there. In the tool's own terms, the -ss option moved in front of the input. One argument, the same frame, 8.4 times faster, and the further into a long video you go, the bigger the gap gets.

Shrinking a 24 megapixel photo to 1920 pixels wide went from 1.71 s to 163 ms. The rewrite uses a resizing library that works on many pixels per instruction, plus the same buffered writer.

Why did zipping photos and videos get so much faster?

Packing two photos and two videos, 86.9 MB in all, into a zip took 1.87 s before and takes 107 ms today. The old service compressed everything it put in a zip. Photos and videos are already compressed, so that work bought almost nothing: the zip came out the same size. The rewrite recognises files that do not compress and stores them as they are. The archive is the same size and opens the same way. It is the same reasoning behind the light setting Nemi uses when you download a folder as one zip.

Packing 40 MB of log files as .tar.xz is 1.5 times faster, because the compressor now uses more than one core. On a file that small it can only split the work into two blocks, so two cores is all it gets; bigger archives can be split further. Every .tar.xz it writes carries a CRC64 checksum, multithreaded or not, so a damaged copy is caught when it is unpacked.

The one that got slower, and how we fixed it

Converting a 12 megapixel photo to WebP took 361 ms before the rewrite and 802 ms right after it: twice as slow. Part of that was deliberate. The old service ignored the quality slider and always wrote lossless WebP, which is quick to make and enormous. The rewrite did what the slider said, and lossy encoding is real work.

But slower was not good enough, so while writing this post we looked at where the time went. Almost all of it was in one setting. The WebP encoder has a dial for how hard it searches for the smallest file, from 0 to 6, and we were on its default of 4. We tried every step on the test photo and on four real camera photos of 8 to 21 megapixels. Step 2 encodes about twice as fast as step 4, and the files land between 4 percent smaller and 7 percent larger, with picture quality the same to within a rounding error. The steps above it spend their time on a search that buys a few percent of file size on a photo. Turning on more threads made no difference.

So today a WebP takes 352 ms, against 361 ms before the rewrite, and the file is 810 KB instead of 19 MB. Same speed, a file you would actually put on a web page.

What did not change, and why that is fine

Converting a 1080p MOV to MP4 took 13.1 s before and takes 13.1 s today. Three minutes of WAV to MP3, and a 41 page Word document to PDF, moved by about ten percent. These jobs are almost entirely the encoder's own work, the encoder runs with the same settings in every version, and the files come out the same to within a few bytes. There was nothing for a rewrite to win here.

The video work in the rewrite went elsewhere: real progress while an encode runs instead of a bar that guesses, and a limit on how many heavy encodes run at once, so a queue of videos no longer starts one encoder per core with every encoder trying to use every core. If what you need is a smaller video rather than a faster one, our guide to making a video small enough to send has the settings.

AVIF is in the same group for a different reason. It is 1 times as fast, but the comparison is not like for like: the rewrite also changed the encoder's speed setting, so it is a slightly different file made faster, not the same file.

The full table

Every median for all three versions. Switch to the log scale to compare the small ones.

How should you read these numbers?

Every job ran five times per version, with the versions interleaved, and every output was checked before a timing counted. Three things are worth keeping in mind.

• Read the ratios, not the seconds. This is an ordinary 8 GB M2 with other work running, and production is a different machine.
• Every run is kept. The harness, the test files and each individual timing are saved, so we can repeat the measurement whenever the service changes.
• Conversion runs on hardware we operate. Your files are not sent to an outside conversion company, as the security page sets out.

If you want to see what the converter can do now, the format explorer lists every pair it supports, and Convert is where you use it. The help page on converting files walks through the panel step by step.