All posts
4 min read

A library default cost me 43x throughput

Every SFTP transfer ran at exactly 1.2 MB/s. The cipher benchmark that explained it also showed the slow part was not the one I expected.

Every SFTP file transfer in Termphin ran at exactly 1.2 MB/s. Not roughly 1.2 on average: the same figure on every server, over every network, for every file, large or small.

A number that stable is not a network. Networks vary. A constant means something local is doing a fixed amount of work per byte.

I ruled out the obvious suspects first: the network path, the disk on either end, and the SFTP implementation’s own buffering. None of them explained the ceiling. Every transfer stayed flat at 1.2 MB/s.

Measuring cipher throughput

The SSH library used by Termphin is dartssh2. Its default cipher order leads with AES-GCM. That is a reasonable default. On a server, or in any client whose crypto reaches the CPU’s AES instructions, AES-GCM is the right thing to prefer.

In this app, however, the crypto runs as Dart code. There is no path to the CPU’s AES instructions, the ones that make AES fast on a modern processor. Without hardware acceleration, every cryptographic operation runs in software.

To see what was happening, I wrote a benchmark. It runs each cipher over the same 32 KB payload, 256 times, and reports throughput.

These are the results, measured on one machine:

Cipher Throughput
chacha20 (256-bit) 74.7 MB/s
chacha20-poly1305 51.0 MB/s
aes-128-ctr 51.7 MB/s
aes-256-ctr 37.4 MB/s
aes-128-gcm 1.2 MB/s
aes-256-gcm 1.2 MB/s

The transfers were running at 1.2 MB/s because the negotiated cipher ran at 1.2 MB/s. The client was never waiting on the network. It was waiting on itself.

The surprise in the key size

Raw chacha20 sits at 74.7 MB/s, but holding that up against GCM would be comparing an unauthenticated cipher with an authenticated one, which is not a comparison. The fair one is between the two authenticated ciphers, because they do the same job: chacha20-poly1305 at 51.0 against aes-256-gcm at 1.2. Roughly 43 times.

The interesting row is the key size.

In CTR mode, the 128-bit key is faster than the 256-bit key, 51.7 against 37.4. That is what you expect: fewer rounds, less work. In GCM, both key sizes give the same 1.2.

If doubling the key size changes nothing, then AES is not what is costing the time. The cost is GHASH, the authentication half of GCM, which is built on carry-less multiplication. Processors that support AES instructions almost always have an instruction for that multiplication too. In software it has none, and it dominates everything else so completely that the cipher underneath it stops mattering.

So the headline is not “AES is slow in software”. AES-CTR manages 37 to 51 MB/s, which is fine. The headline is that GCM’s authentication is what falls off a cliff when the hardware is not there to do it.

What changed

The fix was to change what the client requests during the handshake.

The app now sends its own cipher preference instead of taking the library’s default: chacha20-poly1305 first, then AES-CTR, then AES-GCM last. GCM stays on the list for a server that offers nothing else, because a slow session beats no session. No CBC cipher is on the list at all.

The benchmark lives at benchmark/cipher_benchmark.dart and is run on demand rather than as part of the test suite. Anyone can run it and get their own version of the table.

Limits of the measurement

The measurement has clear limits, and they matter.

These figures are from one machine. A phone is slower. What transfers between machines is the ratio between the rows, not the absolute numbers. The relative gap between software GCM and the other ciphers remains, even if the absolute throughput drops on slower hardware.

The benchmark measures the cipher primitives on their own, not a full SSH session with its packet framing and its round trips. It explains the ceiling; it is not a model of the protocol. A real transfer has protocol overhead, but it cannot run faster than the cipher underneath it.

Nothing here says AES-GCM is a poor choice generally. On hardware with the instructions for it, it is excellent, and that is why it is the common default. The measurement says what it costs without them.

The lesson

A library default encodes an assumption about where the code will run. “Prefer AES-GCM” assumes hardware acceleration, which is true almost everywhere and was not true here. In dartssh2, that default makes complete sense on a server or on a runtime with access to CPU instructions. In a pure Dart runtime without them, that same default turns into an unexpected bottleneck.

The assumption was invisible until something was measured, and the symptom that led there looked like a network problem for a long time. A suspiciously round, suspiciously stable number is worth more attention than a slow one that varies.