This LWN article from last month (I’m behind on my reading…) talks about techniques to reduce TCP latency by adjusting the size of send buffers on the fly so they’re just big enough to keep the hardware busy but not too bloated, which adds latency.
I couldn’t help noticing, though, that the problem is primarily one where bandwidth-optimized work causes problems for latency-sensitive work. This is probably naive, but couldn’t the kernel detect the low-bandwidth, low-latency sessions automatically by simply noticing that they often have no data to send? In contrast, sessions that always have more data to send probably could tolerate waiting a few extra milliseconds to send that data out. It’s like at the grocery store, letting the guy with a couple items go ahead of you and your full cart.
Does that make sense?