[tech] Problems with Mylah's network connection

[David Adam]

On Tue, 3 Jan 2012, David Adam wrote:
> For as long as I can remember, people have been complaining that the 
> network is slow. "Whatever", I usually think, and go back to IRC.
> 
> Then I wrote a backups script that uses SSH and discovered that it would 
> die occasionally with "Corrupted MAC on input. Disconnecting: Packet 
> corrupt". Oops.
> 
> Further investigation reveals that an
>   `ssh root at murasoi cat /dev/zero | pv > /dev/null`
> pipe will eventually die with the same error message (i.e. the stream gets 
> corrupted at some point), but it usually takes hours and up to twenty 
> gigabytes of traffic. It happens between Mylah and Motsugo, too, but it 
> doesn't seem to happen between Murasoi and anything else (including 
> anything on the same switch as Mylah) - over 3TB transferred without a 
> problem.
> 
> Interestingly even though both Mylah and Murasoi are on gigabit 
> connections, the maximum throughput is more like 2-300 megabit as measured 
> by pv(1); on other gigabit-enabled hosts on the machine room it is more 
> like 800 megabit. The throughput also drops every few minutes to basically 
> zero. iperf(1) shows similar information.
> 
> A little bit of analysis with tcpdump(8) shows that captures on Mylah show 
> significant packet loss interrupting the TCP stream - lots of missed ACKs 
> and retransmissions. I suspect this causing the throughput limitations and 
> occasional pauses, but I'm not sure it is responsible for the corrupted 
> packets.
> 
> I'm not really sure where to go from here. iperf is supposed to give some 
> in-depth indication of TCP performance or dropped datagrams in UDP mode 
> but does neither. The tcpdump traces are not particularly enlightening; 
> not much is changing quickly in `netstat -s`.
> 
> I wonder about the performance of a 32-bit 33mHz gigabit network card but 
> have no idea how to measure the PCI utilisation or interrupt frequency on 
> Linux. (pcitop looked promising but only works on HP IA-64 machines.)
> 
> Anyone have any thoughts?

I've been poking at this off and on over the last few weeks as I've had 
time and am no closer to understanding what's going on. I've been testing 
with both SSH and uncompressed nc(1) pipes.

It might be partly a switch issue. The throughput between Meersau (on 
curviceps like Mylah) and Murasoi (on coconut) is about double what it is 
between Mylah and Murasoi, but still only about half of what it is between 
Murasoi and Motsugo (both on coconut).

I'm pretty sure it's not a bus speed issue. There is a tg3-based device in 
Mylah as well, which is on a PCI Express slot, and the throughput is 
equally crappy. We initially switched away from this due to bugs in the 
tg3 driver, but a lot of work has been done in newer kernels which we are 
now running.

I guess it could be a CPU speed thing - Mylah is now at the slower end of 
the spectrum of machines and it also hosts Mussel as a VM which is 
relatively busy. The data that supports this is that the throughput drops 
about 50% when going between Mussel and Mylah as compared to Mylah and 
anything else, which would suggest to me that the bulk of the issue is in 
the Linux networking code.

In terms of the corruption issue, my inexpert use of tcpdump and 
binary-diffs doesn't identify a difference in the captured stream that 
would lead to a "packet corrupt" issue, but there's a possibility the 
stream is getting mangled above the level of the packet capture.

I wonder if we're tickling a bug in OpenSSH. Using the Dropbear client on 
Mylah and the OpenSSH server on Murasoi doesn't produce the same problems 
as the OpenSSH client; presumably Dropbear does the same sort of packet 
integrity checks (Matt?).

I am still very much open to ideas about how to proceed.

[DAA]