Wednesday, August 20, 2014

Running RabbitMQ on Ubuntu 14.04?

Recently during a few test runs we noticed that our AMQP clients would often lock up when interfacing with a RabbitMQ machine running on Ubuntu 14.04 hosted on Amazon AWS.  By installing tshark and capturing the network traces on the host running the AMQP client, I noticed there seemed to be issues reported as "Unknown frame type".  Upon further investigation of the unknown frame packet, I noticed that we seemed to be missing the AMQP header, causing the packet to be classified as unknown by Wireshark.  Normally there should have been Basic.Deliver like all the previous packets:


I noticed on the machine running on RabbitMQ that there were "xen_netfront: xennet: skb rides the rocket" error messages.  It turns out that AMQP is very sensitive to missing frame headers and caused both RabbitMQ and the AMQP client to wait on each other.  The broker thinks there are more messages waiting to be acknowledged, while the client sees no new messages.  The reason appears to be that the missing AMQP header caused by the dropped transmit packet causes this issue:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811
http://www.spinics.net/lists/netdev/msg282340.html

The solution appears to be disable scatter/gather I/O on the network card, since the Xen driver may take longer to resolve:

sudo ethtool -k eth0

The patch that appears to have started contributing to these issues:

https://gitorious.org/linux-can/linux-can-next/commit/f36c374782e40a3812f729838b5b80d2ce601725

1 comment: