I setup the rabbitmq clustering according to the official doc and queue mirroring according to this article. Then I configure a HAProxy in front of the rabbitmq cluster.

Problem

My own script using pika starts fine and is able to connect to the HAProxy and consume the messages in the queue. But it will report the following error every 2 mins or so and cause my client to completely disconnected from HAProxy.

Solution

After googling for hours, this article solves my problem.

In short, on a Linux machine, tcp_keepalive is sent every 7200 seconds.

$ cat /proc/sys/net/ipv4/tcp_keepalive_time
7200

By default, HAProxy has a timeout set to 1m.

defaults
        timeout client 1m
        timeout server 1m

So if your queue is idle for more than 1m, HAProxy will force to drop the connection.
To fix it, increase the HAProxy timeout value to 3h

listen rabbitmq
        timeout client 3h
        timeout server 3h