[httperf] httperf: connection failed with unexpected error 0 & low performance

[Rick Jones]

On 10/19/2012 10:10 AM, Vikash Kumar wrote:
> Hi Raufoeh,
>
>      With Intel back to back 10G NIC card, I was able to around 28K
> reply rate with 1kb file size with Apache server.
>
>      But when using haproxy that means with 3 node setup like
>
>
>                                 *haproxy host*
> *                                /         \*
> *                               /             \*
> *                             /                 \*
> *                       Client                Server*
> *
> *
> **connection rate is good but request rate and reply rate are dropping
> drastically. All the three machines are 8 core desktop m/c. On the
> haproxy site it has been published that haproxy can sustain more than
> 30K connection.

Are you quite sure that you are achieving 10000 connections established 
per second?  Is that backed-up by the networking statistics on the 
systems involved?

>     I am unable to find out the reason to such a  poor performance
> whether it is issue with host m/c or haproxy.

What do the networking statistics look like on all three systems? 
Anything about retransmissions or queue overflows etc?

Also, how do the stacks involved deal with "wrapping" the four-tuple 
space with end-points in TIME_WAIT?  That is, if there is only the one 
client IP, and one haproxy host IP and one well-known port for the 
proxy, there will then be only as many unique "names" for the TCP 
connections as there are port numbers used on the client.  If the client 
code is not making explicit calls to bind() to, for example, try to use 
ports 5000 through 65535 it will be limited to the anonymous port range. 
  Often that is 49152 through 65535.  With a 60 second TIME_WAIT that 
would be (65535-49152)/60 connections per second or higher looking for 
TIME_WAIT reuse.

rick jones