[httperf] Re: Connection rate is very low

Tim Terrill TTerrill at synacor.com
Tue Oct 23 08:23:08 PDT 2012


Hi Vikash,

Error 98 shows is a result of ports sitting in TIME_WAIT state, defined here (on Linux/Centos):

/proc/sys/net/ipv4/tcp_fin_timeout

Can be observed by:

netstat -nt | grep TIME_WAIT | wc

If you do this command in a sleep-loop you can see your time-wait ports grow during a test, and shrink after the test.  If this number approaches the max you're in trouble.  This is an issue that has a few options since everyone is limited to no more than 65k ports:


  1.  Reduce TIME_WAIT time (though you really have to know what your doing with this)
  2.  Use num_calls > 1 to send requests using a persistent connection (you lose the overhead cost of creating/destroying a connection, but you get to multiply the number of requests sent to the server)
  3.  Use test times and rates tuned specifically to ensure that ports become available before they are needed.

Here's a discussion I had started typing in response to one of your earlier posts, so I included it here as its directly relevant to #3…

This may be obvious, but since the following are tightly related:

1. duration of your test (sec)
2. the rate (connections / sec)
3. the TIME_WAIT interval (30 sec in your case)
4. the max number of available ports (less than 65535 -- see Rick's earlier analysis)

Its important to calculate the total number of ports needed by your test
to make sure the test will work as expected.

To calculate the total number of ports needed for your test do this (if
your duration is larger than the TIME_WAIT interval, then use your
TIME_WAIT interval for the duration)

Rate (conn/sec) * duration (sec) = total number of connections

(10,000 conn/sec) * (30 sec) = 300,000 connections (test won't work right)
(1,000 conn/sec) * (30 sec) = 30,000 connections per 30 second TIME_WAIT
interval (test will work right for whatever duration you want (unless
connections are being kept open longer than a second))


2. If your duration is less then the TIME_WAIT interval, then:
Rate (conn/sec) * test duration (sec) = total number of connections

(10,000 conn/sec) * (10 sec) = 100,000 connections (test won't work right)
(1,000 conn/sec) * (20 sec) = 20,000 connections per 20 second (test will work right for whatever duration you want (unless connections are being kept open longer than a second)

Bottom line is: you must keep the number of used ports under the Max (around 65k) in any given TIME_WAIT period of time.

Hope this helps…


Tim

From: Vikash Kumar <vikash.kumar at oneconvergence.com<mailto:vikash.kumar at oneconvergence.com>>
Date: Tuesday, October 23, 2012 10:50 AM
To: "httperf at linux.hpl.hp.com<mailto:httperf at linux.hpl.hp.com>" <httperf at linux.hpl.hp.com<mailto:httperf at linux.hpl.hp.com>>
Subject: Re: [httperf] Re: Connection rate is very low

Hi all,

   I was able to achieve 7000 conn/sec. But this is also very low in compare to 27K conn/sec may be I need to increase server m/c's.

   One error that I observed was TCP SYN flooding message in dmesg. Victor suggested to make tcp_syncookies=0. After doing that my connection rate also got decreased and I got some  message in my dmesg:

  process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.default.retrans_time; Use   net.ipv6.neigh.default.retrans_time_ms instead.
  TCP: Possible SYN flooding on port 80. Dropping request.  Check SNMP counters.

  This clearly mean that disabling syn_cookies is not solution, may be I am wrong or may be their is other way out.

  Is their any way to mitigate this "Possible SYN flooding "? Any detail description will be

  Other error that I got in client side is:

   httperf : connection failed with error 98

  After doing grep for error 98. I got the macro as "Address already in Use".

  How to mitigate that error? I am using Ubuntu 10.04

Regards,
Vikash



On Tue, Oct 23, 2012 at 3:22 PM, Vikash Kumar <vikash.kumar at oneconvergence.com<mailto:vikash.kumar at oneconvergence.com>> wrote:
But Victor that will lead to drop more legitimate connections.

Thnx


On Tue, Oct 23, 2012 at 2:19 PM, Vikash Kumar <vikash.kumar at oneconvergence.com<mailto:vikash.kumar at oneconvergence.com>> wrote:
Thnx Victor.

On Tue, Oct 23, 2012 at 12:24 PM, Gattegno, Victor (CPR&Q GCC - ISS/SW Tower) <victor_gattegno at hp.com<mailto:victor_gattegno at hp.com>> wrote:
Hi Vikash


>  What can be done to stoop this message?

You could do this:

# echo 0 > /proc/sys/net/ipv4/tcp_syncookies

Or, if you want it permanent (after reboot), set: net.ipv4.tcp_syncookies = 0 in /etc/sysctl.conf

Rgds,
Victor

From:httperf-bounces at linux.hpl.hp.com<mailto:httperf-bounces at linux.hpl.hp.com> [mailto:httperf-bounces at linux.hpl.hp.com<mailto:httperf-bounces at linux.hpl.hp.com>] On Behalf Of Vikash Kumar
Sent: lundi 22 octobre 2012 13:58
To: httperf at linux.hpl.hp.com<mailto:httperf at linux.hpl.hp.com>

Subject: [httperf] Re: Connection rate is very low

  I am also getting TCP SYN flooding msg in dmesg of Haproxy host m/c:

  [ 2965.742025] possible SYN flooding on port 80. Sending cookies.

  Command   cat /proc/sys/net/ipv4/tcp_syncookies is set to 1.

   What can be done to stoop this message?

Regards,
Vikash


_______________________________________________
httperf mailing list
httperf at linux.hpl.hp.com<mailto:httperf at linux.hpl.hp.com>
http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://napali.hpl.hp.com/pipermail/httperf/attachments/20121023/98f5344e/attachment-0001.htm


More information about the httperf mailing list