[httperf] httperf error (httperf: connection failed with unexpected error 99 )

Mohit Dhingra mohitdhingras at gmail.com
Tue Feb 14 10:24:51 PST 2012


You are right Arlitt.

During test

cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
28215
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
28233
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
28236
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
28236
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
26413

After completion of test

cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
15163
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
15163
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
15163
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
15163
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
15163
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
13239
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
13237
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
13239
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
13243
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
13244
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
13244
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
13244
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
5999
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
5999
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
5999
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
5999
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
5999
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
5999
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
5999
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
5999
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
5999
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
13
cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
13

Then, I changed some kernel parameters :
fs.file-max = 5000000
net.core.netdev_max_backlog = 400000
net.core.optmem_max = 10000000
net.core.rmem_default = 10000000
net.core.rmem_max = 10000000
net.core.somaxconn = 100000
net.core.wmem_default = 10000000
net.core.wmem_max = 10000000
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_congestion_control = bic
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_max_syn_backlog = 12000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_mem = 30000000 30000000 30000000
net.ipv4.tcp_rmem = 30000000 30000000 30000000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_wmem = 30000000 30000000 30000000
*net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
*
After that I am getting perfect results. Thanks a lot for the help !!!

*
----------------------------
Thanks & Regards
Mohit Dhingra
+919611190435*


On 14 February 2012 22:54, Arlitt, Martin <martin.arlitt at hp.com> wrote:

> That sounds like it could be the TIME_WAIT issue. The next time you
> encounter the error, check how many connections are in the TIME_WAIT state
> (use ‘netstat –an’ to see this; you may of course want to pipe it to other
> commands to actually count the number of connections in different states).
> ****
>
> ** **
>
> Martin****
>
> ** **
>
> ** **
>
> *From:* Mohit Dhingra [mailto:mohitdhingras at gmail.com]
> *Sent:* Tuesday, February 14, 2012 9:19 AM
>
> *To:* Arlitt, Martin
> *Cc:* httperf at linux.hpl.hp.com
> *Subject:* Re: [httperf] httperf error (httperf: connection failed with
> unexpected error 99 )****
>
> ** **
>
> *Hi Arlitt,*
>
> Sometimes I am getting this error at rate of 500/s , and sometimes at
> 1000/s. When I reboot my machine, test with rate of even 1000/s passes
> correctly, and when I would have run so many tests already, then I get it
> even at rate of 200 /s.
>
> **
>
> *---------------------------- *
>
> *Thanks & Regards
> Mohit Dhingra
> +919611190435*
>
> ****
>
> On 14 February 2012 22:26, Arlitt, Martin <martin.arlitt at hp.com> wrote:***
> *
>
> Hi Mohit****
>
>  ****
>
> At what rate are you seeing the “error 99”? if it is at a high rate, you
> may be running out of ports on your client due to TCP TIME_WAIT.****
>
>  ****
>
> Thanks****
>
> Martin****
>
>  ****
>
>  ****
>
> *From:* Mohit Dhingra [mailto:mohitdhingras at gmail.com]
> *Sent:* Tuesday, February 14, 2012 8:37 AM****
>
>
> *To:* Arlitt, Martin
> *Cc:* httperf at linux.hpl.hp.com****
>
> *Subject:* Re: [httperf] httperf error (httperf: connection failed with
> unexpected error 99 )****
>
>  ****
>
> *Hi Arlitt, *
>
> I tried default http port (80), now I do get some output, after certain
> rate, it shows the error now.
> Please check the attached file.
> But I am getting "httperf: connection failed with unexpected error 99"
> error quite randomly, everytime, I am running the tests one after the other
> in a loop. Could that be a problem.?
>
> ****
>
> *---------------------------- *****
>
> *Thanks & Regards
> Mohit Dhingra
> +919611190435*****
>
> On 14 February 2012 08:28, Mohit Dhingra <mohitdhingras at gmail.com> wrote:*
> ***
>
> *Hi Arlitt,*
> ****
>
>
> I was able to get the results some hours back, on the same port, on the
> same server (that's an OpenNebula login page, and I am still able to see
> that on the browser), but now, I am getting this error. Not even a single
> request is being replied. Earlier, I was getting correct results, but with
> some errors of unavailable file descriptors, and I modified that in
> limits.conf and typesize.h, and also by ulimit command.*
> *
> On the client side,
> cadlab:~/Documents # ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 60860
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) 6931664
> open files                      (-n) 165535
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 60860
> virtual memory          (kbytes, -v) 23300320
> file locks                      (-x) unlimited*
>
>
> ---------------------------- *****
>
> *Thanks & Regards
> Mohit Dhingra
> +919611190435*****
>
> ** **
>
> On 14 February 2012 00:30, Arlitt, Martin <martin.arlitt at hp.com> wrote:***
> *
>
> Hi Mohit****
>
>  ****
>
> Have you verified that you are able to complete even a single HTTP
> request/response transaction to the server on that port? Normally, I would
> expect to see at least a few complete successfully, but the output below
> indicates they all failed.****
>
>  ****
>
> Thanks****
>
> Martin****
>
>  ****
>
>  ****
>
> *From:* httperf-bounces at linux.hpl.hp.com [mailto:
> httperf-bounces at linux.hpl.hp.com] *On Behalf Of *Mohit Dhingra
> *Sent:* Monday, February 13, 2012 10:46 AM
> *To:* httperf at linux.hpl.hp.com
> *Subject:* [httperf] httperf error (httperf: connection failed with
> unexpected error 99 )****
>
>  ****
>
>  ****
>
> *Hi,*
> ****
>
> *
> *I am getting an error while running httperf test.
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
> cadlab:~/Documents # httperf --server 10.112.10.35 --port 4567 --num-conns
> 20000 --rate 2000 --timeout 1
> httperf --timeout=1 --client=0/1 --server=10.112.10.35 --port=4567 --uri=/
> --rate=2000 --send-buffer=4096 --recv-buffer=16384 --num-conns=20000
> --num-calls=1
> httperf: connection failed with unexpected error 99
> Connection rate: 470.9 conn/s (2.1 ms/conn, <=1 concurrent connections)
> Connection time [ms]: min 0.0 avg 0.0 max 0.0 median 0.0 stddev 0.0
> Connection time [ms]: connect 0.0
> Connection length [replies/conn]: 0.000
>
> Request rate: 0.0 req/s (0.0 ms/req)
> Request size [B]: 0.0
>
> Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
> Reply time [ms]: response 0.0 transfer 0.0
> Reply size [B]: header 0.0 content 0.0 footer 0.0 (total 0.0)
> Reply status: 1xx=0 2xx=0 3xx=0 4xx=0 5xx=0
>
> CPU time [s]: user 0.00 system 6.71 (user 0.0% system 157.9% total 157.9%)
> Net I/O: 0.0 KB/s (0.0*10^6 bps)
>
> Errors: total 2000 client-timo 0 socket-timo 0 connrefused 0 connreset 0
> Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 2000
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> I searched on Internet about the same, it says, it's due to limited no. of
> ports. Adding --hog solved the problem for them. But, here, i am looking
> for data from a particular port itself 4567. I think --hog doesn't make
> sense. And it just hangs when I do the same.
> *
> ---------------------------- *****
>
> *Thanks & Regards
> Mohit Dhingra
> +919611190435*****
>
>  ****
>
>  ****
>
>  ****
>
> ** **
>
> _______________________________________________
> httperf mailing list
> httperf at linux.hpl.hp.com
> http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://napali.hpl.hp.com/pipermail/httperf/attachments/20120214/6ae2b1f5/attachment-0001.htm


More information about the httperf mailing list