[httperf] httperf error (httperf: connection failed with unexpected error 99 )

Mohit Dhingra mohitdhingras at gmail.com
Wed Feb 15 08:42:43 PST 2012


*Hello,*

I dont know whether this question is directly related here, but if somebody
can give me some pointers, it would be of great help.

I am able to run httperf from any host to Dom0 (OpenSUSE with Xen as
Hypervisor), but I am getting all timeouts if I run httperf with DomU VM as
Server. I am able to ping that particular VM, but not able to ssh or run
netperf or httperf. I can give more details if required.

*
----------------------------
Thanks & Regards
Mohit Dhingra
+919611190435*


On 14 February 2012 23:54, Mohit Dhingra <mohitdhingras at gmail.com> wrote:

> You are right Arlitt.
>
> During test
>
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 28215
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 28233
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 28236
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 28236
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 26413
>
> After completion of test
>
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 15163
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 15163
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 15163
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 15163
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 15163
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 13239
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 13237
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 13239
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 13243
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 13244
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 13244
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 13244
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 5999
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 5999
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 5999
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 5999
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 5999
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 5999
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 5999
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 5999
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 5999
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 13
> cadlab:~/Documents # netstat -an | grep TIME_WAIT | wc -l
> 13
>
> Then, I changed some kernel parameters :
> fs.file-max = 5000000
> net.core.netdev_max_backlog = 400000
> net.core.optmem_max = 10000000
> net.core.rmem_default = 10000000
> net.core.rmem_max = 10000000
> net.core.somaxconn = 100000
> net.core.wmem_default = 10000000
> net.core.wmem_max = 10000000
> net.ipv4.conf.all.rp_filter = 1
> net.ipv4.conf.default.rp_filter = 1
> net.ipv4.ip_local_port_range = 1024 65535
> net.ipv4.tcp_congestion_control = bic
> net.ipv4.tcp_ecn = 0
> net.ipv4.tcp_max_syn_backlog = 12000
> net.ipv4.tcp_max_tw_buckets = 2000000
> net.ipv4.tcp_mem = 30000000 30000000 30000000
> net.ipv4.tcp_rmem = 30000000 30000000 30000000
> net.ipv4.tcp_sack = 1
> net.ipv4.tcp_syncookies = 0
> net.ipv4.tcp_timestamps = 1
> net.ipv4.tcp_wmem = 30000000 30000000 30000000
> *net.ipv4.tcp_tw_reuse = 1
> net.ipv4.tcp_tw_recycle = 1
> *
> After that I am getting perfect results. Thanks a lot for the help !!!
>
>
> *
> ----------------------------
>  Thanks & Regards
> Mohit Dhingra
> +919611190435*
>
>
> On 14 February 2012 22:54, Arlitt, Martin <martin.arlitt at hp.com> wrote:
>
>> That sounds like it could be the TIME_WAIT issue. The next time you
>> encounter the error, check how many connections are in the TIME_WAIT state
>> (use ‘netstat –an’ to see this; you may of course want to pipe it to other
>> commands to actually count the number of connections in different states).
>> ****
>>
>> ** **
>>
>> Martin****
>>
>> ** **
>>
>> ** **
>>
>> *From:* Mohit Dhingra [mailto:mohitdhingras at gmail.com]
>> *Sent:* Tuesday, February 14, 2012 9:19 AM
>>
>> *To:* Arlitt, Martin
>> *Cc:* httperf at linux.hpl.hp.com
>> *Subject:* Re: [httperf] httperf error (httperf: connection failed with
>> unexpected error 99 )****
>>
>> ** **
>>
>> *Hi Arlitt,*
>>
>> Sometimes I am getting this error at rate of 500/s , and sometimes at
>> 1000/s. When I reboot my machine, test with rate of even 1000/s passes
>> correctly, and when I would have run so many tests already, then I get it
>> even at rate of 200 /s.
>>
>> **
>>
>> *---------------------------- *
>>
>> *Thanks & Regards
>> Mohit Dhingra
>> +919611190435*
>>
>> ****
>>
>> On 14 February 2012 22:26, Arlitt, Martin <martin.arlitt at hp.com> wrote:**
>> **
>>
>> Hi Mohit****
>>
>>  ****
>>
>> At what rate are you seeing the “error 99”? if it is at a high rate, you
>> may be running out of ports on your client due to TCP TIME_WAIT.****
>>
>>  ****
>>
>> Thanks****
>>
>> Martin****
>>
>>  ****
>>
>>  ****
>>
>> *From:* Mohit Dhingra [mailto:mohitdhingras at gmail.com]
>> *Sent:* Tuesday, February 14, 2012 8:37 AM****
>>
>>
>> *To:* Arlitt, Martin
>> *Cc:* httperf at linux.hpl.hp.com****
>>
>> *Subject:* Re: [httperf] httperf error (httperf: connection failed with
>> unexpected error 99 )****
>>
>>  ****
>>
>> *Hi Arlitt, *
>>
>> I tried default http port (80), now I do get some output, after certain
>> rate, it shows the error now.
>> Please check the attached file.
>> But I am getting "httperf: connection failed with unexpected error 99"
>> error quite randomly, everytime, I am running the tests one after the other
>> in a loop. Could that be a problem.?
>>
>> ****
>>
>> *---------------------------- *****
>>
>> *Thanks & Regards
>> Mohit Dhingra
>> +919611190435*****
>>
>> On 14 February 2012 08:28, Mohit Dhingra <mohitdhingras at gmail.com> wrote:
>> ****
>>
>> *Hi Arlitt,*
>> ****
>>
>>
>> I was able to get the results some hours back, on the same port, on the
>> same server (that's an OpenNebula login page, and I am still able to see
>> that on the browser), but now, I am getting this error. Not even a single
>> request is being replied. Earlier, I was getting correct results, but with
>> some errors of unavailable file descriptors, and I modified that in
>> limits.conf and typesize.h, and also by ulimit command.*
>> *
>> On the client side,
>> cadlab:~/Documents # ulimit -a
>> core file size          (blocks, -c) 0
>> data seg size           (kbytes, -d) unlimited
>> scheduling priority             (-e) 0
>> file size               (blocks, -f) unlimited
>> pending signals                 (-i) 60860
>> max locked memory       (kbytes, -l) 64
>> max memory size         (kbytes, -m) 6931664
>> open files                      (-n) 165535
>> pipe size            (512 bytes, -p) 8
>> POSIX message queues     (bytes, -q) 819200
>> real-time priority              (-r) 0
>> stack size              (kbytes, -s) 8192
>> cpu time               (seconds, -t) unlimited
>> max user processes              (-u) 60860
>> virtual memory          (kbytes, -v) 23300320
>> file locks                      (-x) unlimited*
>>
>>
>> ---------------------------- *****
>>
>> *Thanks & Regards
>> Mohit Dhingra
>> +919611190435*****
>>
>> ** **
>>
>> On 14 February 2012 00:30, Arlitt, Martin <martin.arlitt at hp.com> wrote:**
>> **
>>
>> Hi Mohit****
>>
>>  ****
>>
>> Have you verified that you are able to complete even a single HTTP
>> request/response transaction to the server on that port? Normally, I would
>> expect to see at least a few complete successfully, but the output below
>> indicates they all failed.****
>>
>>  ****
>>
>> Thanks****
>>
>> Martin****
>>
>>  ****
>>
>>  ****
>>
>> *From:* httperf-bounces at linux.hpl.hp.com [mailto:
>> httperf-bounces at linux.hpl.hp.com] *On Behalf Of *Mohit Dhingra
>> *Sent:* Monday, February 13, 2012 10:46 AM
>> *To:* httperf at linux.hpl.hp.com
>> *Subject:* [httperf] httperf error (httperf: connection failed with
>> unexpected error 99 )****
>>
>>  ****
>>
>>  ****
>>
>> *Hi,*
>> ****
>>
>> *
>> *I am getting an error while running httperf test.
>>
>> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>> cadlab:~/Documents # httperf --server 10.112.10.35 --port 4567
>> --num-conns 20000 --rate 2000 --timeout 1
>> httperf --timeout=1 --client=0/1 --server=10.112.10.35 --port=4567
>> --uri=/ --rate=2000 --send-buffer=4096 --recv-buffer=16384
>> --num-conns=20000 --num-calls=1
>> httperf: connection failed with unexpected error 99
>> Connection rate: 470.9 conn/s (2.1 ms/conn, <=1 concurrent connections)
>> Connection time [ms]: min 0.0 avg 0.0 max 0.0 median 0.0 stddev 0.0
>> Connection time [ms]: connect 0.0
>> Connection length [replies/conn]: 0.000
>>
>> Request rate: 0.0 req/s (0.0 ms/req)
>> Request size [B]: 0.0
>>
>> Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
>> Reply time [ms]: response 0.0 transfer 0.0
>> Reply size [B]: header 0.0 content 0.0 footer 0.0 (total 0.0)
>> Reply status: 1xx=0 2xx=0 3xx=0 4xx=0 5xx=0
>>
>> CPU time [s]: user 0.00 system 6.71 (user 0.0% system 157.9% total 157.9%)
>> Net I/O: 0.0 KB/s (0.0*10^6 bps)
>>
>> Errors: total 2000 client-timo 0 socket-timo 0 connrefused 0 connreset 0
>> Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 2000
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> I searched on Internet about the same, it says, it's due to limited no.
>> of ports. Adding --hog solved the problem for them. But, here, i am looking
>> for data from a particular port itself 4567. I think --hog doesn't make
>> sense. And it just hangs when I do the same.
>> *
>> ---------------------------- *****
>>
>> *Thanks & Regards
>> Mohit Dhingra
>> +919611190435*****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>> ** **
>>
>> _______________________________________________
>> httperf mailing list
>> httperf at linux.hpl.hp.com
>> http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://napali.hpl.hp.com/pipermail/httperf/attachments/20120215/9a85fd2d/attachment-0001.htm


More information about the httperf mailing list