[httperf] Why does httperf take extra time even after completion

Martin Arlitt arlitt at hpl.hp.com
Mon Jul 10 11:39:08 PDT 2006


hi Prakhar

as I'm sure you are aware, there are many potential bottlenecks in a
distributed system.  when you only have measurements from one vantage
point (and from one experiment), it can make it difficult to assess the
exact bottleneck.

for the example you just sent, it appears that the problem is not with
httperf.  the problem could be the Web server software (e.g., helper
processes exhausted), the Web server hardware (e.g., cpu exhausted), or
the client hardware (e.g., cpu exhausted) (I expect it is the first, or
perhaps the second) I am basing this on the following lines from the
httperf output:

> Connection time [ms]: min 147.7 avg 19278.7 max 104047.5 median 13140.5
> stddev 18025.2

as both the avg and max connection time indicate, some of the TCP
connections are open for a very long time (upto 104 seconds).  This
seems like a long time for the requested php page to run.

httperf will not exit until all of the requests it initiated have either
completed or timed out.  since you did not specify a timeout value, there
is no cap on how long httperf will wait for the server response (i.e., the
default timeout value is infinity).

I hope this helps you identify the actual bottleneck.

Martin

On Mon, 10 Jul 2006, Prakhar Goyal wrote:

> Dear Sir,
> Thank you for the explanation.
> But I am mentioning one instance here, in which I faced the contradiction.
>
> Here is the output of httperf:
> ________________Output Starts_________________________________________
>
> httperf --server kahn.cse.iitb.ac.in --port 80
> --uri/model_prakhar/scene1.php --rate 17 --num-conns 2700
>
> httperf --client=0/1 --server=kahn.cse.iitb.ac.in --port=80
> --uri=/model_prakhar/scene1.php --rate=17 --send-buffer=4096
> --recv-buffer=16384 --num-conns=2700 --num-calls=1
> Maximum connect burst length: 1
>
> Total: connections 2700 requests 2698 replies 2698 test-duration 249.089 s
>
> Connection rate: 10.8 conn/s (92.3 ms/conn, <=523 concurrent connections)
> Connection time [ms]: min 147.7 avg 19278.7 max 104047.5 median 13140.5
> stddev 18025.2
> Connection time [ms]: connect 10471.9
> Connection length [replies/conn]: 1.000
>
> Request rate: 10.8 req/s (92.3 ms/req)
> Request size [B]: 94.0
>
> Reply rate [replies/s]: min 0.0 avg 11.0 max 14.2 stddev 5.3 (49 samples)
> Reply time [ms]: response 8939.0 transfer 0.0
> Reply size [B]: header 225.0 content 505.0 footer 2.0 (total 732.0)
> Reply status: 1xx=0 2xx=2698 3xx=0 4xx=0 5xx=0
>
> CPU time [s]: user 12.02 system 237.06 (user 4.8% system 95.2% total 100.0%)
> Net I/O: 8.7 KB/s (0.1*10^6 bps)
>
> Errors: total 2 client-timo 0 socket-timo 2 connrefused 0 connreset 0
> Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
> _________________________Output Ends___________________________________
>
> Here the "fd-anvail" errors is zero, and the system had to open max 523
> concurrent connections. That means it has not reached the max no of the
> system possible (1024 or so).. Then also it is giving a time lag of some
> 30 secs.
> Does any explaination exists for this ?
>
> While I also observed that at lower rates, it doesnt take extra time. So,
> that means HTTPERF TAKES EXTRA TIME AT HIGHER RATES, due to which the
> request rate and throughput values of httperf are wrong.
> Am I right or wrong ?
> Please rectify me, if I am wrong.
>
> It will be my pleasure to provide you any other information required
> regarding my expt.
>
> Thanking you a lot for your precious responses,
>
> Sincerely yours,
> Prakhar Goyal
> IIT Bombay, India.
>
> > Prakhar,
> >
> > As Martin suggested, your tests are not running "cleanly".
> >
> > 1)  The server seems to be overloaded. (significant number of socket
> > timeouts, 51 seconds to open a connection with the server,  16 seconds
> > to receive reply from the server)
> > 2)  httperf is not able to find enough unused file descriptors to
> > achieve the connection rate you have specified - there is a significant
> > number of  fails on fd-unavail.  It looks like your client machine has a
> > limit of 1024 for the maximum number of file descriptors - you can
> > notice that the max # of concurrent connections that could be opened was
> > 1022.  You can fix this problem by referring to the following link:
> >
> > http://devel.squid-cache.org/hno/linux-lfd.html
> >
> > Diwakar.
> >
> > Prakhar Goyal wrote:
> >
> >>Dear Sir,
> >>Thankyou very much for your prompt reply. I am sorry that I could not
> >>reply early as I was materialising the data.
> >>
> >>Here is the report given by httperf in one of my expts, which involves
> >> the
> >>performance analysis of web server:
> >>
> >>____________Report Starts______________________________
> >>
> >>httperf --server servername --port 80 --uri /model_prakhar/scene1.php
> >>--rate 22 --num-conns 2000
> >>httperf --client=0/1 --server=kahn.cse.iitb.ac.in --port=80
> >>--uri=/model_prakhar/scene1.php --rate=22 --send-buffer=4096
> >>--recv-buffer=16384 --num-conns=2000 --num-calls=1
> >>Maximum connect burst length: 1
> >>
> >>Total: connections 1799 requests 1627 replies 1623 test-duration 266.972
> >> s
> >>
> >>Connection rate: 6.7 conn/s (148.4 ms/conn, <=1022 concurrent
> >> connections)
> >>Connection time [ms]: min 307.8 avg 53450.4 max 142786.6 median 36645.5
> >>stddev 38681.8
> >>Connection time [ms]: connect 51866.2
> >>Connection length [replies/conn]: 1.000
> >>
> >>Request rate: 6.1 req/s (164.1 ms/req)
> >>Request size [B]: 94.0
> >>
> >>Reply rate [replies/s]: min 0.0 avg 6.1 max 9.4 stddev 3.8 (53 samples)
> >>Reply time [ms]: response 16056.6 transfer 0.0
> >>Reply size [B]: header 206.0 content 504.0 footer 2.0 (total 712.0)
> >>Reply status: 1xx=0 2xx=1623 3xx=0 4xx=0 5xx=0
> >>
> >>CPU time [s]: user 5.44 system 261.53 (user 2.0% system 98.0% total
> >> 100.0%)
> >>Net I/O: 4.8 KB/s (0.0*10^6 bps)
> >>
> >>Errors: total 377 client-timo 0 socket-timo 172 connrefused 0 connreset 4
> >>Errors: fd-unavail 201 addrunavail 0 ftab-full 0 other 0
> >>________________Report Ends_____________________________
> >>
> >>The client is a normal P4 machine having 2 CPUs.
> >>The errors indicate that some connections failed.I had fixed the timeout
> >>to be 100 secs.
> >>The service time of the web server is 112 ms.
> >>But after the web server finishes the handling requests,( ie, the machine
> >>on which web server is deployed, shows 0% cpu usage), then also upto 1
> >>minute(in this test case shown above), httperf continues running.
> >>
> >>I think that this affects the request rate and the thruput too.
> >>I know the 'request rate' shown by the htperf is diff from actual arrival
> >>rate. I had read this in your mailing list archive. :)
> >>
> >>Please reply to this matter soon as per your convenience.
> >>I will be happy to provide you any other information required.
> >>
> >>Waiting for the reply,
> >>
> >>Thanks & Regards,
> >>Prakhar Goyal.
> >>
> >>
> >>
> >>
> >>
> >>>hi Prakhar
> >>>
> >>>without knowing anything about the test you were running, or about the
> >>>configuration of the client, it is difficult to know what was happening.
> >>>
> >>>I expect that your test did not run cleanly in some way; e.g., some
> >>>connections failed, or httperf had trouble getting file descriptors at
> >>>some point during the test. However, without knowing more about the test
> >>>or the httperf output from the test this is just speculation.
> >>>
> >>>Martin
> >>>
> >>>
> >>
> >>
> >
> >
> >
>
>
> --
> Prakhar Goyal
> Computer Science & Engineering Dept.
> IIT Bombay.
> Homepage: http://www.cse.iitb.ac.in/~prakhargoyal
> _______________________________________________
> httperf mailing list
> httperf at linux.hpl.hp.com
> http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/
>


More information about the httperf mailing list