[httperf] Why does httperf take extra time even after completion

Martin Arlitt arlitt at hpl.hp.com
Thu Jul 13 12:55:02 PDT 2006


hi Prakhar

thanks for the feedback.  I'm glad your problem has been addressed.

Martin


On Thu, 13 Jul 2006, Prakhar Goyal wrote:

> Dear Sir,
> Thank you very much for the explanation you have given for my problem.
>
> Actually in my last expts, I had not specified the timeout values, since I
> wanted the server to be overloaded. But now I have clearly understood that
> for the correct performance analysis, the specification of timeout value
> is quite essential.
>
> Therefore, I performed the same expt, but now, the timeout was kept as
> 30secs, and the no of conns was increased so as to 'smoothen' the effect
> of the 'cooling down' and 'warming up' of the server, so that the steady
> state will be maintained for a longer time.
> There, I found that the HTTPERF DOES NOT TAKE EXTRA TIME. So, my problem
> has been solved.
>
> Thus , I like to mention that for the performnace analysis in overloading,
> with the help of httperf should be done with the specification of
> appropriate timeout. HTTPERF gives correct outputs of throughput and
> response time.
>
> In addition to this, I like to say that the httperf would give better
> (more exact) request rate and thruput(reply rate ,here), if there is also
> a provision available in the httperf to mention the 2 time values by the
> user (starting time and the finishing time), and the httperf calculates
> the desired request rate & reply rate based on the performance during that
> time period.
> So, if the 2 time values happen to be the time values, when the steady
> state has begun and finished respectively, then the calculation done on
> that duration will give the exact desired values.
>
> Really saying sir, this mailing list is very benificial for us.
>
> Thanking you a lot ,
>
> Yours sincerely,
> Prakhar Goyal
> India.
>
> > hi Prakhar
> >
> > as I'm sure you are aware, there are many potential bottlenecks in a
> > distributed system.  when you only have measurements from one vantage
> > point (and from one experiment), it can make it difficult to assess the
> > exact bottleneck.
> >
> > for the example you just sent, it appears that the problem is not with
> > httperf.  the problem could be the Web server software (e.g., helper
> > processes exhausted), the Web server hardware (e.g., cpu exhausted), or
> > the client hardware (e.g., cpu exhausted) (I expect it is the first, or
> > perhaps the second) I am basing this on the following lines from the
> > httperf output:
> >
> >> Connection time [ms]: min 147.7 avg 19278.7 max 104047.5 median 13140.5
> >> stddev 18025.2
> >
> > as both the avg and max connection time indicate, some of the TCP
> > connections are open for a very long time (upto 104 seconds).  This
> > seems like a long time for the requested php page to run.
> >
> > httperf will not exit until all of the requests it initiated have either
> > completed or timed out.  since you did not specify a timeout value, there
> > is no cap on how long httperf will wait for the server response (i.e., the
> > default timeout value is infinity).
> >
> > I hope this helps you identify the actual bottleneck.
> >
> > Martin
> >
> > On Mon, 10 Jul 2006, Prakhar Goyal wrote:
> >
> >> Dear Sir,
> >> Thank you for the explanation.
> >> But I am mentioning one instance here, in which I faced the
> >> contradiction.
> >>
> >> Here is the output of httperf:
> >> ________________Output Starts_________________________________________
> >>
> >> httperf --server kahn.cse.iitb.ac.in --port 80
> >> --uri/model_prakhar/scene1.php --rate 17 --num-conns 2700
> >>
> >> httperf --client=0/1 --server=kahn.cse.iitb.ac.in --port=80
> >> --uri=/model_prakhar/scene1.php --rate=17 --send-buffer=4096
> >> --recv-buffer=16384 --num-conns=2700 --num-calls=1
> >> Maximum connect burst length: 1
> >>
> >> Total: connections 2700 requests 2698 replies 2698 test-duration 249.089
> >> s
> >>
> >> Connection rate: 10.8 conn/s (92.3 ms/conn, <=523 concurrent
> >> connections)
> >> Connection time [ms]: min 147.7 avg 19278.7 max 104047.5 median 13140.5
> >> stddev 18025.2
> >> Connection time [ms]: connect 10471.9
> >> Connection length [replies/conn]: 1.000
> >>
> >> Request rate: 10.8 req/s (92.3 ms/req)
> >> Request size [B]: 94.0
> >>
> >> Reply rate [replies/s]: min 0.0 avg 11.0 max 14.2 stddev 5.3 (49
> >> samples)
> >> Reply time [ms]: response 8939.0 transfer 0.0
> >> Reply size [B]: header 225.0 content 505.0 footer 2.0 (total 732.0)
> >> Reply status: 1xx=0 2xx=2698 3xx=0 4xx=0 5xx=0
> >>
> >> CPU time [s]: user 12.02 system 237.06 (user 4.8% system 95.2% total
> >> 100.0%)
> >> Net I/O: 8.7 KB/s (0.1*10^6 bps)
> >>
> >> Errors: total 2 client-timo 0 socket-timo 2 connrefused 0 connreset 0
> >> Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
> >> _________________________Output Ends___________________________________
> >>
> >> Here the "fd-anvail" errors is zero, and the system had to open max 523
> >> concurrent connections. That means it has not reached the max no of the
> >> system possible (1024 or so).. Then also it is giving a time lag of some
> >> 30 secs.
> >> Does any explaination exists for this ?
> >>
> >> While I also observed that at lower rates, it doesnt take extra time.
> >> So,
> >> that means HTTPERF TAKES EXTRA TIME AT HIGHER RATES, due to which the
> >> request rate and throughput values of httperf are wrong.
> >> Am I right or wrong ?
> >> Please rectify me, if I am wrong.
> >>
> >> It will be my pleasure to provide you any other information required
> >> regarding my expt.
> >>
> >> Thanking you a lot for your precious responses,
> >>
> >> Sincerely yours,
> >> Prakhar Goyal
> >> IIT Bombay, India.
> >>
> >> > Prakhar,
> >> >
> >> > As Martin suggested, your tests are not running "cleanly".
> >> >
> >> > 1)  The server seems to be overloaded. (significant number of socket
> >> > timeouts, 51 seconds to open a connection with the server,  16 seconds
> >> > to receive reply from the server)
> >> > 2)  httperf is not able to find enough unused file descriptors to
> >> > achieve the connection rate you have specified - there is a
> >> significant
> >> > number of  fails on fd-unavail.  It looks like your client machine has
> >> a
> >> > limit of 1024 for the maximum number of file descriptors - you can
> >> > notice that the max # of concurrent connections that could be opened
> >> was
> >> > 1022.  You can fix this problem by referring to the following link:
> >> >
> >> > http://devel.squid-cache.org/hno/linux-lfd.html
> >> >
> >> > Diwakar.
> >> >
> >> > Prakhar Goyal wrote:
> >> >
> >> >>Dear Sir,
> >> >>Thankyou very much for your prompt reply. I am sorry that I could not
> >> >>reply early as I was materialising the data.
> >> >>
> >> >>Here is the report given by httperf in one of my expts, which involves
> >> >> the
> >> >>performance analysis of web server:
> >> >>
> >> >>____________Report Starts______________________________
> >> >>
> >> >>httperf --server servername --port 80 --uri /model_prakhar/scene1.php
> >> >>--rate 22 --num-conns 2000
> >> >>httperf --client=0/1 --server=kahn.cse.iitb.ac.in --port=80
> >> >>--uri=/model_prakhar/scene1.php --rate=22 --send-buffer=4096
> >> >>--recv-buffer=16384 --num-conns=2000 --num-calls=1
> >> >>Maximum connect burst length: 1
> >> >>
> >> >>Total: connections 1799 requests 1627 replies 1623 test-duration
> >> 266.972
> >> >> s
> >> >>
> >> >>Connection rate: 6.7 conn/s (148.4 ms/conn, <=1022 concurrent
> >> >> connections)
> >> >>Connection time [ms]: min 307.8 avg 53450.4 max 142786.6 median
> >> 36645.5
> >> >>stddev 38681.8
> >> >>Connection time [ms]: connect 51866.2
> >> >>Connection length [replies/conn]: 1.000
> >> >>
> >> >>Request rate: 6.1 req/s (164.1 ms/req)
> >> >>Request size [B]: 94.0
> >> >>
> >> >>Reply rate [replies/s]: min 0.0 avg 6.1 max 9.4 stddev 3.8 (53
> >> samples)
> >> >>Reply time [ms]: response 16056.6 transfer 0.0
> >> >>Reply size [B]: header 206.0 content 504.0 footer 2.0 (total 712.0)
> >> >>Reply status: 1xx=0 2xx=1623 3xx=0 4xx=0 5xx=0
> >> >>
> >> >>CPU time [s]: user 5.44 system 261.53 (user 2.0% system 98.0% total
> >> >> 100.0%)
> >> >>Net I/O: 4.8 KB/s (0.0*10^6 bps)
> >> >>
> >> >>Errors: total 377 client-timo 0 socket-timo 172 connrefused 0
> >> connreset 4
> >> >>Errors: fd-unavail 201 addrunavail 0 ftab-full 0 other 0
> >> >>________________Report Ends_____________________________
> >> >>
> >> >>The client is a normal P4 machine having 2 CPUs.
> >> >>The errors indicate that some connections failed.I had fixed the
> >> timeout
> >> >>to be 100 secs.
> >> >>The service time of the web server is 112 ms.
> >> >>But after the web server finishes the handling requests,( ie, the
> >> machine
> >> >>on which web server is deployed, shows 0% cpu usage), then also upto 1
> >> >>minute(in this test case shown above), httperf continues running.
> >> >>
> >> >>I think that this affects the request rate and the thruput too.
> >> >>I know the 'request rate' shown by the htperf is diff from actual
> >> arrival
> >> >>rate. I had read this in your mailing list archive. :)
> >> >>
> >> >>Please reply to this matter soon as per your convenience.
> >> >>I will be happy to provide you any other information required.
> >> >>
> >> >>Waiting for the reply,
> >> >>
> >> >>Thanks & Regards,
> >> >>Prakhar Goyal.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>>hi Prakhar
> >> >>>
> >> >>>without knowing anything about the test you were running, or about
> >> the
> >> >>>configuration of the client, it is difficult to know what was
> >> happening.
> >> >>>
> >> >>>I expect that your test did not run cleanly in some way; e.g., some
> >> >>>connections failed, or httperf had trouble getting file descriptors
> >> at
> >> >>>some point during the test. However, without knowing more about the
> >> test
> >> >>>or the httperf output from the test this is just speculation.
> >> >>>
> >> >>>Martin
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >>
> >>
> >> --
> >> Prakhar Goyal
> >> Computer Science & Engineering Dept.
> >> IIT Bombay.
> >> Homepage: http://www.cse.iitb.ac.in/~prakhargoyal
> >> _______________________________________________
> >> httperf mailing list
> >> httperf at linux.hpl.hp.com
> >> http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/
> >>
> > _______________________________________________
> > httperf mailing list
> > httperf at linux.hpl.hp.com
> > http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/
> >
>
>
> --
> Prakhar Goyal
> Computer Science & Engineering Dept.
> IIT Bombay.
> Homepage: http://www.cse.iitb.ac.in/~prakhargoyal
>


More information about the httperf mailing list