[httperf] Why does httperf take extra time even after completion

Prakhar Goyal prakhargoyal at cse.iitb.ac.in
Thu Jul 13 11:17:34 PDT 2006


Dear Sir,
Thank you very much for the explanation you have given for my problem.

Actually in my last expts, I had not specified the timeout values, since I
wanted the server to be overloaded. But now I have clearly understood that
for the correct performance analysis, the specification of timeout value
is quite essential.

Therefore, I performed the same expt, but now, the timeout was kept as
30secs, and the no of conns was increased so as to 'smoothen' the effect
of the 'cooling down' and 'warming up' of the server, so that the steady
state will be maintained for a longer time.
There, I found that the HTTPERF DOES NOT TAKE EXTRA TIME. So, my problem
has been solved.

Thus , I like to mention that for the performnace analysis in overloading,
with the help of httperf should be done with the specification of
appropriate timeout. HTTPERF gives correct outputs of throughput and
response time.

In addition to this, I like to say that the httperf would give better
(more exact) request rate and thruput(reply rate ,here), if there is also
a provision available in the httperf to mention the 2 time values by the
user (starting time and the finishing time), and the httperf calculates
the desired request rate & reply rate based on the performance during that
time period.
So, if the 2 time values happen to be the time values, when the steady
state has begun and finished respectively, then the calculation done on
that duration will give the exact desired values.

Really saying sir, this mailing list is very benificial for us.

Thanking you a lot ,

Yours sincerely,
Prakhar Goyal
India.

> hi Prakhar
>
> as I'm sure you are aware, there are many potential bottlenecks in a
> distributed system.  when you only have measurements from one vantage
> point (and from one experiment), it can make it difficult to assess the
> exact bottleneck.
>
> for the example you just sent, it appears that the problem is not with
> httperf.  the problem could be the Web server software (e.g., helper
> processes exhausted), the Web server hardware (e.g., cpu exhausted), or
> the client hardware (e.g., cpu exhausted) (I expect it is the first, or
> perhaps the second) I am basing this on the following lines from the
> httperf output:
>
>> Connection time [ms]: min 147.7 avg 19278.7 max 104047.5 median 13140.5
>> stddev 18025.2
>
> as both the avg and max connection time indicate, some of the TCP
> connections are open for a very long time (upto 104 seconds).  This
> seems like a long time for the requested php page to run.
>
> httperf will not exit until all of the requests it initiated have either
> completed or timed out.  since you did not specify a timeout value, there
> is no cap on how long httperf will wait for the server response (i.e., the
> default timeout value is infinity).
>
> I hope this helps you identify the actual bottleneck.
>
> Martin
>
> On Mon, 10 Jul 2006, Prakhar Goyal wrote:
>
>> Dear Sir,
>> Thank you for the explanation.
>> But I am mentioning one instance here, in which I faced the
>> contradiction.
>>
>> Here is the output of httperf:
>> ________________Output Starts_________________________________________
>>
>> httperf --server kahn.cse.iitb.ac.in --port 80
>> --uri/model_prakhar/scene1.php --rate 17 --num-conns 2700
>>
>> httperf --client=0/1 --server=kahn.cse.iitb.ac.in --port=80
>> --uri=/model_prakhar/scene1.php --rate=17 --send-buffer=4096
>> --recv-buffer=16384 --num-conns=2700 --num-calls=1
>> Maximum connect burst length: 1
>>
>> Total: connections 2700 requests 2698 replies 2698 test-duration 249.089
>> s
>>
>> Connection rate: 10.8 conn/s (92.3 ms/conn, <=523 concurrent
>> connections)
>> Connection time [ms]: min 147.7 avg 19278.7 max 104047.5 median 13140.5
>> stddev 18025.2
>> Connection time [ms]: connect 10471.9
>> Connection length [replies/conn]: 1.000
>>
>> Request rate: 10.8 req/s (92.3 ms/req)
>> Request size [B]: 94.0
>>
>> Reply rate [replies/s]: min 0.0 avg 11.0 max 14.2 stddev 5.3 (49
>> samples)
>> Reply time [ms]: response 8939.0 transfer 0.0
>> Reply size [B]: header 225.0 content 505.0 footer 2.0 (total 732.0)
>> Reply status: 1xx=0 2xx=2698 3xx=0 4xx=0 5xx=0
>>
>> CPU time [s]: user 12.02 system 237.06 (user 4.8% system 95.2% total
>> 100.0%)
>> Net I/O: 8.7 KB/s (0.1*10^6 bps)
>>
>> Errors: total 2 client-timo 0 socket-timo 2 connrefused 0 connreset 0
>> Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
>> _________________________Output Ends___________________________________
>>
>> Here the "fd-anvail" errors is zero, and the system had to open max 523
>> concurrent connections. That means it has not reached the max no of the
>> system possible (1024 or so).. Then also it is giving a time lag of some
>> 30 secs.
>> Does any explaination exists for this ?
>>
>> While I also observed that at lower rates, it doesnt take extra time.
>> So,
>> that means HTTPERF TAKES EXTRA TIME AT HIGHER RATES, due to which the
>> request rate and throughput values of httperf are wrong.
>> Am I right or wrong ?
>> Please rectify me, if I am wrong.
>>
>> It will be my pleasure to provide you any other information required
>> regarding my expt.
>>
>> Thanking you a lot for your precious responses,
>>
>> Sincerely yours,
>> Prakhar Goyal
>> IIT Bombay, India.
>>
>> > Prakhar,
>> >
>> > As Martin suggested, your tests are not running "cleanly".
>> >
>> > 1)  The server seems to be overloaded. (significant number of socket
>> > timeouts, 51 seconds to open a connection with the server,  16 seconds
>> > to receive reply from the server)
>> > 2)  httperf is not able to find enough unused file descriptors to
>> > achieve the connection rate you have specified - there is a
>> significant
>> > number of  fails on fd-unavail.  It looks like your client machine has
>> a
>> > limit of 1024 for the maximum number of file descriptors - you can
>> > notice that the max # of concurrent connections that could be opened
>> was
>> > 1022.  You can fix this problem by referring to the following link:
>> >
>> > http://devel.squid-cache.org/hno/linux-lfd.html
>> >
>> > Diwakar.
>> >
>> > Prakhar Goyal wrote:
>> >
>> >>Dear Sir,
>> >>Thankyou very much for your prompt reply. I am sorry that I could not
>> >>reply early as I was materialising the data.
>> >>
>> >>Here is the report given by httperf in one of my expts, which involves
>> >> the
>> >>performance analysis of web server:
>> >>
>> >>____________Report Starts______________________________
>> >>
>> >>httperf --server servername --port 80 --uri /model_prakhar/scene1.php
>> >>--rate 22 --num-conns 2000
>> >>httperf --client=0/1 --server=kahn.cse.iitb.ac.in --port=80
>> >>--uri=/model_prakhar/scene1.php --rate=22 --send-buffer=4096
>> >>--recv-buffer=16384 --num-conns=2000 --num-calls=1
>> >>Maximum connect burst length: 1
>> >>
>> >>Total: connections 1799 requests 1627 replies 1623 test-duration
>> 266.972
>> >> s
>> >>
>> >>Connection rate: 6.7 conn/s (148.4 ms/conn, <=1022 concurrent
>> >> connections)
>> >>Connection time [ms]: min 307.8 avg 53450.4 max 142786.6 median
>> 36645.5
>> >>stddev 38681.8
>> >>Connection time [ms]: connect 51866.2
>> >>Connection length [replies/conn]: 1.000
>> >>
>> >>Request rate: 6.1 req/s (164.1 ms/req)
>> >>Request size [B]: 94.0
>> >>
>> >>Reply rate [replies/s]: min 0.0 avg 6.1 max 9.4 stddev 3.8 (53
>> samples)
>> >>Reply time [ms]: response 16056.6 transfer 0.0
>> >>Reply size [B]: header 206.0 content 504.0 footer 2.0 (total 712.0)
>> >>Reply status: 1xx=0 2xx=1623 3xx=0 4xx=0 5xx=0
>> >>
>> >>CPU time [s]: user 5.44 system 261.53 (user 2.0% system 98.0% total
>> >> 100.0%)
>> >>Net I/O: 4.8 KB/s (0.0*10^6 bps)
>> >>
>> >>Errors: total 377 client-timo 0 socket-timo 172 connrefused 0
>> connreset 4
>> >>Errors: fd-unavail 201 addrunavail 0 ftab-full 0 other 0
>> >>________________Report Ends_____________________________
>> >>
>> >>The client is a normal P4 machine having 2 CPUs.
>> >>The errors indicate that some connections failed.I had fixed the
>> timeout
>> >>to be 100 secs.
>> >>The service time of the web server is 112 ms.
>> >>But after the web server finishes the handling requests,( ie, the
>> machine
>> >>on which web server is deployed, shows 0% cpu usage), then also upto 1
>> >>minute(in this test case shown above), httperf continues running.
>> >>
>> >>I think that this affects the request rate and the thruput too.
>> >>I know the 'request rate' shown by the htperf is diff from actual
>> arrival
>> >>rate. I had read this in your mailing list archive. :)
>> >>
>> >>Please reply to this matter soon as per your convenience.
>> >>I will be happy to provide you any other information required.
>> >>
>> >>Waiting for the reply,
>> >>
>> >>Thanks & Regards,
>> >>Prakhar Goyal.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>>hi Prakhar
>> >>>
>> >>>without knowing anything about the test you were running, or about
>> the
>> >>>configuration of the client, it is difficult to know what was
>> happening.
>> >>>
>> >>>I expect that your test did not run cleanly in some way; e.g., some
>> >>>connections failed, or httperf had trouble getting file descriptors
>> at
>> >>>some point during the test. However, without knowing more about the
>> test
>> >>>or the httperf output from the test this is just speculation.
>> >>>
>> >>>Martin
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >
>> >
>>
>>
>> --
>> Prakhar Goyal
>> Computer Science & Engineering Dept.
>> IIT Bombay.
>> Homepage: http://www.cse.iitb.ac.in/~prakhargoyal
>> _______________________________________________
>> httperf mailing list
>> httperf at linux.hpl.hp.com
>> http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/
>>
> _______________________________________________
> httperf mailing list
> httperf at linux.hpl.hp.com
> http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/
>


-- 
Prakhar Goyal
Computer Science & Engineering Dept.
IIT Bombay.
Homepage: http://www.cse.iitb.ac.in/~prakhargoyal


More information about the httperf mailing list