[httperf] HTTP Web Server performance limitaitons

Martin F. Arlitt httperf@napali.hpl.hp.com
Fri, 07 Sep 2001 20:48:58 +0000


Deepak

> I am requesting for a single empty HTML file of
> size 50 bytes or less. In the simplest case,
> the httperf options are as follows:
> --hog
> --server=linux03
> --uri=/docs/empty.html
> --timeout=3
> --rate=X  (X ranging from 100 to 1000)
> --num-conns=25000
> --num-calls=1
>
> The two machines are connected by Gigabit network.

from your description of the workload and the network,
it would appear that there is not a network bottleneck.
however, it never hurts to verify this, if you haven't already
done so.  I typically use netperf (www.netperf.org) to
verify that the network is functioning as expected.  I
test the TCP stream throughput as well as the request-
response throughput.

> I add the values appropriately to arrive at the final
> request anr reply rates. I have attached a file which
> gives the sample out put from httperf for two cases.
> One without timeout and another with timeout=3 sec.
> If there is something that can be inferred from this
> about where I might be going wrong, then it would be
> of lot of help to me.

looking at the httperf output below, I notice several problems,
although you seem to have picked up on these already:

in the first test the client ran out of file descriptors,
so it simply wasn't able to generate the requested load.
using a timeout value in the second set of tests lessened
this problem, but did not solve it.  you may need to reconfigure
your clients to support more file descriptors (or at least
be aware that your clients may be nearing their peak capacity).

another problem in the second test is the TCP connect time;
the average time to set up a TCP connection in the second
test was 266.3 & 345.9ms, compared to 1.7 & 4.4 ms for test 1.
assuming that the network isn't a problem (which it likely isn't),
it looks to me that the Web server simply isn't removing requests
quickly enough from the listen queue/backlog.  making the
listen queue larger will only delay how long it takes before
things go bad (i.e., make sure you run your tests long enough);
I would focus on the Apache parameters. I can't offer any insights
on setting those parameters, however.

Martin

>
> Thanks for your help.
> Deepak
>
> arlitt@hpl.hp.com wrote:
> > Deepak
> >
> > please provide some more information about your
> > experiment:
> >
> > what workload are you generating? are you
> > generating 600 requests/second for the same file
> > (if so, what size is the file)?  or are you requesting
> > a bunch of different files?  if so, what is the average
> > size of those files? are you requesting static files
> > or dynamic (e.g., cgi-bin) files?
> >
> > what type of network are you using (e.g., 100 Mb/s
> > switched Ethernet)?
> >
> > what httperf options are you using?
> >
> > if you could include the httperf output from the two
> > clients that may offer some insights as well (just for one
> > test to start with).
> >
> > Martin
> >
> > Deepak Hegde wrote:
> >
> > > Hi,
> > >
> > > I am running httperf to test the web server performance. The web
> server runs
> > > on a high end Linux machine with sufficiently large memory and disk
> space.
> > > However, I am able to achieve only about 600 connections per second,
> even with
> > > two clients running the httperf simultaneously. I have tried tuning
> the Web
> > > Server parameters (MaxClients, MaxSpareServers, MaxRequestsPerChild
> etc) and
> > > OS parameters (open file descriptors, tcp/apache listen backlogs, SYN
> Cookies
> > > etc) to increase the resource availability. The CPU seems to be pretty
> busy
> > > when the test are being run. This seems to be the limiting factor. Can
> anyone
> > > suggest other possible bottlenecks or how to detect them ?
> > >
> > > Thanks
> > >
> > > Deepak
> > >
> > >

~/utils > ./httperf --hog --server=linux03 --num-conns=10000 --num-calls=1 --rate=500
--uri=/docs/empty.html
httperf --hog --client=0/1 --server=linux03 --port=80 --uri=/docs/empty.html
--rate=500 --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 --num-calls=1
Maximum connect burst length: 37

Total: connections 7248 requests 7248 replies 7248 test-duration 41.798 s

Connection rate: 173.4 conn/s (5.8 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 3.7 avg 3246.7 max 28127.9 median 302.5 stddev 7190.5
Connection time [ms]: connect 4.4
Connection length [replies/conn]: 1.000

Request rate: 173.4 req/s (5.8 ms/req)
Request size [B]: 74.0

Reply rate [replies/s]: min 1.0 avg 180.9 max 375.2 stddev 149.2 (8 samples)
Reply time [ms]: response 3242.3 transfer 0.0
Reply size [B]: header 315.0 content 14.0 footer 0.0 (total 329.0)
Reply status: 1xx=0 2xx=7248 3xx=0 4xx=0 5xx=0

CPU time [s]: user 1.04 system 33.90 (user 2.5% system 81.1% total 83.6%)
Net I/O: 68.2 KB/s (0.6*10^6 bps)

Errors: total 2752 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 2752 addrunavail 0 ftab-full 0 other 0
------------------------------------------------------------------------
~/utils > ./httperf --hog --server=linux03 --num-conns=10000 --num-calls=1 --rate=500
--uri=/docs/empty.html
httperf --hog --client=0/1 --server=linux03 --port=80 --uri=/docs/empty.html
--rate=500 --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to
FD_SETSIZE
Maximum connect burst length: 48

Total: connections 6810 requests 6810 replies 6810 test-duration 51.281 s

Connection rate: 132.8 conn/s (7.5 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 29.8 avg 3573.6 max 45127.0 median 297.5 stddev 7682.7
Connection time [ms]: connect 1.7
Connection length [replies/conn]: 1.000

Request rate: 132.8 req/s (7.5 ms/req)
Request size [B]: 74.0

Reply rate [replies/s]: min 0.0 avg 136.0 max 336.7 stddev 139.1 (10 samples)
Reply time [ms]: response 3571.9 transfer 0.0
Reply size [B]: header 315.0 content 14.0 footer 0.0 (total 329.0)
Reply status: 1xx=0 2xx=6810 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.83 system 49.86 (user 1.6% system 97.2% total 98.8%)
Net I/O: 52.3 KB/s (0.4*10^6 bps)

Errors: total 3190 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 3190 addrunavail 0 ftab-full 0 other 0
============================================================================================

WITH TIMEOUT=3
~/utils > ./httperf --hog --server=linux03 --num-conns=10000 --num-calls=1 --rate=500
--uri=/docs/empty.html --timeout=3
httperf --hog --timeout=3 --client=0/1 --server=linux03 --port=80
--uri=/docs/empty.html --rate=500 --send-buffer=4096 --recv-buffer=16384
--num-conns=10000 --num-calls=1
Maximum connect burst length: 39

Total: connections 9665 requests 6224 replies 4708 test-duration 23.002 s

Connection rate: 420.2 conn/s (2.4 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 5.2 avg 729.0 max 4706.7 median 317.5 stddev 955.8
Connection time [ms]: connect 266.3
Connection length [replies/conn]: 1.000

Request rate: 270.6 req/s (3.7 ms/req)
Request size [B]: 74.0

Reply rate [replies/s]: min 189.7 avg 214.6 max 273.3 stddev 39.5 (4 samples)
Reply time [ms]: response 404.4 transfer 0.0
Reply size [B]: header 315.0 content 14.0 footer 0.0 (total 329.0)
Reply status: 1xx=0 2xx=4708 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.62 system 18.57 (user 2.7% system 80.7% total 83.4%)
Net I/O: 85.3 KB/s (0.7*10^6 bps)

Errors: total 5292 client-timo 4957 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 335 addrunavail 0 ftab-full 0 other 0
-----------------------------------------------------------------------------------------

~/utils > ./httperf --hog --server=linux03 --num-conns=10000 --num-calls=1 --rate=500
--uri=/docs/empty.html --timeout=3
httperf --hog --timeout=3 --client=0/1 --server=linux03 --port=80
--uri=/docs/empty.html --rate=500 --send-buffer=4096 --recv-buffer=16384
--num-conns=10000 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to
FD_SETSIZE
Maximum connect burst length: 51

Total: connections 9962 requests 7442 replies 5611 test-duration 23.089 s

Connection rate: 431.5 conn/s (2.3 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 196.4 avg 730.4 max 5858.2 median 301.5 stddev 1017.6
Connection time [ms]: connect 345.9
Connection length [replies/conn]: 1.000

Request rate: 322.3 req/s (3.1 ms/req)
Request size [B]: 74.0

Reply rate [replies/s]: min 211.6 avg 253.0 max 278.2 stddev 28.8 (4 samples)
Reply time [ms]: response 364.2 transfer 0.0
Reply size [B]: header 315.0 content 14.0 footer 0.0 (total 329.0)
Reply status: 1xx=0 2xx=5611 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.44 system 22.40 (user 1.9% system 97.0% total 98.9%)
Net I/O: 101.4 KB/s (0.8*10^6 bps)

Errors: total 4389 client-timo 4351 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 38 addrunavail 0 ftab-full 0 other 0



--
To unsubscribe: echo unsubscribe httperf | mail majordomo@linux.hpl.hp.com