[httperf] SIGSEGV when multiple instances are running?

Jim Whitehead II jnwhiteh at gmail.com
Wed Feb 2 15:10:11 PST 2011


On Wed, Feb 2, 2011 at 10:06 PM, Arlitt, Martin <martin.arlitt at hp.com> wrote:
> Hi Jim
>
> A student I work with at the University of Calgary ran a test with two httperf instances on one client, but she could not replicate the problem. Could you please describe the client you are using, and provide a/the command line that results in the crash?

The project I've written is currently called autohttperf and the code
can be found on github:

https://github.com/jnwhiteh/autohttperf

It consists of two Go programs, autohttperf_daemon and autohttperf.
The former is the worker daemon whereas the first is the main client
command that is used to request new benchmarks. The premise is very
simple, any benchmarks you request will be split over multiple
different clients-- helping to distribute the load while still
maintaining saturation of the appropriate load.

The only prerequisites for using the package is httperf on any client
machines, and the Go programming language.

Once Go is installed and you've cloned the repository, you should be
able to run 'gomake' in both the client and server directories.

1. Spin up a web server, port and host can be specified
2. Spin up an instance of httperf_daemon

Run the autohttperf client with the following commandline switches:

./autohttperf --server <host or ip> --port <port if not 80> --manual
--numconns 5000 <host:port of the machine running the daemon>

So if you're doing this all on localhost with default ports:

./autohttperf --server localhost --port 80 --manual --numconns 5000
localhost:1717

This should succeed with no issues, and you will receive debug
messages on both client and server. Now run the following:

./autohttperf --server localhost --port 80 --manual --numconns 5000
localhost:1717 localhost:1717

This tells the client to connect to the localhost client twice and
distribute the load amongst those clients. It is in these cases that I
see the crash and can reliably reproduce this. Thinking it was
initially a problem with my use of exec.Run in Go, I changed the
command to use bash -c to execute the program, and fell back on
/usr/bin/env to see if that made a difference. Neither of these fixed
the issue, which pointed towards httperf.

For me, the issue happens mostly when I have two of the same client
listed as the final arguments, but I've just had to add a third in
order to trigger the problem.. if that helps.

You can see the error occurs by the following client messages:

2011/02/02 23:07:33 [localhost:1717:0] Got results
2011/02/02 23:07:33 [localhost:1717:0] Error state reported: Command
did not properly exit: signal 11

On the plus side, I've been using this all day to do some stress
testing of servers and its working fine as long as the workers are on
different machines.

I know its a bit of a twisted issue, but this is the only way I've
been able to reliably reproduce this crash. If I can be of assistance,
let me know.

--
Jim Whitehead
Oxford University Computing Laboratory


>
> Thanks
> Martin
>
>
>> -----Original Message-----
>> From: httperf-bounces at linux.hpl.hp.com [mailto:httperf-
>> bounces at linux.hpl.hp.com] On Behalf Of Jim Whitehead II
>> Sent: Wednesday, February 02, 2011 7:44 AM
>> To: httperf at linux.hpl.hp.com
>> Subject: [httperf] SIGSEGV when multiple instances are running?
>>
>> I'm currently writing a tool that helps to automate benchmarking using
>> httperf, distributing the load over a number of worker machines to generate
>> the appropriate load. Everything works fine as long as I only issue one
>> request to each worker machine, but as soon as I spawn two processes on
>> the same machine to perform the httperf benchmark, one or both processes
>> will frequently crash with a SIGSEGV. The core dump shows that this is
>> happening in the conn_inc_ref(conn) macro call on core.c:1172. This error
>> happens on both SVN trunk as well as 0.9.0 as downloaded from the HP ftp
>> server.
>>
>> The following shows the backtrace and the core dump itself (shown last)
>> #0  0x000000010000796c in core_loop () at core.c:1221
>> #1  0x00000001000044b2 in main (argc=14, argv=0x7fff5fbff5d8) at
>> httperf.c:971
>>
>> #0  0x000000010000796c in core_loop () at core.c:1221
>> 1221                        conn_inc_ref (conn);
>>
>> The problem is I have difficulty getting httperf to reproduce this problem on
>> its own; it seems that it only occurs when I am spawning new (multiple)
>> instances from my RPC server that is handling requests.
>>
>> The RPC server is written in Go, so in an attempt to better isolate the
>> problem, I tried running httperf using both bash -c and env. In each of these
>> cases, the problem persisted.
>>
>> My question is: Are there any known issues with running concurrent
>> instances of httperf on the same machine that might be causing this
>> problem? I'd rather have a nice working tool than one with a dirty caveat of
>> "httperf will crash is you do this, so don't".
>>
>> Beyond this snag, httperf has been instrumental in my benchmarking, so
>> thank you!
>>
>> - Jim
>> _______________________________________________
>> httperf mailing list
>> httperf at linux.hpl.hp.com
>> http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/
>
> _______________________________________________
> httperf mailing list
> httperf at linux.hpl.hp.com
> http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/
>



More information about the httperf mailing list