[httperf] a sniffer to create the session file for httperf

[anoop aryal]

hi,
a while ago, i had posted a link on this mailing list to a script that used 
tcpflow to create the session file for httperf by sniffing the wire while 
real people used the website which resulted in real think-times etc. it had 
several shortcomings but it worked fine for me at that point in time. i'm 
working on an improved version of that and there is one issue that i'm having 
trouble with.

i'd like to create sub-requests as supported by httperf. (the script does this 
too but the way it does it is not fool proof.) how can i distinguish a 'sub 
request' from the real request? i thought about using:

1) referrer. but all subsequent clicks would also have a referrer and 
therefore would end up as sub-requests. ie. i don't think i can tell apart an 
image fetch due to an <img> tag from a click on an image link this way.

2) timed threshold. ie. if the requests are all within x amount of time, they 
are subrequests, otherwise, they are new requests. the problems are obvious. 
what is the right amount of time used for the threshold? it's too hackish.

3) parsing the response to see if the request string for subsequent requests 
match some part of the previous response. this would work but will be 
non-trivial to get right. (this is what i did with the original script and it 
still got certain things wrong.) also, it would probably figure all subequent 
clicks as subrequests because the click would be on a link in that response 
and as such, without careful html parsing, it would match.

4) treat all html as initial-requests and all non-html as sub-requests. again, 
if the link was to an image, this fails. it also fails if the html is part of 
an iframe or such.


ideas would be appreciated.

anoop
aaryal at foresightint.com