[httperf] file descriptors

Mark Nottingham mnot at yahoo-inc.com
Thu Aug 7 22:34:50 PDT 2008


On 30/07/2008, at 12:35 PM, Adrian Chadd wrote:
>
> Could you get oprofile going and generate some traces?

My box doesn't seem to want to do a callgraph, but here it is flat  
(fairly low rate testing for a few seconds):

~# opreport -l bin/httperf
CPU: P4 / Xeon with 2 hyper-threads, speed 3067.28 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not  
stopped) with a unit mask of 0x01 (mandatory) count 100000
samples  %        image name               symbol name
2360     39.0599  ld-2.3.4.so              _dl_sysinfo_int80
326       5.3956  libc-2.3.4.so            _IO_vfscanf
188       3.1116  httperf                  timer_now_forced
162       2.6812  libc-2.3.4.so            strchr
159       2.6316  httperf                  event_base_loop
154       2.5488  libc-2.3.4.so            memcpy
143       2.3668  httperf                  conn_handle_read_event
104       1.7213  libc-2.3.4.so            ____strtoul_l_internal
103       1.7047  httperf                  conn_handle_write_event
101       1.6716  httperf                  get_line
97        1.6054  httperf                  event_signal
85        1.4068  httperf                  timer_has_expired
85        1.4068  libc-2.3.4.so            gettimeofday
78        1.2910  httperf                  epoll_dispatch
74        1.2248  httperf                  parse_headers
73        1.2082  httperf                  epoll_add
69        1.1420  httperf                  http_process_reply_bytes
68        1.1255  libc-2.3.4.so            free
67        1.1089  httperf                  epoll_del
64        1.0593  httperf                  list_remove_if_true
63        1.0427  libc-2.3.4.so            malloc
63        1.0427  libc-2.3.4.so            strncasecmp
58        0.9599  httperf                  event_queue_remove
47        0.7779  httperf                  timer_schedule
45        0.7448  libc-2.3.4.so            epoll_wait
40        0.6620  httperf                  list_for_each
40        0.6620  httperf                  timer_deactivate
40        0.6620  libc-2.3.4.so            _int_malloc
39        0.6455  libc-2.3.4.so            _int_free
36        0.5958  httperf                  object_new
34        0.5627  httperf                  core_send
34        0.5627  libc-2.3.4.so            __i686.get_pc_thunk.cx
33        0.5462  httperf                  object_dec_ref
33        0.5462  httperf                  set_active_shared
32        0.5296  httperf                  event_queue_insert
32        0.5296  libc-2.3.4.so            __ctype_tolower_loc
27        0.4469  httperf                  recv_stop
27        0.4469  httperf                  timer_tick
27        0.4469  libc-2.3.4.so            __i686.get_pc_thunk.bx
27        0.4469  libc-2.3.4.so            __strtoul_internal
26        0.4303  httperf                  core_loop
26        0.4303  httperf                  event_del
26        0.4303  libc-2.3.4.so            writev
24        0.3972  httperf                  call_destroyed
23        0.3807  httperf                  event_tree_RB_MINMAX
23        0.3807  libc-2.3.4.so            _IO_str_init_static_internal
23        0.3807  libc-2.3.4.so            epoll_ctl
22        0.3641  httperf                  anonymous symbol from  
section .plt
22        0.3641  httperf                  is_list_empty
22        0.3641  httperf                  recv_done
21        0.3476  httperf                  epoll_recalc
21        0.3476  libc-2.3.4.so            __errno_location
20        0.3310  ld-2.3.4.so              do_lookup_x
19        0.3145  httperf                  event_loop
19        0.3145  libc-2.3.4.so            _IO_setb
18        0.2979  libc-2.3.4.so            _IO_sputbackc
18        0.2979  libc-2.3.4.so            vsscanf
17        0.2814  httperf                  gettime
17        0.2814  libc-2.3.4.so            sscanf
15        0.2483  httperf                  event_active
15        0.2483  libc-2.3.4.so            rawmemchr
13        0.2152  httperf                  list_push
13        0.2152  httperf                  timer_now
12        0.1986  httperf                  list_pop
12        0.1986  httperf                  set_uri
10        0.1655  httperf                  call_created
10        0.1655  httperf                  conn_write_clear
10        0.1655  httperf                  event_add
10        0.1655  httperf                  issue_calls
10        0.1655  httperf                  recv_start
10        0.1655  httperf                  tick
10        0.1655  libc-2.3.4.so            read
9         0.1490  httperf                  call_done
8         0.1324  httperf                  core_connect
8         0.1324  libc-2.3.4.so            _IO_old_init
7         0.1159  httperf                  send_stop
7         0.1159  ld-2.3.4.so              strcmp
6         0.0993  httperf                  conn_read_clear
6         0.0993  httperf                  core_close
6         0.0993  libc-2.3.4.so            _IO_no_init
5         0.0828  httperf                  conn_destroyed
5         0.0828  httperf                  parse_data
4         0.0662  httperf                  conn_init
4         0.0662  httperf                  send_start
4         0.0662  ld-2.3.4.so              _dl_lookup_symbol_x
4         0.0662  libc-2.3.4.so            __close_nocancel
4         0.0662  libc-2.3.4.so            __read_nocancel
3         0.0497  httperf                  conn_created
3         0.0497  httperf                  event_set
3         0.0497  httperf                  hash_code
3         0.0497  httperf                  set_active_read
3         0.0497  ld-2.3.4.so              _dl_relocate_object
3         0.0497  libc-2.3.4.so            strcmp
2         0.0331  httperf                  call_append_request_header
2         0.0331  httperf                  call_init
2         0.0331  httperf                  set_active_write
2         0.0331  httperf                  timer_cancel
2         0.0331  ld-2.3.4.so              _dl_load_cache_lookup
2         0.0331  ld-2.3.4.so              _dl_map_object_deps
2         0.0331  ld-2.3.4.so              _dl_map_object_from_fd
2         0.0331  libc-2.3.4.so            _dl_addr
2         0.0331  libc-2.3.4.so            fcntl
2         0.0331  libc-2.3.4.so            setsockopt
2         0.0331  libc-2.3.4.so            socket
1         0.0166  httperf                  conn_connected
1         0.0166  httperf                  conn_connecting
1         0.0166  httperf                  destroyed
1         0.0166  ld-2.3.4.so              _dl_important_hwcaps
1         0.0166  ld-2.3.4.so              _dl_name_match_p
1         0.0166  ld-2.3.4.so              _dl_sysdep_read_whole_file
1         0.0166  ld-2.3.4.so              malloc
1         0.0166  ld-2.3.4.so              match_symbol
1         0.0166  ld-2.3.4.so              munmap
1         0.0166  libc-2.3.4.so            ____strtod_l_internal
1         0.0166  libc-2.3.4.so            __default_morecore
1         0.0166  libc-2.3.4.so            __nss_lookup_function
1         0.0166  libc-2.3.4.so            __uflow
1         0.0166  libc-2.3.4.so            _nl_intern_locale_data
1         0.0166  libc-2.3.4.so            connect
1         0.0166  libc-2.3.4.so            fork
1         0.0166  libc-2.3.4.so            mbrtowc
1         0.0166  libc-2.3.4.so            mempcpy
1         0.0166  libc-2.3.4.so            set_binding_values
1         0.0166  libc-2.3.4.so            strlen
1         0.0166  libresolv-2.3.4.so       __libc_res_nsend



>> The biggest perf limit I've got now is that I can generate about 550
>> *connections* a second; past that gets me a connection failed with
>> unexpected error 99, which is ENOSTR (not a stream). This is Red Hat
>> Linux. Any thoughts here? I see this with both select and libevent,  
>> so
>> it's not the new code. It always happens around the same rate; it's
>> not dependent upon how long the test has run, or the timeout
>> (suggesting that it's not cumulative resource exhaustion a la
>> ephemeral ports).
>
> Cannot assign requested address (in another email); check your MSL  
> times
> and ephermeral port ranges.


Yeah, this has bitten me before, the confusing bit is why it doesn't  
show up in errors as addrunavail. I don't see that counter  
(num_sock_addrunavail) being incremented anywhere...


--
Mark Nottingham       mnot at yahoo-inc.com




More information about the httperf mailing list