Re[39]: [Gc]: GC + Windows Mobile + Threads + Patch for WINCE

Ivan Maidanski ivmai at mail.ru
Fri Oct 23 08:55:44 PDT 2009


Hi!
Today I wrote:
> Hi!
> "Boehm, Hans" <hans.boehm at hp.com> wrote:
> > This looks semi-healthy to me.
> >
> > In the log you attempted to send earlier (which I can see, even if nobody else can), it looked like almost 50 MB out of 65 (30MB live + 19MB newly allocated) were in use when the heap was expanded.  With a bit of fragmentation, that seems quite plausible.
> 
> Yes, I believe the collector is not broken (but the collect/expand strategy is somewhat poor...).
> 
> >
> > However, I am a little surprised that it needed to expand the heap when it did.  It seems to me the almost 19MB newly allocated should have been far more than scanned_memory/GC_free_space_divisor, which should have been closer to 5 MB in incremental mode.  This was presumably with the default GC_free_space_divisor?
> 
> Yes, default GC_free_space_divisor (you can reproduce the test on Win32 if compile with -DALL_INTERIOR_POINTERS -DGC_THREADS -DTHREAD_LOCAL_ALLOC -DPARALLEL_MARK, w/o other define macros).
> 
> Since GC_incremental is on, GC_free_space_divisor matters only how much blocks_to_get on every GC_collect_or_expand (GC_should_collect and GC_try_to_collect_inner are never called).
> 
> 
> > If so, can you debug in the last GC_expand_heap_inner and make sure that this looks plausible?  GC_dump() output at that point, together with the stack trace to see how it decided to expand, would be interesting.
> 
> Nothing interesting, I think. GC_collect_or_expand is called GC_alloc_large (after doing GC_collect_a_little_inner).
> 
> >  The trace makes it seem that the previous GC was long finished at the time of the expansion, so it's unclear to me why it should have been out of space at that point.
> 
> I think the collect strategy should be changed a bit (if incremental): GC_collect_or_expand should ignore GC_incremental if GC_collect_or_expand is called from GC_alloc_large. Opinions?

This doesn't help. I've failed to find out why does this happen.

I've worked out a small patch for min_bytes_allocd() which, I think, is irrelevant to the problem, but it's not - the situation became worse (the problem occurs even if the collector is built with only -DALL_INTERIOR_POINTERS -DGC_THREADS on Win32).


Hans -
1. the patch (attached) itself seems to look good/useful, right?

2. I'd like you to try this test (CVS + this patch, compiled for Win32 (32-bit) with -DALL_INTERIOR_POINTERS -DGC_THREADS (or similar)) yourself. (I'm really a bit short of time next week.)

> 
> >
> > The previous GC was also started only after > 16MB of allocation, which seems high with 30MB live and a smallish root set.  That seems like it's much later than it should be.
> 
> Full GC is only initiated from GC_maybe_gc() which is, in turn, called from GC_collect_a_little_inner when collection is not in progress.
> 
> >
> > Hans
> >
> > > -----Original Message-----
> > > From: gc-bounces at napali.hpl.hp.com
> > > [mailto:gc-bounces at napali.hpl.hp.com] On Behalf Of Ivan Maidanski
> > > Sent: Sunday, October 18, 2009 11:12 PM
> > > To: gc at napali.hpl.hp.com
> > > Subject: Fw: Re[36]: [Gc]: GC + Windows Mobile + Threads +
> > > Patch for WINCE
> > >
> > > This post originally contains bigger log (more "block in use"
> > > entries - probably uninteresting) but the ML daemon says
> > > "Message body is too big" and offers me to cancel it (but the
> > > link is not working because that server still blocks IPs, so
> > > the original post may appear on the list)...
> > >
> > > -----Original Message-----
> > > > Hi!
> > > > "Boehm, Hans" <hans.boehm at hp.com> wrote:
> > > > > ...
> > > > > > > Is the output with "Unexpected heap growth - collector may
> > > > > > be broken" means the collector could not work?
> > > > > >
> > > > > > I believe this doesn't imply the GC is broken - the message
> > > > > > printed because the corresponding assert condition is too
> > > > > > restrictive (I had tried to relax it a little but failed, so I
> > > > > > leave it as-is).
> > > > > >
> > > > > I remain a bit nervous about this one, especially since
> > > the difference between total bytes allocated and final heap
> > > size seems very small.  I'd at least check GC_dump() output
> > > to make sure that the root set looks reasonable, i.e. that
> > > we're not scanning memory areas we shouldn't, and that the
> > > heap doesn't overlap with the root set.  If this is
> > > reproducible, it sometimes helps to set a breakpoint in one
> > > of the last calls to GC_expand_hp_inner to try to understand
> > > why the heap got so big.
> > > > >
> > > > > I guess an unusually large amount of blacklisting might also
> > > > > contribute.  That could again be caused by scanning
> > > memory regions
> > > > > we shouldn't really be scanning (code segments, graphics memory,
> > > > > etc.)
> > > > >
> > > > > Hans
> > > >
> > > > The problematic case (with which I had tried to relax the
> > > condition a
> > > > little but failed) for Win32/x86 (mingw): gcc -g
> > > > -DALL_INTERIOR_POINTERS -DGC_THREADS -DTHREAD_LOCAL_ALLOC
> > > > -DPARALLEL_MARK -Iinclude -Ilibatomic_ops\src tests\test.c *.c
> > > >
> > > > It fails at least on every 7th run on my 2-core box.
> > > >
> > > > I've inserted GC_dump() in check_heap_stats:
> > > >
> > > > ***Static roots:
> > > > From 0x419000 to 0x41a000 (temporary)
> > > >
> > > > From 0x41d000 to 0x42a000 (temporary) // if run failed From
> > > 0x41d000
> > > > to 0x42b000 (temporary) // if ok
> > > >
> > > > From 0x430000 to 0x44a000 (temporary)
> > > > From 0x44f000 to 0x453000 (temporary)
> > > > From 0x76359000 to 0x7635a000 (temporary) From 0x7682a000 to
> > > > 0x7682b000 (temporary) From 0x7682c000 to 0x7682f000
> > > (temporary) From
> > > > 0x769c6000 to 0x769c7000 (temporary) From 0x76a26000 to 0x76a27000
> > > > (temporary) From 0x76afe000 to 0x76b01000 (temporary) From
> > > 0x76b9b000
> > > > to 0x76b9c000 (temporary) From 0x76d77000 to 0x76d78000 (temporary)
> > > > From 0x76ec9000 to 0x76eca000 (temporary) From 0x76f62000 to
> > > > 0x76f63000 (temporary) From 0x77d04000 to 0x77d07000
> > > (temporary) From
> > > > 0x77d08000 to 0x77d0a000 (temporary) From 0x77e0e000 to 0x77e0f000
> > > > (temporary) From 0x77e10000 to 0x77e12000 (temporary) Total size:
> > > > 270336
> > > >
> > > > No intersections between data roots and heap sections present.
> > > >
> > > > The rest output is mostly different on each run (so, this
> > > is for a failed run):
> > > >
> > > > ***Heap sections:
> > > > Total heap size: 73539584
> > > > Section 0 from 0x340000 to 0x350000 0/16 blacklisted Section 1 from
> > > > 0x380000 to 0x390000 0/16 blacklisted Section 2 from 0x390000 to
> > > > 0x3a0000 0/16 blacklisted Section 3 from 0x3b0000 to 0x3c1000 0/17
> > > > blacklisted Section 4 from 0x3d0000 to 0x3e6000 0/22 blacklisted
> > > > Section 5 from 0x5a0000 to 0x5be000 0/30 blacklisted Section 6 from
> > > > 0x5c0000 to 0x5e8000 0/40 blacklisted Section 7 from 0x600000 to
> > > > 0x635000 0/53 blacklisted Section 8 from 0x1700000 to
> > > 0x1747000 0/71
> > > > blacklisted Section 9 from 0x1750000 to 0x17ae000 1/94 blacklisted
> > > > Section 10 from 0x17b0000 to 0x182e000 0/126 blacklisted Section 11
> > > > from 0x24c0000 to 0x2568000 0/168 blacklisted Section 12 from
> > > > 0x2570000 to 0x2650000 0/224 blacklisted Section 13 from
> > > 0x2650000 to
> > > > 0x277a000 0/298 blacklisted Section 14 from 0x2780000 to 0x290e000
> > > > 0/398 blacklisted Section 15 from 0x2910000 to 0x2b22000 0/530
> > > > blacklisted Section 16 from 0x2b50000 to 0x2e14000 0/708
> > > blacklisted
> > > > Section 17 from 0x2e50000 to 0x3200000 1/944 blacklisted Section 18
> > > > from 0x3460000 to 0x394a000 0/1258 blacklisted Section 19 from
> > > > 0x3b60000 to 0x41ed000 0/1677 blacklisted Section 20 from
> > > 0x41f0000 to
> > > > 0x49f0000 0/2048 blacklisted Section 21 from 0x3a0000 to
> > > 0x3b0000 0/16
> > > > blacklisted Section 22 from 0x4bf0000 to 0x53f0000 1/2048
> > > blacklisted
> > > > Section 23 from 0x33d0000 to 0x33f0000 0/32 blacklisted Section 24
> > > > from 0x57f0000 to 0x5ff0000 0/2048 blacklisted Section 25 from
> > > > 0x5ff0000 to 0x67f0000 1/2048 blacklisted Section 26 from
> > > 0x39e0000 to
> > > > 0x3a20000 0/64 blacklisted Section 27 from 0x4ad0000 to 0x4b50000
> > > > 0/128 blacklisted Section 28 from 0x53f0000 to 0x54f0000 0/256
> > > > blacklisted Section 29 from 0x67f0000 to 0x69f0000 0/512
> > > blacklisted
> > > > Section 30 from 0x6df0000 to 0x75f0000 0/2048 blacklisted
> > > >
> > > > ***Free blocks:
> > > > Free list 5 (Total size 20480):
> > > >         0x361f000 size 20480 not black listed Free list 33
> > > (Total size
> > > > 1036288):
> > > >         0x3b60000 size 188416 not black listed
> > > >         0x257f000 size 176128 not black listed
> > > >         0x263b000 size 172032 not black listed
> > > >         0x268c000 size 172032 not black listed
> > > >         0x26b7000 size 163840 not black listed
> > > >         0x1762000 size 163840 not black listed Free list 34 (Total
> > > > size 2088960):
> > > >         0x4cb2000 size 221184 not black listed
> > > >         0x41f0000 size 196608 not black listed
> > > >         0x4383000 size 212992 not black listed
> > > >         0x3c9d000 size 208896 not black listed
> > > >         0x30d0000 size 221184 not black listed
> > > >         0x2cfc000 size 217088 not black listed
> > > >         0x2fd1000 size 196608 not black listed
> > > >         0x2b50000 size 200704 not black listed
> > > >         0x17b1000 size 217088 not black listed
> > > >         0x51fd000 size 196608 not black listed Free list 35 (Total
> > > > size 471040):
> > > >         0x3677000 size 233472 not black listed
> > > >         0x25bb000 size 237568 not black listed Free list 36 (Total
> > > > size 270336):
> > > >         0x17ec000 size 270336 not black listed Free list 41 (Total
> > > > size 880640):
> > > >         0x5844000 size 446464 not black listed
> > > >         0x48df000 size 434176 not black listed Free list 44 (Total
> > > > size 1085440):
> > > >         0x4168000 size 544768 not black listed
> > > >         0x3021000 size 540672 not black listed Free list 45 (Total
> > > > size 1130496):
> > > >         0x5053000 size 557056 partially black listed
> > > >         0x36b2000 size 573440 not black listed Free list 46 (Total
> > > > size 589824):
> > > >         0x5262000 size 589824 not black listed Free list 47 (Total
> > > > size 1908736):
> > > >         0x2bbb000 size 630784 not black listed
> > > >         0x2e50000 size 630784 not black listed
> > > >         0x2f11000 size 647168 partially black listed Free list 53
> > > > (Total size 835584):
> > > >         0x387e000 size 835584 not black listed Free list 57 (Total
> > > > size 966656):
> > > >         0x373f000 size 966656 not black listed Free list 58 (Total
> > > > size 987136):
> > > >         0x4013000 size 987136 not black listed Free list 59 (Total
> > > > size 1015808):
> > > >         0x2808000 size 1015808 not black listed Free list 60 (Total
> > > > size 26005504):
> > > >         0x70ed000 size 3379200 not black listed
> > > >         0x744e000 size 1712128 not black listed
> > > >         0x5ad5000 size 1441792 not black listed
> > > >         0x4d54000 size 2801664 not black listed
> > > >         0x3460000 size 1343488 not black listed
> > > >         0x611a000 size 4493312 not black listed
> > > >         0x4221000 size 1110016 not black listed
> > > >         0x6df1000 size 2551808 not black listed
> > > >         0x67f0000 size 2097152 not black listed
> > > >         0x5308000 size 1998848 not black listed
> > > >         0x4574000 size 3076096 not black listed Total of 39292928
> > > > bytes on free list
> > > >
> > > > ***Blocks in use:
> > > > (kind(0=ptrfree,1=normal,2=unc.):size_in_bytes, #_marks_set)
> > > >
> > > (1:2544,1)(1:2272,1)(1:2176,1)(1:2272,1)(1:2544,1)(0:4096,1)(1:4264,1)
> > > > (1:2176,1)
> > > >
> > > (1:4264,1)(1:144,3)(1:2544,1)(1:16,31)(1:4264,1)(1:16,66!=69)(1:2648,1
> > > > )(1:2840,1)
> > > > ...
> > > >
> > > (1:2552,1)(1:2184,1)(1:2552,1)(1:2552,1)(1:2184,1)(1:2552,1)(1:2552,1)
> > > > (1:2184,1)
> > > >
> > > (1:2552,1)(1:2552,1)(1:2184,1)(1:2552,1)(1:2552,1)(1:2184,1)(1:2552,1)
> > > > (1:2552,1)
> > > >
> > > (1:2184,1)(1:2544,1)(1:2544,1)(1:2184,1)(1:2544,1)(1:2544,1)(1:2176,1)
> > > > blocks = 6904, bytes = 34246656
> > > > Completed 6 tests
> > > > Allocated 3481011 collectable objects
> > > > Allocated 1212 uncollectable objects
> > > > Allocated 7314102 atomic objects
> > > > Allocated 137564 stubborn objects
> > > > Finalized 13243/13243 objects - finalization is probably ok Total
> > > > number of bytes allocated is 577175056 Final heap size is 73539584
> > > > bytes Unexpected heap growth - collector may be broken Test failed
> > > >
> > > > The number of blocks floats from 2000 to 8000 (at least for
> > > successful runs).
> > > >
> > > > The test output for successful run:
> > > >
> > > > Completed 6 tests
> > > > Allocated 3437857 collectable objects
> > > > Allocated 1212 uncollectable objects
> > > > Allocated 7252954 atomic objects
> > > > Allocated 137396 stubborn objects
> > > > Finalized 13388/13397 objects - finalization is probably ok Total
> > > > number of bytes allocated is 552486152 Final heap size is 60952576
> > > > bytes Collector appears to work
> > > >
> > > > This is the place of log (for failed run) of the last heap
> > > expansion:
> > > >
> > > > --> Marking for collection 107 after 16530416 allocated bytes
> > > > Marked from 598 dirty pages
> > > > Collection 106 reclaimed 8188736 bytes ---> heapsize =
> > > 65138688 bytes
> > > > World-stopped marking took 31 msecs (21 in average) Heap contains
> > > > 30045648 pointer-containing + 221328 pointer-free reachable bytes
> > > > 8366 finalization table entries; 7909 disappearing links alive 0
> > > > objects are eligible for immediate finalization; 0 links cleared
> > > > Finalize + initiate sweep took 16 + 0 msecs -------------Finished
> > > > tree_test at time 4078 (0x208feb4) Increasing heap size by 8388608
> > > > after 18599072 allocated bytes ***>Full mark for collection
> > > 108 after
> > > > 20145864 allocd bytes
> > > >

Bye.


More information about the Gc mailing list