[Gc] Re: Want to find reason of heap size keep growth
hans.boehm at hp.com
Tue Jul 13 14:44:38 PDT 2010
As Ivan and Bruce have pointed out, large blocks cause two distinct kinds of problems:
1) Fragmentation. The collector tries to be reasonably clever about minimizing the effect and not unnecessarily breaking up large blocks, but it's not always perfect. And the best the collector can theoretically do in the worst case is an overhead of a factor of log(largest_obj_size/smallest_obj_size), which is not as good as what you might hope for. And I'm not sure the current algorithm actually guarantees even that in any sense.
2) The probability of a misidentified pointer (e.g. integer) appearing to point into the large block gets too high. The actual symptom of that is a bit different. The collector "blacklists" addresses in unallocated memory that it knows are "referenced" by such bogus pointers, and it avoids allocating there. If you try to allocated a large block and it can't find a sufficiently big hole between such blacklisted addresses you get this warning. To simplify bookkeeping, the collector actually blacklists entire blocks (often a page), not just bytes. The count of the number of blacklisted blocks is given below. It tends to be an underestimate, since the collector gives up on such blocks after a while, and just goes ahead and allocates them as pointerfree regions, a block in size. This avoids some internal overhead. Such blocks are expected to "leak", but will never be touched, and thus don't need physical memory. And they can't point to and thus retain other memory.
Bruce and Ivan already gave some common strategies for avoiding this. Others:
- Since there is a lot of blacklisting going on here, you might try to reduce that. Building the collector with PRINT_BLACK_LIST defined will tell you were the bogus pointers are coming from. It might be possible to change the representation of the offending fields so they don't look like addresses.
- It may be possible to allocate the large objects with GC_malloc_ignore_off_page, or turn off interior pointer recognition and use explicit GC_register_displacement calls.
If you were on a different platform, I'd suggest a 64-bit ABI.
The last PLDI proceedings has a couple of nice papers on efficiently representing large arrays using smaller sections (look for Schism and Z-rays), which may be applicable here.
> -----Original Message-----
> From: gc-bounces at linux.hpl.hp.com [mailto:gc-bounces at linux.hpl.hp.com]
> On Behalf Of biosli
> Sent: Monday, July 12, 2010 12:02 AM
> To: gc at linux.hpl.hp.com
> Subject: [Gc] Re: Want to find reason of heap size keep growth
> biosli <biosli at ...> writes:
> > Hi all,
> > There is a further test for heap size growth.
> > I checked GC_dump(), finding when heap size grows.
> > I got the following warnings:
> > GC Warning: Repeated allocation of very large block (appr. size
> > May lead to memory leak and poor performance.
> > Then the heap size grows.
> > I've used GC_MALLOC_ATOMIC_IGNORE_OFF_PAGE malloc large block(as
> > data array, or string).
> > Can you kindly advise how i can prevent this warning from happending?
> > Thanks for all you help.
> Hi all:
> I print "Heap sections" as following(use GC_dump):
> Section 18 from 00630000 to 00680000 0/320 blacklisted
> Section 19 from 00680000 to 006D8000 0/352 blacklisted
> Section 20 from 006E0000 to 00740000 0/384 blacklisted
> Section 21 from 00740000 to 007AA000 0/424 blacklisted
> Section 22 from 007B0000 to 00825000 0/468 blacklisted
> Section 23 from 003B0000 to 0046A000 9/744 blacklisted
> Section 24 from 00930000 to 009C3000 0/588 blacklisted
> Section 25 from 009D0000 to 00A61000 1/580 blacklisted
> Section 26 from 00A70000 to 00B44000 0/848 blacklisted
> Section 27 from 00B50000 to 00D84000 8/2256 blacklisted
> Section 28 from 00D90000 to 00FC3000 3/2252 blacklisted
> Section 29 from 00FD0000 to 01204000 117/2256 blacklisted
> Section 30 from 01210000 to 01443000 36/2252 blacklisted
> Could you tell me what the number before "blacklisted" meaning? Is
> there have
> correlation with large block malloc?
> And if there are objects in blacklist, how can I find them? Can I print
> chain about that?
> Thanks for all you help.
> Gc mailing list
> Gc at linux.hpl.hp.com
More information about the Gc