[Gc] "Too many heap Sections" using allocations that span multiple pages.

Alec Orr Alec.Orr at wbemsolutions.com
Tue Jan 15 16:06:21 PST 2008


Bruce:

Thank you for your suggestions on this.  We opted to improve large buffer 
management by reusing using large buffers and managing them, as ultimately, this 
is the most responsible choice.  We saw significant reductions in heap usage 
(from 30 - 60% depending on the test).  We observe that tweaking the 
GC_use_entire_heap and GC_dont_expand variables did help slightly; but for 
preventing our memory fragmentation issues, there was simply no replacement for 
large buffer management, especially with repeated allocation/frees against 
buffers that span several dozen or hundreds of pages.

I hope this thread solves others' future issues as well.

Thanks again for your assistance,
Alec



Bruce Hoult wrote:

> On Jan 5, 2008 8:40 AM, Alec Orr <Alec.Orr at wbemsolutions.com> wrote:
> 
>>Good Day everyone:
>>
>>We are doing stress testing of a multithreaded application using the GC 6.8 on
>>RH/SLES-10 Linux and Win32.  We are seeing a "Too Many Heap Sections" error on
>>Win32 during stress testing, we also see this error in
>>real world testing but it takes several days (or week) to hit.
> 
> 
> The number of heap sections is just a compile-time constant used to
> size a global array.  It can be trivially increased or with just a
> change to a few lines of code changed to a dynamically-allocated
> resizable array.  Just change to GC_heap_sects (ok,
> GC_arrays._heap_sects) to be a pointer and  GC_add_to_heap() to do a
> realloc() or similar on it instead of blowing up.
> 
> 
> 
>>During stress testing, our application has a need to intermittently allocate
>>very large blocks of memory (anywheres between 1 page, or as large as 200 - 500
>>MB) as we build up an XML response.
> 
> 
> This will cause severe fragmentation for any non-compacting memory
> allocator (including malloc()).  I'd really recommend that you limit
> your block size to some reasonable value and use a list or array or
> tree of blocks to store such huge strings.
> 
> 
> 
>>We can alleviate the issue completely by coding the large blocks to come from
>>another heap (the C-Runtime rather than the GC).  It appears that when a very
>>large block (greater than 1 page) is returned to the GC, that the GC may use the
>>large block as a new root for future, smaller allocations (please correct me if
>>I am wrong here).
> 
> 
> The GC has a setting that controls whether blocks of memory greater
> than the GC page size are reserved for future possible allocations of
> the same size, or broken up to fulfill smaller allocations when memory
> is otherwise exhausted (and the GC would otherwise ask the OS for
> more).
> 
> See: GC_use_entire_heap
> _______________________________________________
> Gc mailing list
> Gc at linux.hpl.hp.com
> http://www.hpl.hp.com/hosted/linux/mail-archives/gc/
> 
> 
> 


-- 
Alec Orr
Staff Engineer
http://wbemsolutions.com
978-947-3609


More information about the Gc mailing list