[Gc] Allocating Executable Memory

BGB cr88192 at hotmail.com
Fri Jul 23 16:27:14 PDT 2010


just my quick thoughts:
in my GC (a different, custom GC), I use an alternate call to request that 
the allocation be made executable (like, one can allocate an object as 
normal memory, or as atomic, executable, or others).


as far as the implementation goes, I have some heap chunks which are 
allocated as executable, and others which are not (data only), and when 
allocating executable objects, it will allocate from chunks marked as 
executable (whereas, the normal allocator will try to avoid using executable 
chunks unless it can't find any free space in non-executable memory).

the main reason for keeping them separate is to try to allow DEP/NX/... to 
do its job.

mine also uses a cell+bitmap allocator for small objects (to improve memory 
density, as small objects tend to be common in my code and can easily eat up 
lots of additional memory with a traditional free-list allocator).

for similar reasons, special handling is also used for cons cells.
as well, some callbacks are provided to allow plugging manually-managed 
memory into the GC so that it can still trace through them, and also still 
do type-checking on custom memory.

this is because not everything is ideal for GC IME, and for some things it 
makes sense to manage memory in a more task-specialized manner. allowing 
callbacks to register some of this code allows not entirely abandoning the 
GC's facilities though. (partial reason: I have a C compiler, and trying to 
allocate and free large amounts of memory in short bursts didn't make the GC 
happy, so I have generally resorted to speciallized MM code for this).


I also use JIT, but for my uses ended up opting with using an in memory 
assembler (where I produce textual ASM and feed it into the assembler). it 
seems to work fine for my purposes, and abstracts away some of the low level 
machine-specific details (x86 and x86-64 opcode-encoding funkiness) from the 
compiler machinery. (also producing textual ASM is more intuitive and less 
effort than producing ASM via masses of API calls).

arguments have come up though regarding performance (like whether one can 
afford to produce textual ASM and convert this to machine code), but IME it 
works without issue (I don't produce anywhere near the volumes of ASM where 
I would need to worry about this). (it pulls off around 10MB/s or so on my 
system).

(I guess YASM can do similar, but admittedly I have not looked into 
this...).


a much bigger performance matter is all of the XML-based DOM-like crap going 
on in the upper-end (most of my compiler logic is based on working with XML 
nodes...).

note: I use XML, rather than lists / s-exps, for most of the compiler logic, 
mostly because I was running into flexibility problems with s-exps. it is 
much easier to add new information (such as line-numbers, side-info, ...) to 
or tweak the structure of XML (like, the same node, but adding or removing 
children without effecting existing code), whereas s-exps tend to be much 
more "brittle" in these regards.

however, s-exps are much better WRT memory density and performance (in 
general), whereas XML/DOM/... is much better at eating up lots of time and 
memory...

so, it is tradeoffs (and not likely to matter as much for parsing most non-C 
languages...).


dunno if any of this is relevant or helpful...


----- Original Message ----- 
From: "Ivan Maidanski" <ivmai at mail.ru>
To: "Noah Lavine" <noah.b.lavine at gmail.com>
Cc: <gc at linux.hpl.hp.com>
Sent: Friday, July 23, 2010 1:10 PM
Subject: Re[2]: [Gc] Allocating Executable Memory


>
> The patch is incomplete. gc.h should be affected too. Also supply better 
> doc comments (in gc.h). Update NO_EXECUTE_PERMISSION documentation. Send 
> me Changelog entries please. Also, PROT_EXEC is not defined for every 
> platform (e.g. Win32 which supports executable code in the allocated 
> memory) - we shouldn't define a static variable which is always a const. 
> GC_get_pages_executable() should better actually return 1/0 instead of 
> exec_flag.
>
> Fri, 23 Jul 2010 15:21:34 -0400 Noah Lavine <noah.b.lavine at gmail.com>:
>
>> Thanks a lot!
>>
>> Here is a patch against the latest CVS which implements this. On my
>> computer libgc passes all tests with it applied.
>>
>> Noah
>>
>> On Fri, Jul 23, 2010 at 3:10 PM, Boehm, Hans <hans.boehm at hp.com> wrote:
>> > I don't mind this either. ?We've generally been moving static 
>> > configuration options to runtime wherever the performance impact was 
>> > minor. ?We didn't get around to this one yet.
>> >
>> > Hans
>> >
>> >> -----Original Message-----
>> >> From: gc-bounces at linux.hpl.hp.com [mailto:gc-bounces at linux.hpl.hp.com]
>> >> On Behalf Of Ivan Maidanski
>> >> Sent: Friday, July 23, 2010 11:40 AM
>> >> To: Noah Lavine
>> >> Cc: gc at linux.hpl.hp.com
>> >> Subject: Re: [Gc] Allocating Executable Memory
>> >>
>> >> Hello!
>> >>
>> >> I like the idea.
>> >> I don't mind adding this feature in this release (probably Hans
>> >> wouldn't neither since the added coded is small and easily 
>> >> verifiable).
>> >>
>> >> The API should be:
>> >> - GC_set_pages_executable(int) // non-zero means executable
>> >> - GC_get_pages_executable() // returns non-zero if it is allowed to
>> >> execute code in allocated memory
>> >>
>> >> NO_EXECUTE_PERMISSION controls the initial value only.
>> >>
>> >> If you'd like to add it, I expect you'll provide the patch against the
>> >> current CVS.
>> >>
>> >>
>> >> Fri, 23 Jul 2010 13:28:17 -0400 Noah Lavine <noah.b.lavine at gmail.com>:
>> >>
>> >> > Hello GC Developers,
>> >> >
>> >> > I am writing with a feature request for the next version of your
>> >> library.
>> >> >
>> >> > I am working on adding JIT compilation support to GNU Guile (a 
>> >> > Scheme
>> >> > implementation), which uses this library for garbage collection. In
>> >> > order to make GC work, we'll need to allocate executable memory.
>> >> > However, I have discovered that allocating executable memory is a
>> >> > build-time configuration option (in version 7.1, it appears to be 
>> >> > set
>> >> > in configure.ac, at lines 401 and 495). This gives Guile something 
>> >> > of
>> >> > a problem - currently we use whatever libgc is on a user's computer.
>> >> > However, the only way to ensure that libgc will allocate executable
>> >> > memory would be to build our own version of it, which would add 
>> >> > space
>> >> > to our executable and be redundant. We could also hack together our
>> >> > own memory allocator only for executable memory, but we like using
>> >> > your library. :-)
>> >> >
>> >> > Therefore, I'd like to request that the next version of the library
>> >> > have allocating executable memory as a runtime configuration option
>> >> > that can be set by programs. I don't think it would affect
>> >> performance
>> >> > very much - in fact, I can find only two uses of it in the entire
>> >> > program (calls to mmap and mprotect in os_dep.c).
>> >> >
>> >> > I also have a question - is there some way to make an older version
>> >> of
>> >> > libgc manage executable memory anyway? For instance, if I mmap some
>> >> > pages of executable memory, will libgc scan those for pointers even
>> >> > though it didn't allocate them? Are there other ways to work around
>> >> > this?
>> >>
>> >> See GC_add_roots.
>> >>
>> >> >
>> >> > Thank you very much
>> >> > Noah Lavine
>> >> > _______________________________________________
>> >> > Gc mailing list
>> >> > Gc at linux.hpl.hp.com
>> >> > http://www.hpl.hp.com/hosted/linux/mail-archives/gc/
>> >>
>> >> _______________________________________________
>> >> Gc mailing list
>> >> Gc at linux.hpl.hp.com
>> >> http://www.hpl.hp.com/hosted/linux/mail-archives/gc/
>> >
>>
>>
>> ATTACHMENT: application/x-patch 
>> add_executable_option.patch_______________________________________________
>> Gc mailing list
>> Gc at linux.hpl.hp.com
>> http://www.hpl.hp.com/hosted/linux/mail-archives/gc/
>
> _______________________________________________
> Gc mailing list
> Gc at linux.hpl.hp.com
> http://www.hpl.hp.com/hosted/linux/mail-archives/gc/
> 



More information about the Gc mailing list