[: Re: [kaffe] What is the rationale for Kaffe's VM memory limits? (The "-mx" and "-ss" switches.)]

Fri Mar 14 21:01:01 PST 2003

----- Forwarded message from  -----

To: Dalibor Topic <robilad at yahoo.com>
Subject: Re: [kaffe] What is the rationale for Kaffe's VM memory limits? (The "-mx" and "-ss" switches.)

On Fri, Mar 14, 2003 at 05:55:21AM -0800, Dalibor Topic wrote:
> hi Mark,
> 
> --- Mark J Roberts <mjr at znex.org> wrote:
> > I am sick and tired of manually overriding the heap
> > size limit in
> > order to run Freenet without hitting the arbitrary
> > default 64MB
> > limit.
> 
> I think that's due to a bug in kaffe ('s class
> library) as JDK seems to be able to run Freenet in
> 30M, or so, according to what Greg posted here. I'm
> trying to track that one down with Greg, but it would
> be helpful if you joined in, as a Freenet developer.
> 
> I've looked at the gcstats outputs provided by Greg,
> and most of the Objects lying around are either
> java.lang.Objects or java.util.HashMap$Entries, with a
> bunch of freenet.fs.dir.* Objects following. Grepping
> throuh the source indicates that java.lang.Objects are
> used by freenet for synchronization, right?
> 
> My prime suspect at the moment are the
> HashMap.Entries, so I posted a patch to track down
> who's creating all these HashMap.Entries, hoping that
> would provide some clue where they are used, and why
> they don't go away ;)
> 
> The idea is to create a stack trace for every
> constructed instance of HashMap.Entry and map the
> trace to the number of instances created with the same
> stack trace. So the HashMap.Entry constructor
> increases the counter for its stack trace, or inserts
> it into the map.
Well, my immediate suspicion would be that: every once in a while,
freenet writes out the datastore index. In order to minimize the time
the datastore is locked for, we clone the index in a lock, and then
write it out from the clone. HashMap.clone() clones the individual items
as well as the table, so we end up with a whole lot of rather short 
lived objects. If some way could be found to make the clone() lazily
copy (refcounts?), this would greatly reduce the GC/allocation stress.
Freenet on Kaffe still suffers from mysterious unexplained pauses in
execution that cause much higher routingTime's (a very bad thing, it
causes the node to assume it is overloaded and reject requests) than
Sun; maybe this is caused by a combination of this and the fact that
Kaffe uses a monolithic GC that locks the whole VM.
> 
> The patch has (at least) one problem, it relies on
> Runtime.exit() to print the map of stack traces.
> Apparently it seems to be hard to get freenet to
> exit(). So I'm not sure how to proceed from here: if
> it is possible to tell a freenet node to shut down
> through exit(), that would be helpful, otherwise  I
> could add a thread that prints out the map in regular
> intervals.
> 
> The other course of action would be to replace kaffe's
> HashMap (if that's the culprit) with another
> implementation, let's say Classpath's, and see if that
> yields any benefits.
> 
> > I just don't understand why these options are even
> > there, or why
> > they are not unlimited by default. It is _breaking_
> > applications.
> 
> In theory, they are not necessary. In practice, it's
> rather unconvenient to have a maximum memory setting
> higher that the amount of available RAM. In that case,
> the gc might spend a lot of time asking the virtual
> memory manager to shuffle pages around, and that
> degrades performance severely. The gc kicks in when a
> certain percentage of memory is used.
> 
> The 64Mb limit is the same that Sun's using on the JDK
> 1.4.1 for example. See
> http://java.sun.com/j2se/1.4/docs/tooldocs/linux/java.html
> under -Xmx . So if you're getting problems because
> your application is running out of memory on kaffe,
> but works fine on Sun's JDK, then that's a bug in
> kaffe, in my opinion. Setting -mx to unlimited by
> default would just mask it ;)
> 
> Additionally, here's a blurb on usefullness of the
> java heap setting from the Java Server documentation (
> http://salamander.merit.edu/system/doc/administration/performance.html
> ):
> 
> "The heap limit helps determine when a garbage
> collection (GC) will take place. You can set the limit
> high to avoid a GC, but then you could end up paging.
> Generally, we have found that it is better to take the
> hit from a GC fairly often and to avoid paging. The
> other thing to keep in mind about garbage collection
> is that the entire server stops while it is going. So
> if you have an occasional, very long GC, the clients
> will hang for that time. That leads to more
> variability in service quality than more frequent but
> smaller GCs."
> 
> In short: let's fix the bug ;)
> 
> cheers,
> dalibor topic
> 
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Web Hosting - establish your business online
> http://webhosting.yahoo.com
> 
> _______________________________________________
> kaffe mailing list
> kaffe at kaffe.org
> http://kaffe.org/cgi-bin/mailman/listinfo/kaffe
> 

-- 
Matthew Toseland
toad at amphibian.dyndns.org/amphibian at users.sourceforge.net
Full time freenet hacker.
http://freenetproject.org/
Freenet Distribution Node (temporary) at http://80-192-4-36.cable.ubr09.na.blueyonder.co.uk:8889/0bIANZmR~K8/
ICTHUS.

----- End forwarded message -----

-- 
Matthew Toseland
toad at amphibian.dyndns.org/amphibian at users.sourceforge.net
Full time freenet hacker.
http://freenetproject.org/
Freenet Distribution Node (temporary) at http://80-192-4-36.cable.ubr09.na.blueyonder.co.uk:8889/0bIANZmR~K8/
ICTHUS.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://kaffe.org/pipermail/kaffe/attachments/20030314/bba0fed5/attachment-0003.pgp 