SPECjvm98

[Dalibor Topic]

On Thursday 21 March 2002 10:24, Jukka Santala wrote:
> On Tue, 19 Mar 2002, Dalibor Topic wrote:
> > Speaking of benchmarking suites, there is Ashes from the SableVM people.
> > http://www.sable.mcgill.ca/ashes/
> > I think it's open source, so I'd prefer that. :)
>
> Okay, thanks, I'll take a look at that. There are also some open source
> conformance test-suites, I believe.

There is mauve. There is jacks (compiler conformance). Then there are the 
regression test suites of various implementations, like libgcj's. I have run 
kaffe with mauve. I haven't looked into other conformance test-suites.

> Also, it does give you a rough idea on conformance if the benchmarks break
> too badly :) I was thinking more in the terms of "breaking some
> optimization". The problem with optimizations is that they're quite
> platform specific and sometimes longer to run than simple conformance/
> regression tests, so they're difficult for a single developer to run. In
> addition Kaffe has pretty good conformance, but it would appear
> performance could use some work.

Optimizations in core & library code should be beneficial to all platforms. 
But I guess you are not talking about those, since they are not platform 
specific.

I don't think I understood the term "breaking some optimization" properly. Do 
you mean "breaking some benchmark" ? If so, yes, meaningful benchmarks will 
take longer to run than conformance tests in general. You have to run a 
benchmark for some time to avoid measuring side effects like disk I/O, VM 
startup etc. Unless of course that's exactly what you want to measure :) 
Conformance tests usually run for a couple of seconds each.

In general, if people run kaffe on benchmarks & put the results on the web, I 
assume that Jim will link to them. If people volunteer to do so regularly, 
like the nightly builds done by the flex people, someone could write a couple 
of scripts to collect & present the information in a nice way. All it takes 
is to pick the benchmarks, find volunteers and agree on the procedure :)

I could submit results on x86-linux, interpreter/jit/jit3. But that's a rather 
common platform, and not that interesting, I guess. Unless there is a massive 
amount of interest in seeing how kaffe performs on old, slow computers :)

dalibor topic



_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com