SPECjvm98

[Jukka Santala]

On Tue, 19 Mar 2002, Dalibor Topic wrote:
> Speaking of benchmarking suites, there is Ashes from the SableVM people.
> http://www.sable.mcgill.ca/ashes/
> I think it's open source, so I'd prefer that. :)

Okay, thanks, I'll take a look at that. There are also some open source
conformance test-suites, I believe.

> I like the idea, but that only gives us a rough estimate about performance, 
> not conformance.  I think that running a performance benchmark on a 
> web/cvs/ftp/build server under load will produce rough results most of the 
> time.

Good catch; no, obiviously you can't run the benchmark on a machine being
used for something else at the time and expect to get reasonable results.
However, since you'll have to run the benchmark on different architectures
anyway, you might as well run them on separate machines that are idle
overnight or the like.

Also, it does give you a rough idea on conformance if the benchmarks break
too badly :) I was thinking more in the terms of "breaking some
optimization". The problem with optimizations is that they're quite
platform specific and sometimes longer to run than simple conformance/
regression tests, so they're difficult for a single developer to run. In
addition Kaffe has pretty good conformance, but it would appear
performance could use some work.

 -Jukka Santala