[kaffe] Improving regression testing

[Dalibor Topic]

hi Tim,

--- Timothy Stack <stack@cs.utah.edu> wrote:

> > b) When you're cross-compiling, the version of
> kaffe
> > you're building
> > most probably can't run on the build system. That
> > means 'make check'
> > is not very useful.
> 
> For the JanosVM we added a 'remote test' mode to
> TestScript.  Basically,
> you would run 'make check' on a FreeBSD/Linux
> machine, but the tests were
> actually run on a remote machine running the JanosVM
> on bare hardware.  
> Each test loop worked like this:
> 
> 	compile class file locally
> 	upload file to test host
> 	execute test on test host
> 	download result files
> 	compare actual results against the expected ones
> locally
> 
> This reduced the requirements on the test host since
> it doesn't require 
> make, diff, javac/jikes, etc...

That sounds very good. a) was not really a big issue,
and that solves b) elegantly. I'll try to get it
merged in.

> > If kaffe optionally installed the test script and
> the
> > test jar. you
> > could run the tests on your target. That *should*
> help
> > cross-compiling
> > developers to find & fix kaffe's bugs easier.
> 
> Well, you still need the source files to compare the
> output, it just seems
> like a lot of baggage to install.

Yeah. I didn't think about the client-server approach,
you are using. It's the better solution in most cases.

> I added the 'internal' test directory to the JanosVM
> for exactly this 
> purpose.  Although, the only test in it at the
> moment is for the jitter.
> 
> I would also like a 'profile' test directory that
> did some basic 
> measurements of kaffe's behavior.  For example, how
> much memory does it 
> use to run HelloWorld or measured how many times a
> method could recurse 
> before a StackOverflowError was thrown.

sounds good to me. I'll get the jitter tests merged
in. there were some include file name changes the last
time around I tried, which broke the C part of the
tests. I'll follow them in kaffe's CVS to make further
core merges with Janos VM easier.

> I'd hesitate to make too many changes, the current
> stuff works well 
> enough, it just needs to be fleshed out a bit.

O.K., I'll give it a try. Being able to use prebuilt
tests might matter though, when the host compiler
miscompiles them (see the JanosVM Preempt test, which
breaks because of a kjc bug [1]).

As you've apparently done most of the hard work
already in JanosVM's regression test code, I'll use
that instead of reinventing the wheel. Thanks!

cheers,
dalibor topic

[1] on a side note, I've received a mail from a kjc
developer that he has confirmed the "kjc vs. mauve
Thread test" compilation bug, and is working on it.

__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - forms, calculators, tips, more
http://taxes.yahoo.com/