[kaffe] Improving regression testing

[Timothy Stack]

> Hi,

hi,

> I've been thinking recently about adding a
> --with-test-jar
> configuration flag, to aid those developers who are
> cross-compiling,
> or have to cope with kjc crashing its way through the
> regression test
> suite without a viable alternative.
> 
> This is the current regression test system:
> [...]
> the drawbacks of this system are:
> 
> a) it is hard to run the tests on an installed version
> of kaffe

I don't see this as that big of a problem.  Certainly not large enough to 
sink time into.

> b) especially after cross compiling
> 
> c) it depends on kjc working well enough to compile
> the files

Agreed

> d) it is hard to process failure information

Agreed

> I'll explain the points in more detail:
> 
> b) When you're cross-compiling, the version of kaffe
> you're building
> most probably can't run on the build system. That
> means 'make check'
> is not very useful.

For the JanosVM we added a 'remote test' mode to TestScript.  Basically,
you would run 'make check' on a FreeBSD/Linux machine, but the tests were
actually run on a remote machine running the JanosVM on bare hardware.  
Each test loop worked like this:

	compile class file locally
	upload file to test host
	execute test on test host
	download result files
	compare actual results against the expected ones locally

This reduced the requirements on the test host since it doesn't require 
make, diff, javac/jikes, etc...

> If kaffe optionally installed the test script and the
> test jar. you
> could run the tests on your target. That *should* help
> cross-compiling
> developers to find & fix kaffe's bugs easier.

Well, you still need the source files to compare the output, it just seems
like a lot of baggage to install.

> c) Currently, if the compiler has a problem compiling
> a test, the
> test fails. If you're using kjc to compile the tests
> (the default),
> it means that a rather complex piece of software fails
> on the
> compiled version of kaffe.

I always figured that anyone doing a port/lowlevel stuff would have jikes 
installed and then this isn't a problem.

> I believe that developers would be able to fix kaffe's
> problems much
> faster if they could focus on getting simple tests to
> run, instead of
> getting the complicated java compiler to work. I
> assume that change
> would make the life of porters to new platforms
> easier.

I added the 'internal' test directory to the JanosVM for exactly this 
purpose.  Although, the only test in it at the moment is for the jitter.

I would also like a 'profile' test directory that did some basic 
measurements of kaffe's behavior.  For example, how much memory does it 
use to run HelloWorld or measured how many times a method could recurse 
before a StackOverflowError was thrown.

> (One such platform, where kjc breaks apart on kaffe
> compiling the
> regression test suite is Cygwin, btw.)
> 
> d) In the current setup, when a test fails it produces
> a
> test-name.fail file, which can be diffed against the
> respective
> test-name.out file. It requires manual intervention,
> knowledge of the
> test setup and diff[1] to log failed tests. I assume
> that's the reason
> why we rarely see failed regression tests posted to
> the mailing
> list.

> I think the test script should log failed tests and
> diff them
> automatically,

Agreed

> proposing where to send them for
> inspection and how to
> go about debugging them yourself.

The latest automake will tell to send an email to the maintainer address 
when there's a failure.

> 
> All of this could be achieved in the current
> regression test
> framework, I assume. If someone wants to go ahead and
> implement it,
> I'd like to hear from you.
> 
> I've done some research to see if someone else has
> already built a
> better wheel (i.e. test suite script).

I'd hesitate to make too many changes, the current stuff works well 
enough, it just needs to be fleshed out a bit.

> cheers,
> dalibor topic

tim stack