[kaffe] Improving regression testing

[Dalibor Topic]

Hi,

I've been thinking recently about adding a
--with-test-jar
configuration flag, to aid those developers who are
cross-compiling,
or have to cope with kjc crashing its way through the
regression test
suite without a viable alternative.

This is the current regression test system:

* the tests are in test/regression. 

* kaffe uses automake's support for test scripts.

* every test is first compiled and then run

* you can pick a subset of tests to run

* you can turn on verbose test logging

* each test is a java file, with some extra comments
at the end

* these comments are used by the TestScript to
determine the expected
  output, whether to sort test results and what files
to compile
  additionally.


the drawbacks of this system are:

a) it is hard to run the tests on an installed version
of kaffe

b) especially after cross compiling

c) it depends on kjc working well enough to compile
the files

d) it is hard to process failure information


I'll explain the points in more detail:

a) kaffe doesn't install the regression test suite,
not even
optionally. Neither does it install the test script.
So if you want
to run the regression test suite with an installed
version of kaffe,
you have to put a bit of effort into it.

I'd like to have kaffe optionally install a test
script and a
tests.jar.

b) When you're cross-compiling, the version of kaffe
you're building
most probably can't run on the build system. That
means 'make check'
is not very useful.

If kaffe optionally installed the test script and the
test jar. you
could run the tests on your target. That *should* help
cross-compiling
developers to find & fix kaffe's bugs easier.

c) Currently, if the compiler has a problem compiling
a test, the
test fails. If you're using kjc to compile the tests
(the default),
it means that a rather complex piece of software fails
on the
compiled version of kaffe.

I believe that developers would be able to fix kaffe's
problems much
faster if they could focus on getting simple tests to
run, instead of
getting the complicated java compiler to work. I
assume that change
would make the life of porters to new platforms
easier.

(One such platform, where kjc breaks apart on kaffe
compiling the
regression test suite is Cygwin, btw.)

d) In the current setup, when a test fails it produces
a
test-name.fail file, which can be diffed against the
respective
test-name.out file. It requires manual intervention,
knowledge of the
test setup and diff[1] to log failed tests. I assume
that's the reason
why we rarely see failed regression tests posted to
the mailing
list.

I think the test script should log failed tests and
diff them
automatically, proposing where to send them for
inspection and how to
go about debugging them yourself.


All of this could be achieved in the current
regression test
framework, I assume. If someone wants to go ahead and
implement it,
I'd like to hear from you.

I've done some research to see if someone else has
already built a
better wheel (i.e. test suite script).

The current most popular java testing solution, junit,
is definitely
*not* suited for the job. Testing kaffe using a test
script written
in Java has some 'hack value', but it would be quite
pointless when
you're porting kaffe and you've barely got HelloWorld
working. It
would lead to the same problems that compiling the
tests using kjc
has: it's nice when it works, it's very hard to debug
when it doesn't.

That brings us to DejaGNU, the other well known (among
GNUs) testing
solution. It looks very nice, but using DejaGNU means
that one has to
get DejaGNU to work on a system first. DejaGNU in turn
needs tcl and
expect to be installed. In my opinion 'make check'
should be
lightweight and perform its work with as little
external dependencies
as possible. that's why I don't think DejaGNU is
suited for the job.

The latest releases of autoconf come with a tool that
looks quite
suited for the job, though: GNU Autotest. Yet another
auto* tool,
barely documented in the autoconf manual. There is
quite good
documentation available in the CVS tree of
http://gnuprog2.sf.net . It
is apperently good enough for use as the test suite
script in GNU
autoconf, GNU m4 and GNU bison. According to the
documentation in the
Programming with GNU tools 2.ed book, It should
support a) - d) well.


So I'd like to hear your feedback on these ideas, as
usual. Especially
if you are cross-compiling, hacking the build system,
writing
regression tests or have some experience with
different test suite
scripts.


cheers,
dalibor topic

__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - forms, calculators, tips, more
http://taxes.yahoo.com/