Kaffe and Mauve (Was: Re: [kaffe] Re: LinkedList.subList severely broken)

[Dalibor Topic]

Salut Daniel,

--- Daniel Bonniot <Daniel.Bonniot at inria.fr> wrote:
> >
> >
> >time make check KEYS="java. \!java.lang.Character.unicode"
> >234 of 2329 tests failed
> >
> Ouch, that sounds like a lot of failures. Are most of those 
> unimplemented features, or implementation bugs? (the latter is much more 
> serious, because the program might run, but produce incorrect results, 
> and it will be difficult to track down).

I'm afraid it's the latter in most cases.

> Is it a lack of manpower that prevents from solving these failures?

That's pretty much it, noone volunteered to do a mauve failure cleanup yet. I
ocassionally find some time to fix a bug or two, and I've done a 1.0 failure
clean up last year.

If you'd like to have a go, I'd propose cleaning up the single JDK 1.0 failure,
and then proceeding to JDK 1.1, starting with lang, util, and then the rest of
the bunch. I'm sure Jim could grant you CVS write access.

> >real    0m51.953s
> >
> That is 0 minutes and 51 seconds, right? That's quite short.
> 
> Another problem with having so many failures is that it makes it more 
> difficult to spot if a change in Kaffe introduced regressions. Is there 
> an easy way to know that (the same for Kaffe's own suite: are some cases 
> know to fail at a given time, like now?).

Current failures for JDK 1.0 and JDK 1.1 are attached. We could use this as the
base line.
 
> >there is no automatic daily run. if you want to setup one, that would be
> really
> >cool. Check out automauve
> >http://mail.gnu.org/archive/html/classpath/2003-02/msg00001.html
> >
> I'm not sure I can find the time resources to do that. Maybe if it is a 
> straightforward adaptation of the existing framework.
> On what machine would be tests run?

Jim, would it be possible to set it up on kaffe.org?

> What would be useful would be a way to track changes in the results 
> (progress/regression), for each config (VM). Is there support for this 
> in Mauve?
> I remember seeing such a feature in the Jacks compiler testsuite.

Automauve. See http://www.gnu.org/software/classpath/mauve/ for an example.

> >That's fine with me, as I can run the mauve suite fairly quickly. If someone
> >sets up a daily mauve run, I don't think we would lose anything by 'only'
> >having the test only once ;)
> >

> Is a daily run sufficient? When I modify my programs, I prefer to get 
> feedback from the testsuite in the next minutes, not the next day.

sure, I was just thinking that a daily run is better than nothing ;)

> Isn't it better to have a local copy of mauve ready for running after 
> each local change to Kaffe?

Yes.

> How easy is it to run Mauve of kaffe? Is this documented somewhere?

Running kaffe with mauve is very easy:

1. get kaffe from CVS, build and install it somewhere

2. get mauve from CVS

3. cd mauve

4. autoreconf (you need latest auto* tools for this step)

5. ./configure JAVA=/path/to/kaffe JAVAC=/path/to/jikes-or-kjc

6. make check KEYS="JDK 1.something \!java.lang.Character.unicode" > kaffe.log
(in order to avoid running the long, and probably erranous mauve unicode tests)

7. if you want extra verbose output, use make check KEYS=above
TESTFLAGS="-verbose"

8. you can pick single tests to run, by using make check KEYS="JDK1.something
\!java. \!javax. package.class.test"

9. you can pipe the output of the verbose runs into a perl script that
generates nice web pages: http://www.japhar.org/mauve-results-12.html . The
script is Japhar's CVS.

I guess I should really write a FAQ.mauve ;)

cheers,
dalibor topic

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kaffe-1.0-failures
Type: application/octet-stream
Size: 2839 bytes
Desc: kaffe-1.0-failures
Url : http://kaffe.org/pipermail/kaffe/attachments/20030522/d317f7a1/attachment-0004.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kaffe-1.1-failures
Type: application/octet-stream
Size: 34821 bytes
Desc: kaffe-1.1-failures
Url : http://kaffe.org/pipermail/kaffe/attachments/20030522/d317f7a1/attachment-0005.obj