[kaffe] Re: Regression/build testing using Xenofarm

[Dalibor Topic]

Hi Jim,

Jim Pick wrote:

> Ultimately, I think we'll end up with a variety of regression testing
> systems.  I'd like to get results for Gump, Mauve, SpecJVM, and others
> up on the website.
> 
> Here's what I've been thinking:
> 
> 1) Let's not have one "official" regression testing framework.  Let's
>    encourage people to develop their own regression testing systems in
>    a distributed manner.
 >
> 2) Let's define a standard reporting format, as an XML format.  People
>    format their results in this format, and upload them to the kaffe.org
>    server.
> 
> 3) I'll write some scripts to suck in the various submitted XML files,
>    and produce useful reports.  We could even use a database (for
>    example, to track benchmark performance over time).
> 
> 4) Install something like CVSps so we can more easily cross reference
>    individual checkins.

That sounds like the better plan in the long run. Though it also sounds 
like a nice project in itself, "designing and implementing a 
regression-test meta-framework". ;)

I'm not quite sure about 2) though. What should go into the reporting 
format? A lot of build tools just output (not very formatted) text to 
stdout and stderr. Or do you have filters in mind, that XMLize the 
output of 'make', for example?

cheers,
dalibor topic