[kaffe] Re: Regression/build testing using Xenofarm

[Jim Pick]

On Fri, 2003-07-18 at 06:43, Dalibor Topic wrote:
> Hi Jim, hi all,
> 
> When I went to LinuxTag, I came across a very interesting project called 
> Xenofarm http://www.lysator.liu.se/xenofarm/ , which is " a distributed 
> system for cross platform compilation and verification of software 
> components. It is primarily used to help achieve and maintain portable 
> source code across hardware architechtures, operating system and 
> libraries and to quickly detect and fix regression bugs in source code."
> 
> It looks like something we could use to improve status reports on kaffe. 
> See for example the Pikefarm, used by the Pike project : 
> http://pike.ida.liu.se/development/pikefarm/ , results at : 
> http://pike.ida.liu.se/development/pikefarm/7.5.xml . It is also being 
> used by the python project here 
> http://www.lysator.liu.se/xenofarm/python/latest.html . All in all, it 
> looks more appealing than mozillas tinderbox to me ;)
> 
> Would someone be interested in setting it up on kaffe.org, and would 
> people be interested in running clients on their platforms under 
> whatever configurations you use?

We really need something like this.

When I worked at Transvirtual, we cooked up a homebrew system that did
something similar.

Ultimately, I think we'll end up with a variety of regression testing
systems.  I'd like to get results for Gump, Mauve, SpecJVM, and others
up on the website.

Here's what I've been thinking:

1) Let's not have one "official" regression testing framework.  Let's
   encourage people to develop their own regression testing systems in
   a distributed manner.

2) Let's define a standard reporting format, as an XML format.  People
   format their results in this format, and upload them to the kaffe.org
   server.

3) I'll write some scripts to suck in the various submitted XML files,
   and produce useful reports.  We could even use a database (for
   example, to track benchmark performance over time).

4) Install something like CVSps so we can more easily cross reference
   individual checkins.

One of the things holding me up on all this was that I wanted to migrate
the website to Jetty or Tomcat running on Kaffe.  Maybe I'll get it
working this weekend.  :-)

Cheers,

 - jim