[kaffe] Bug report (java.io.StreamTokenizer)

[Ito Kazumitsu]

>>>>> ":" == Dalibor Topic <robilad at yahoo.com> writes:

:> --- Ito Kazumitsu <ito.kazumitsu at hitachi-cable.co.jp> wrote:

>> In message "Re: [kaffe] Bug report (java.io.StreamTokenizer)"
>> on 03/07/01, Ito Kazumitsu <ito.kazumitsu at hitachi-cable.co.jp> writes:

>> > (And I guess) BUT ONLY THE ATTRIBUTE MOST RECENTLY ASSIGNED IS EFFECTIVE.
>> 
>> And additionally,  THE NUMERIC ATTRIBUTE IS KEPT EFFECTIVE EVEN IF
>> OTHER ATTIBUTES ARE ASSIGNED.
>> 
>> Applying the attached patch,  I am getting the same results as Sun's.

:> great work, Ito! please check it in.

I thought I was getting the same results as Sun's, but finally,
I could not get exactly the same result.

Attached is my new test program that generates various patterns
of test cases and prints the results of them.

And I put the results at:

   Sun's
     http://www.jsdi.or.jp/kaffe/StreamTokenizerTest2.out.Sun

   Kaffe's original java.io.StreamTokenizer
     http://www.jsdi.or.jp/kaffe/StreamTokenizerTest2.out.Kaffe

   Kaffe's patched java.io.StreamTokenizer
     http://www.jsdi.or.jp/kaffe/StreamTokenizerTest2.out.Kaffe2

Further guessing on how Sun's StreamTokenizer is working is needed.

bash-2.05b$ cat StreamTokenizerTest2.java
import java.io.*;
public class StreamTokenizerTest2 {

  public static void main(String[] args) throws Exception {
      String[] a = new String[] {"S", "C", "Q", "W", "N"};
      for (int i=1; i<=5; i++) {
          Permutation.generate(a, i, new MainHandler());
      }
  }

  private static class MainHandler extends Permutation.Handler {
    public void doit(Object[] array) {
        String[] args = new String[array.length + 1];
        System.arraycopy(array, 0, args, 1, array.length);
        args[0] = "1";
        try {
            for (int i=0; i<args.length; i++) {
                System.out.print(args[i] + " ");
            }
            System.out.println();
            test(args);
        }
        catch (Exception e) {}
    }
  }

  private static void test(String[] args) throws Exception {
    StreamTokenizer tok = new StreamTokenizer(new FileInputStream("StreamTokenizerTest2.txt"));
    tok.resetSyntax();
    int c = args[0].charAt(0);
    for (int i=1; i<args.length; i++) {
       if (args[i].equals("S")) tok.whitespaceChars(c, c);
       else if (args[i].equals("C")) tok.commentChar(c);
       else if (args[i].equals("Q")) tok.quoteChar(c);
       else if (args[i].equals("N")) tok.parseNumbers();
       else if (args[i].equals("W")) tok.wordChars(c, c);
    }
    while (true) {
      int t = tok.nextToken();
      if (t == StreamTokenizer.TT_NUMBER) {
	  System.out.println(tok.nval + ": " + t);
      }
      else {
	  System.out.println(tok.sval + ": " + t);
      }
      if (t == StreamTokenizer.TT_EOF) break;
    }
  }
}
bash-2.05b$ cat Permutation.java
public class Permutation {

    public static void generate(Object[] array, int n, Handler h) {
        int l = array.length;
        if (n == 1) {
            for (int i = 0; i < l; i++) {
                h.doit(new Object[] {array[i]});
            }
            return;
        }
        final int N = n;
        final Handler H = h;
        for (int i = 0; i < l; i++) {
            final Object OBJ = array[i];
            Object[] a1 = new Object[l - 1];
            System.arraycopy(array, 0, a1, 0, i);
            System.arraycopy(array, i+1, a1, i, l-i-1);
            class Handler1 extends Handler {
                public void doit(Object[] a2) {
                    Object[] a3 = new Object[N];
                    System.arraycopy(a2, 0, a3, 1, N-1);
                    a3[0] = OBJ;
                    H.doit(a3);
                }
            };
            generate(a1, n-1, new Handler1());
        }
        return;
    }

    public static class Handler {
        public void doit(Object[] array) {}
    }

}
bash-2.05b$ cat StreamTokenizerTest2.txt 
121
bash-2.05b$