Another Bugfix patch for java.io.StreamTokenizer

[Archie Cobbs]

Dalibor-
I tried applying your patch but it didn't apply cleanly.
Could you resubmit it against the current CVS version of
StreamTokenizer.java?

Thanks,
-Archie

Dalibor Topic writes:
> here is a new patch for java.io.StreamTokenizer fixing a bunch of bugs and
> glitches.
> 
> Bugs that are fixed with this patch:
> * when parsing numbers, a '-' alone without following numbers should be an
> ordinary character, not a string. Reference: Java Language Specification.
> 
> * when parsing numbers, a single '.' should evaluate to 0.0
> . Reference: Sun JDK 1.1.x/1.2.x behaviour.
> 
> * when parsing numbers, a single "-." should evaluate to -0.0
> . Reference: Sun JDK 1.1.x/1.2.x behaviour.
> 
> * it was possible to set characters > 255 or characters < 0 to whitespace
> chars, or to word chars, resulting in a corrupted TableEntry ordinary,
> which is used for non ASCII characters. Setting it white space led to the
> character -1 (EOF) being interpreted as white space, thus Kaffe would hang
> on input forever. Reference: The Java Class Libraries Second Edition
> Vol. 1, StreamTokenizer wordChars etc. method descriptions
> 
> * there was a bug when parsing numbers in streams that were something like
> .4.4.4, Kaffe wouldn't stop before the second '.' Reference: Java Language
> Specification
> 
> * when parsing quoted strings, kaffe wasn't able to handle octal escape
> sequances. Reference: Java Language Specification
> 
> * when EOL was pushed back, the line number wasn't decreased so the last
> token before the EOL had the wrong line number (too high).
> 
> * when parsing C or C++ comments and when / is not a comment character,
> kaffe would drop the / in /4
> 
> * line numbers used to be wrong if the parsed quoted string contained
> escaped EOL characters like in "\\\n"
> 
> * skipLine would be stuck in an infinite loop if the line ended
> with an EOF instead of an EOL
> 
> * when parsing quoted strings, Kaffe would drop the newline if the string
> ended prematurely
> 
> * when parsing numbers, Kaffe would try to parse 1.2- as a number instead
> of parsing 1.2 as a number and - as an ordinary character.
> 
> Other things fixed:
> 
> * private class variable pushBack has the value false by default, so I
> modified its declaration accordingly
> 
> * besides the three cases described ("-", "." and "-."), there should be
> no other exceptions in number generation [1], thus I have removed the code
> that would set ttype to TT_WORD and return a string with the number that
> could not be parsed.
> 
> * I removed some code to handle EOL that was redundant in nextTokenType.
> 
> * with all the necessary modifications to make StreamTokenizer more
> standard Java like, nextTokenType grew to big and I've split it into
> several small functions. The semantics of the function should be clearer
> now.
> 
> * I also changed the return type from nextTokenType to void, that allowed
> to simplify nextToken, as well as to eliminate all the labourous token
> type passing between nextToken, nextTokenType and token-type-specific
> parsing functions.
> 
> 
> Things to fix:
> * The LineNumberReader eats \r ... so we don't see "\\\r" in a string as
> "\\\r" but as "\\\n" .
> 
> * The numbers parsed are sometimes just a little different from what they
> should be. For example, .4 comes out as n=0.40000000000000002, instead of
> n=0.4 .

___________________________________________________________________________
Archie Cobbs   *   Whistle Communications, Inc.  *   http://www.whistle.com