Package org.apache.flink.cdc.common.text

Class TokenStream

java.lang.Object

org.apache.flink.cdc.common.text.TokenStream

@NotThreadSafe
public class TokenStream
extends Object

A stream of tokens that can be consumed by a parser. This class is not thread-safe.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected class	TokenStream.CaseInsensitiveToken	An immutable TokenStream.Token that implements matching using case-insensitive logic.
class	TokenStream.CaseInsensitiveTokenFactory	An implementation of TokenStream.Tokens that creates TokenStream.CaseInsensitiveToken objects.
protected class	TokenStream.CaseSensitiveToken	An immutable TokenStream.Token that implements matching using case-sensitive logic.
class	TokenStream.CaseSensitiveTokenFactory	An implementation of TokenStream.Tokens that creates TokenStream.CaseSensitiveToken objects.
static class	TokenStream.CharacterArrayStream	An implementation of TokenStream.CharacterStream that works with a single character array.
static interface	TokenStream.CharacterStream	Interface used by a TokenStream.Tokenizer to iterate through the characters in the content input to the TokenStream.
static class	TokenStream.Marker	An opaque marker for a position within the token stream.
static interface	TokenStream.Token	The interface defining a token, which references the characters in the actual input character stream.
protected class	TokenStream.TokenFactory	An implementation of TokenStream.Tokens that creates TokenStream.CaseSensitiveToken objects.
static interface	TokenStream.Tokenizer	Interface for a Tokenizer component responsible for processing the characters in a TokenStream.CharacterStream and constructing the appropriate TokenStream.Token objects.
static interface	TokenStream.Tokens	A factory for Token objects, used by a TokenStream.Tokenizer to create tokens in the correct order.

Field Summary

Fields

Modifier and Type	Field	Description
static int	ANY_TYPE
static String	ANY_VALUE
protected String	inputString

Constructor Summary

Constructors

Constructor	Description
TokenStream(String content, TokenStream.Tokenizer tokenizer, boolean caseSensitive)

Method Summary

Modifier and Type	Method	Description
String	consume()	Return the value of this token and move to the next token.
boolean	hasNext()	Determine if this stream has another token to be consumed.
protected List<TokenStream.Token>	initializeTokens(List<TokenStream.Token> tokens)	Method to allow subclasses to pre-process the set of tokens and return the correct tokens to use.
TokenStream.Marker	mark()	Obtain a marker that records the current position so that the stream can be rewind(Marker) back to the mark even after having been advanced beyond the mark.
boolean	matches(int type, String expected)	Determine if the current token matches the expected type and a value.
boolean	matches(String expected)	Determine if the current token matches the expected value.
boolean	matchesAnyOf(String[] options)	Determine if the next token matches one of the supplied values.
String	peek()
boolean	rewind(TokenStream.Marker marker)	Reset the stream back to the position described by the supplied marker.
TokenStream	start()	Begin the token stream, including (if required) the tokenization of the input content.
protected void	throwNoMoreContent()
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

ANY_VALUE

public static final String ANY_VALUE

See Also:

Constant Field Values

ANY_TYPE

public static final int ANY_TYPE

See Also:

Constant Field Values

inputString

protected final String inputString

Constructor Detail

TokenStream

public TokenStream(String content,
                   TokenStream.Tokenizer tokenizer,
                   boolean caseSensitive)

Method Detail

start

public TokenStream start()
                  throws ParsingException

Begin the token stream, including (if required) the tokenization of the input content.

Returns:

this object for easy method chaining; never null

Throws:

ParsingException - if an error occurs during tokenization of the content

initializeTokens

protected List<TokenStream.Token> initializeTokens(List<TokenStream.Token> tokens)

Method to allow subclasses to pre-process the set of tokens and return the correct tokens to use. The default behavior is to simply return the supplied tokens.

Parameters:

tokens - the tokens

Returns:

list of tokens.

mark

public TokenStream.Marker mark()

Obtain a marker that records the current position so that the stream can be rewind(Marker) back to the mark even after having been advanced beyond the mark.

Returns:

the marker; never null

Throws:

IllegalStateException - if this method was called before the stream was started

NoSuchElementException - if there are no more tokens

rewind

public boolean rewind(TokenStream.Marker marker)

Reset the stream back to the position described by the supplied marker. This method does nothing if the mark is invalid. For example, it is not possible to advance the token stream beyond the current position.

Parameters:

marker - the marker

Returns:

true if the token stream was reset, or false if the marker was invalid

consume

public String consume()
               throws ParsingException,
                      IllegalStateException

Return the value of this token and move to the next token.

Returns:

the value of the current token

Throws:

ParsingException - if there is no such token to consume

IllegalStateException - if this method was called before the stream was started

throwNoMoreContent

protected void throwNoMoreContent()
                           throws ParsingException

Throws:

ParsingException

peek

public String peek()
            throws IllegalStateException

Throws:

IllegalStateException

matches

public boolean matches(String expected)
                throws IllegalStateException

Determine if the current token matches the expected value.

The ANY_VALUE constant can be used as a wildcard.

Parameters:

expected - the expected value of the current token

Returns:

true if the current token did match, or false if the current token did not match

Throws:

IllegalStateException - if this method was called before the stream was started

matches

public boolean matches(int type,
                       String expected)
                throws IllegalStateException

Determine if the current token matches the expected type and a value.

The ANY_VALUE constant can be used as a wildcard.

Parameters:

type - the expected type of the curent token

expected - the expected value of the current token

Returns:

true if the current token did match, or false if the current token did not match

Throws:

IllegalStateException - if this method was called before the stream was started

matchesAnyOf

public boolean matchesAnyOf(String[] options)
                     throws IllegalStateException

Determine if the next token matches one of the supplied values.

Parameters:

options - the options for the value of the current token

Returns:

true if the current token's value did match one of the supplied options, or false otherwise

Throws:

IllegalStateException - if this method was called before the stream was started

hasNext

public boolean hasNext()

Determine if this stream has another token to be consumed.

Returns:

true if there is another token ready for consumption, or false otherwise

Throws:

IllegalStateException - if this method was called before the stream was started

toString

public String toString()

Overrides:

toString in class Object