Package org.apache.commons.csv
Class Lexer
java.lang.Object
org.apache.commons.csv.Lexer
- All Implemented Interfaces:
Closeable
,AutoCloseable
Lexical analyzer.
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final int
private static final String
private final char[]
private final char[]
private final int
private final char[]
private String
private final boolean
private final boolean
private boolean
private final boolean
private static final String
private final int
private final ExtendedBufferedReader
The buffered reader.private final boolean
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprivate void
Appends the next escaped character to the token's content.void
close()
Closes resources.(package private) long
Returns the current character position(package private) long
Returns the current line number(package private) String
(package private) boolean
isClosed()
(package private) boolean
isCommentStart
(int ch) (package private) boolean
isDelimiter
(int ch) Determine whether the next characters constitute a delimiter throughUnsynchronizedBufferedReader.peek(char[])
.(package private) boolean
isEndOfFile
(int ch) Tests if the given character indicates the end of the file.(package private) boolean
isEscape
(int ch) Tests if the given character is the escape character.(package private) boolean
Tests if the next characters constitute a escape delimiter throughUnsynchronizedBufferedReader.peek(char[])
.private boolean
isMetaChar
(int ch) (package private) boolean
isQuoteChar
(int ch) (package private) boolean
isStartOfLine
(int ch) Tests if the current character represents the start of a line: a CR, LF, or is at the start of the file.(package private) Token
Returns the next token.private int
private Token
parseEncapsulatedToken
(Token token) Parses an encapsulated token.private Token
parseSimpleToken
(Token token, int ch) Parses a simple token.(package private) boolean
readEndOfLine
(int ch) Greedily accepts \n, \r and \r\n This checker consumes silently the second control-character...(package private) int
Handle an escape sequence.(package private) void
trimTrailingSpaces
(StringBuilder buffer)
-
Field Details
-
CR_STRING
-
LF_STRING
-
delimiter
private final char[] delimiter -
delimiterBuf
private final char[] delimiterBuf -
escapeDelimiterBuf
private final char[] escapeDelimiterBuf -
escape
private final int escape -
quoteChar
private final int quoteChar -
commentStart
private final int commentStart -
ignoreSurroundingSpaces
private final boolean ignoreSurroundingSpaces -
ignoreEmptyLines
private final boolean ignoreEmptyLines -
lenientEof
private final boolean lenientEof -
trailingData
private final boolean trailingData -
reader
The buffered reader. -
firstEol
-
isLastTokenDelimiter
private boolean isLastTokenDelimiter
-
-
Constructor Details
-
Lexer
Lexer(CSVFormat format, ExtendedBufferedReader reader)
-
-
Method Details
-
appendNextEscapedCharacterToToken
Appends the next escaped character to the token's content.- Parameters:
token
- the current token- Throws:
IOException
- on stream access errorCSVException
- Thrown on invalid input.
-
close
Closes resources.- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Throws:
IOException
- If an I/O error occurs
-
getCharacterPosition
long getCharacterPosition()Returns the current character position- Returns:
- the current character position
-
getCurrentLineNumber
long getCurrentLineNumber()Returns the current line number- Returns:
- the current line number
-
getFirstEol
String getFirstEol() -
isClosed
boolean isClosed() -
isCommentStart
boolean isCommentStart(int ch) -
isDelimiter
Determine whether the next characters constitute a delimiter throughUnsynchronizedBufferedReader.peek(char[])
.- Parameters:
ch
- the current character.- Returns:
- true if the next characters constitute a delimiter.
- Throws:
IOException
- If an I/O error occurs.
-
isEndOfFile
boolean isEndOfFile(int ch) Tests if the given character indicates the end of the file.- Returns:
- true if the given character indicates the end of the file.
-
isEscape
boolean isEscape(int ch) Tests if the given character is the escape character.- Returns:
- true if the given character is the escape character.
-
isEscapeDelimiter
Tests if the next characters constitute a escape delimiter throughUnsynchronizedBufferedReader.peek(char[])
. For example, for delimiter "[|]" and escape '!', return true if the next characters constitute "![!|!]".- Returns:
- true if the next characters constitute an escape delimiter.
- Throws:
IOException
- If an I/O error occurs.
-
isMetaChar
private boolean isMetaChar(int ch) -
isQuoteChar
boolean isQuoteChar(int ch) -
isStartOfLine
boolean isStartOfLine(int ch) Tests if the current character represents the start of a line: a CR, LF, or is at the start of the file.- Parameters:
ch
- the character to check- Returns:
- true if the character is at the start of a line.
-
nextToken
Returns the next token.A token corresponds to a term, a record change or an end-of-file indicator.
- Parameters:
token
- an existing Token object to reuse. The caller is responsible for initializing the Token.- Returns:
- the next token found.
- Throws:
IOException
- on stream access error.CSVException
- Thrown on invalid input.
-
nullToDisabled
-
parseEncapsulatedToken
Parses an encapsulated token.Encapsulated tokens are surrounded by the given encapsulating string. The encapsulator itself might be included in the token using a doubling syntax (as "", '') or using escaping (as in \", \'). Whitespaces before and after an encapsulated token is ignored. The token is finished when one of the following conditions becomes true:
- An unescaped encapsulator has been reached and is followed by optional whitespace then:
- delimiter (TOKEN)
- end of line (EORECORD)
- end of stream has been reached (EOF)
- Parameters:
token
- the current token- Returns:
- a valid token object
- Throws:
IOException
- Thrown when in an invalid state: EOF before closing encapsulator or invalid character before delimiter or EOL.CSVException
- Thrown on invalid input.
-
parseSimpleToken
Parses a simple token.Simple tokens are tokens that are not surrounded by encapsulators. A simple token might contain escaped delimiters (as \, or \;). The token is finished when one of the following conditions becomes true:
- The end of line has been reached (EORECORD)
- The end of stream has been reached (EOF)
- An unescaped delimiter has been reached (TOKEN)
- Parameters:
token
- the current tokench
- the current character- Returns:
- the filled token
- Throws:
IOException
- on stream access errorCSVException
- Thrown on invalid input.
-
readEndOfLine
Greedily accepts \n, \r and \r\n This checker consumes silently the second control-character...- Returns:
- true if the given or next character is a line-terminator
- Throws:
IOException
-
readEscape
Handle an escape sequence. The current character must be the escape character. On return, the next character is available by callingExtendedBufferedReader.getLastChar()
on the input stream.- Returns:
- the unescaped character (as an int) or
IOUtils.EOF
if char following the escape is invalid. - Throws:
IOException
- if there is a problem reading the stream or the end of stream is detected: the escape character is not allowed at end of streamCSVException
- Thrown on invalid input.
-
trimTrailingSpaces
-