Lexer

Lexical analysis components.

Classes

Name	Description
`Lexer`	Stateful lexer cursor over an input source buffer.
`Token`	Represents a single token in the source code.

Enumerations

Name	Description
`TokenKind`

TokenKind

enum TokenKind

Value	Description
`TOKEN_INT`
`TOKEN_FLOAT`
`TOKEN_STRING`
`TOKEN_IDENTIFIER`
`TOKEN_ELSE`
`TOKEN_IF`
`TOKEN_PRINT`
`TOKEN_RETURN`
`TOKEN_WHILE`
`TOKEN_FALSE`
`TOKEN_TRUE`
`TOKEN_FOR`
`TOKEN_BREAK`
`TOKEN_CONTINUE`
`TOKEN_FUNCTION`
`TOKEN_PLUS`
`TOKEN_MINUS`
`TOKEN_STAR`
`TOKEN_SLASH`
`TOKEN_PERCENT`
`TOKEN_EQUAL`
`TOKEN_CARET`
`TOKEN_PIPE`
`TOKEN_AMP`
`TOKEN_BANG`
`TOKEN_EQEQ`
`TOKEN_BANGEQ`
`TOKEN_LT`
`TOKEN_GT`
`TOKEN_LTEQ`
`TOKEN_GTEQ`
`TOKEN_AMPAMP`
`TOKEN_PIPEPIPE`
`TOKEN_LSHIFT`
`TOKEN_RSHIFT`
`TOKEN_TILDE`
`TOKEN_PLUS_PLUS`
`TOKEN_MINUS_MINUS`
`TOKEN_PLUS_EQUAL`
`TOKEN_MINUS_EQUAL`
`TOKEN_COMMA`
`TOKEN_LPAREN`
`TOKEN_RPAREN`
`TOKEN_LBRACE`
`TOKEN_RBRACE`
`TOKEN_SEMICOLON`
`TOKEN_EOF`
`TOKEN_INVALID`
`TOKEN_ERROR`

Functions

Return	Name	Description
`void`	`lexer_init`	Initializes a lexer from null-terminated source text.
`Token`	`lexer_next_token`	Scans and returns the next token from source.
`void`	`lexer_init`	Initializes a lexer from null-terminated source text.
`bool`	`isAtEnd` `static`	Checks if lexer has reached end of input.
`char`	`lexer_advance` `static`
`char`	`peek` `static`
`char`	`peekNext` `static`
`void`	`skip_whitespace` `static`
`Token`	`make_token` `static`
`Token`	`errorToken` `static`
`bool`	`match` `static`
`Token`	`string` `static`
`Token`	`number` `static`
`TokenKind`	`checkKeyword` `static`
`TokenKind`	`identifierType` `static`
`Token`	`identifier` `static`
`Token`	`lexer_next_token`	Scans and returns the next token from source.

lexer_init

void lexer_init(Lexer * lexer, const char * source)

Initializes a lexer from null-terminated source text.

Parameters

lexer Lexer state output.
source Source input buffer.

Initializes a lexer from null-terminated source text.

Parameters

lexer Lexer state output.
source Source input buffer.

Initializes a lexer from null-terminated source text.

lexer_next_token

Token lexer_next_token(Lexer * lexer)

Scans and returns the next token from source.

Parameters

lexer Lexer to advance.

Returns

Next token with lexeme slice into source.

Scans and returns the next token from source.

Parameters

lexer Lexer to advance.

Returns

Next token with lexeme slice into source.

Scans and returns the next token from source.

lexer_init

void lexer_init(Lexer * lexer, const char * source)

Initializes a lexer from null-terminated source text.

Parameters

lexer Lexer state output.
source Source input buffer.

Initializes a lexer from null-terminated source text.

Parameters

lexer Lexer state output.
source Source input buffer.

isAtEnd

static

static bool isAtEnd(Lexer * lexer)

Checks if lexer has reached end of input.

Parameters

lexer Lexer state.

lexer_advance

static

static char lexer_advance(Lexer * lexer)

peek

static

static char peek(Lexer * lexer)

peekNext

static

static char peekNext(Lexer * lexer)

skip_whitespace

static

static void skip_whitespace(Lexer * lexer)

make_token

static

static Token make_token(Lexer * lexer, TokenKind type)

errorToken

static

static Token errorToken(Lexer * lexer, const char * message)

match

static

static bool match(Lexer * lexer, char expected)

string

static

static Token string(Lexer * lexer)

number

static

static Token number(Lexer * lexer)

checkKeyword

static

static TokenKind checkKeyword(Lexer * lexer, int start, int length, const char * rest, TokenKind type)

identifierType

static

static TokenKind identifierType(Lexer * lexer)

identifier

static

static Token identifier(Lexer * lexer)

lexer_next_token

Token lexer_next_token(Lexer * lexer)

Scans and returns the next token from source.

Parameters

lexer Lexer to advance.

Returns

Next token with lexeme slice into source.

Scans and returns the next token from source.

Parameters

lexer Lexer to advance.

Returns

Next token with lexeme slice into source.

Lexer

#include <lexer.h>

Stateful lexer cursor over an input source buffer.

Public Attributes

Return	Name	Description
`const char *`	`start`	Start pointer of current lexeme.
`const char *`	`current`	Current scanning cursor.
`int`	`line`	Current 1-based line in source.
`int`	`column`	Current 1-based column in source.

start

const char * start

Start pointer of current lexeme.

current

const char * current

Current scanning cursor.

line

int line

Current 1-based line in source.

column

int column

Current 1-based column in source.

Token

#include <token.h>

Represents a single token in the source code.

Lexeme is represented as a slice (start pointer + length) into the original source buffer.

Public Attributes

Return	Name	Description
`TokenKind`	`type`	Token category.
`const char *`	`start`	Pointer to first character of lexeme
`size_t`	`length`	Length of lexeme
`int`	`line`	Line number where the token appears
`int`	`column`	Column number where the token starts

type

TokenKind type

Token category.

start

const char * start

Pointer to first character of lexeme

length

size_t length

Length of lexeme

line

int line

Line number where the token appears

column

int column

Column number where the token starts