# Lexer

Lexical analysis components.

### Classes

| Name | Description |
|------|-------------|
| [`Lexer`](#lexer-1) | Stateful lexer cursor over an input source buffer. |
| [`Token`](#token) | Represents a single token in the source code. |

### Enumerations

| Name | Description |
|------|-------------|
| [`TokenKind`](#tokenkind)  |  |

---

#### TokenKind

```cpp
enum TokenKind
```

| Value | Description |
|-------|-------------|
| `TOKEN_INT` |  |
| `TOKEN_FLOAT` |  |
| `TOKEN_STRING` |  |
| `TOKEN_IDENTIFIER` |  |
| `TOKEN_ELSE` |  |
| `TOKEN_IF` |  |
| `TOKEN_PRINT` |  |
| `TOKEN_RETURN` |  |
| `TOKEN_WHILE` |  |
| `TOKEN_FALSE` |  |
| `TOKEN_TRUE` |  |
| `TOKEN_FOR` |  |
| `TOKEN_BREAK` |  |
| `TOKEN_CONTINUE` |  |
| `TOKEN_FUNCTION` |  |
| `TOKEN_PLUS` |  |
| `TOKEN_MINUS` |  |
| `TOKEN_STAR` |  |
| `TOKEN_SLASH` |  |
| `TOKEN_PERCENT` |  |
| `TOKEN_EQUAL` |  |
| `TOKEN_CARET` |  |
| `TOKEN_PIPE` |  |
| `TOKEN_AMP` |  |
| `TOKEN_BANG` |  |
| `TOKEN_EQEQ` |  |
| `TOKEN_BANGEQ` |  |
| `TOKEN_LT` |  |
| `TOKEN_GT` |  |
| `TOKEN_LTEQ` |  |
| `TOKEN_GTEQ` |  |
| `TOKEN_AMPAMP` |  |
| `TOKEN_PIPEPIPE` |  |
| `TOKEN_LSHIFT` |  |
| `TOKEN_RSHIFT` |  |
| `TOKEN_TILDE` |  |
| `TOKEN_PLUS_PLUS` |  |
| `TOKEN_MINUS_MINUS` |  |
| `TOKEN_PLUS_EQUAL` |  |
| `TOKEN_MINUS_EQUAL` |  |
| `TOKEN_COMMA` |  |
| `TOKEN_LPAREN` |  |
| `TOKEN_RPAREN` |  |
| `TOKEN_LBRACE` |  |
| `TOKEN_RBRACE` |  |
| `TOKEN_SEMICOLON` |  |
| `TOKEN_EOF` |  |
| `TOKEN_INVALID` |  |
| `TOKEN_ERROR` |  |

### Functions

| Return | Name | Description |
|--------|------|-------------|
| `void` | [`lexer_init`](#lexer_init)  | Initializes a lexer from null-terminated source text. |
| `Token` | [`lexer_next_token`](#lexer_next_token)  | Scans and returns the next token from source. |
| `void` | [`lexer_init`](#lexer_init-1)  | Initializes a lexer from null-terminated source text. |
| `bool` | [`isAtEnd`](#isatend) `static` | Checks if lexer has reached end of input. |
| `char` | [`lexer_advance`](#lexer_advance) `static` |  |
| `char` | [`peek`](#peek) `static` |  |
| `char` | [`peekNext`](#peeknext) `static` |  |
| `void` | [`skip_whitespace`](#skip_whitespace) `static` |  |
| `Token` | [`make_token`](#make_token) `static` |  |
| `Token` | [`errorToken`](#errortoken) `static` |  |
| `bool` | [`match`](#match) `static` |  |
| `Token` | [`string`](#string) `static` |  |
| `Token` | [`number`](#number) `static` |  |
| `TokenKind` | [`checkKeyword`](#checkkeyword) `static` |  |
| `TokenKind` | [`identifierType`](#identifiertype) `static` |  |
| `Token` | [`identifier`](#identifier) `static` |  |
| `Token` | [`lexer_next_token`](#lexer_next_token-1)  | Scans and returns the next token from source. |

---

#### lexer_init

```cpp
void lexer_init(Lexer * lexer, const char * source)
```

Initializes a lexer from null-terminated source text.

Initializes a lexer from null-terminated source text.   

#### Parameters
* `lexer` [Lexer](#lexer-1) state output. 

* `source` Source input buffer.

Initializes a lexer from null-terminated source text.

#### Parameters
* `lexer` [Lexer](#lexer-1) state output. 

* `source` Source input buffer.

Initializes a lexer from null-terminated source text.

---

#### lexer_next_token

```cpp
Token lexer_next_token(Lexer * lexer)
```

Scans and returns the next token from source.

Scans and returns the next token from source.   

#### Parameters
* `lexer` [Lexer](#lexer-1) to advance. 

#### Returns
Next token with lexeme slice into source.

Scans and returns the next token from source.

#### Parameters
* `lexer` [Lexer](#lexer-1) to advance. 

#### Returns
Next token with lexeme slice into source.

Scans and returns the next token from source.

---

#### lexer_init

```cpp
void lexer_init(Lexer * lexer, const char * source)
```

Initializes a lexer from null-terminated source text.

Initializes a lexer from null-terminated source text.

#### Parameters
* `lexer` [Lexer](#lexer-1) state output. 

* `source` Source input buffer.

Initializes a lexer from null-terminated source text.

Initializes a lexer from null-terminated source text.   

#### Parameters
* `lexer` [Lexer](#lexer-1) state output. 

* `source` Source input buffer.

---

#### isAtEnd

`static`

```cpp
static bool isAtEnd(Lexer * lexer)
```

Checks if lexer has reached end of input.

#### Parameters
* `lexer` [Lexer](#lexer-1) state.

---

#### lexer_advance

`static`

```cpp
static char lexer_advance(Lexer * lexer)
```

---

#### peek

`static`

```cpp
static char peek(Lexer * lexer)
```

---

#### peekNext

`static`

```cpp
static char peekNext(Lexer * lexer)
```

---

#### skip_whitespace

`static`

```cpp
static void skip_whitespace(Lexer * lexer)
```

---

#### make_token

`static`

```cpp
static Token make_token(Lexer * lexer, TokenKind type)
```

---

#### errorToken

`static`

```cpp
static Token errorToken(Lexer * lexer, const char * message)
```

---

#### match

`static`

```cpp
static bool match(Lexer * lexer, char expected)
```

---

#### string

`static`

```cpp
static Token string(Lexer * lexer)
```

---

#### number

`static`

```cpp
static Token number(Lexer * lexer)
```

---

#### checkKeyword

`static`

```cpp
static TokenKind checkKeyword(Lexer * lexer, int start, int length, const char * rest, TokenKind type)
```

---

#### identifierType

`static`

```cpp
static TokenKind identifierType(Lexer * lexer)
```

---

#### identifier

`static`

```cpp
static Token identifier(Lexer * lexer)
```

---

#### lexer_next_token

```cpp
Token lexer_next_token(Lexer * lexer)
```

Scans and returns the next token from source.

Scans and returns the next token from source.

#### Parameters
* `lexer` [Lexer](#lexer-1) to advance. 

#### Returns
Next token with lexeme slice into source.

Scans and returns the next token from source.

Scans and returns the next token from source.   

#### Parameters
* `lexer` [Lexer](#lexer-1) to advance. 

#### Returns
Next token with lexeme slice into source.



## Lexer

```cpp
#include <lexer.h>
```

Stateful lexer cursor over an input source buffer.

### Public Attributes

| Return | Name | Description |
|--------|------|-------------|
| `const char *` | [`start`](#start)  | Start pointer of current lexeme. |
| `const char *` | [`current`](#current)  | Current scanning cursor. |
| `int` | [`line`](#line)  | Current 1-based line in source. |
| `int` | [`column`](#column)  | Current 1-based column in source. |

---

#### start

```cpp
const char * start
```

Start pointer of current lexeme.

---

#### current

```cpp
const char * current
```

Current scanning cursor.

---

#### line

```cpp
int line
```

Current 1-based line in source.

---

#### column

```cpp
int column
```

Current 1-based column in source.



## Token

```cpp
#include <token.h>
```

Represents a single token in the source code.

Lexeme is represented as a slice (start pointer + length) into the original source buffer.

### Public Attributes

| Return | Name | Description |
|--------|------|-------------|
| `TokenKind` | [`type`](#type)  | [Token](#token) category. |
| `const char *` | [`start`](#start-1)  | Pointer to first character of lexeme |
| `size_t` | [`length`](#length)  | Length of lexeme |
| `int` | [`line`](#line-1)  | Line number where the token appears |
| `int` | [`column`](#column-1)  | Column number where the token starts |

---

#### type

```cpp
TokenKind type
```

[Token](#token) category.

---

#### start

```cpp
const char * start
```

Pointer to first character of lexeme

---

#### length

```cpp
size_t length
```

Length of lexeme

---

#### line

```cpp
int line
```

Line number where the token appears

---

#### column

```cpp
int column
```

Column number where the token starts