Struct Lexer 

pub struct Lexer {
    input: Vec<char>,
    position: usize,
}

A lexical analyzer that converts Markdown text into a sequence of tokens. Handles nested structures and special Markdown syntax elements while maintaining proper context and state during parsing.

Fields

input: Vec<char>

Input text as character vector for efficient parsing

position: usize

Current position in the input stream

Implementations

impl Lexer

pub fn new(input: String) -> Self

Creates a new lexer instance from input string

pub fn parse(&mut self) -> Result<Vec<Token>, LexerError>

Parses the entire input string into a sequence of tokens. Returns a Result containing either a Vec of parsed tokens or a LexerError.

pub fn parse_with_context( &mut self, ctx: ParseContext, ) -> Result<Vec<Token>, LexerError>

Parses the entire input string into a sequence of tokens for a given context. Returns a Result containing either a Vec of parsed tokens or a LexerError. Takes in a ParseContext that determines which tokens are valid in the current location.

fn parse_nested_content<F>( &mut self, is_delimiter: F, ctx: ParseContext, ) -> Result<Vec<Token>, LexerError>

where F: Fn(char) -> bool,

Helper function to parse nested content until a delimiter is encountered. Used for parsing content within emphasis, headings, and list items.

fn next_token(&mut self, ctx: ParseContext) -> Result<Option<Token>, LexerError>

Determines the next token in the input stream based on the current character and context. Handles special cases like line starts differently.

fn parse_heading(&mut self) -> Result<Token, LexerError>

Parses a heading token, counting ‘#’ characters for level and collecting nested content

fn parse_emphasis(&mut self) -> Result<Token, LexerError>

Parses emphasis tokens (* or _) with support for multiple levels (1-3). Ensures proper matching of opening and closing delimiters. Follows CommonMark rules: underscore emphasis requires flanking whitespace/punctuation.

fn parse_code(&mut self) -> Result<Token, LexerError>

Parses code blocks, handling both inline code and fenced code blocks

fn count_backticks(&mut self) -> usize

Helper method to count consecutive backticks

fn parse_math(&mut self) -> Result<Token, LexerError>

Parses LaTeX math blocks ($…$ for inline, $$…$$ for display)

fn parse_blockquote(&mut self) -> Result<Token, LexerError>

Parses a blockquote, collecting text until newline

fn parse_link(&mut self) -> Result<Token, LexerError>

Parses a link token, extracting display text and URL

fn parse_image(&mut self) -> Result<Token, LexerError>

Parses an image token, extracting alt text and URL

fn parse_newline(&mut self) -> Result<Token, LexerError>

Parses a newline token

fn parse_text(&mut self, ctx: ParseContext) -> Result<Token, LexerError>

Parses regular text until a special token start or newline is encountered. Returns an error if no text could be parsed.

fn parse_html_comment(&mut self) -> Result<Token, LexerError>

Parses an HTML comment, extracting the comment content

fn is_at_line_start(&self) -> bool

Checks if current position is at the start of a line

fn skip_whitespace(&mut self)

Skips whitespace characters except newlines

fn advance(&mut self)

Advances the position counter by one

fn current_char(&self) -> char

Returns the current character or ‘\0’ if at end of input

fn read_until_newline(&mut self) -> String

Reads characters until a newline is encountered

fn read_until_char(&mut self, delimiter: char) -> String

Reads characters until a specific delimiter is encountered

fn is_html_comment_start(&self) -> bool

Checks if current position starts an HTML comment

fn is_start_of_special_token(&self, ctx: ParseContext) -> bool

Checks if current position could start a special token given a context

fn is_after_special_token(&self) -> bool

Checks if we’re immediately after a special token that should preserve following spaces

fn check_horizontal_rule(&mut self) -> Result<bool, LexerError>

Checks if the current position contains a horizontal rule (—)

fn check_ordered_list_marker(&mut self) -> Option<usize>

Checks if current position starts an ordered list marker (e.g., “1.”)

fn parse_list_item( &mut self, ordered: bool, indent_level: usize, parent_ctx: ParseContext, ) -> Result<Token, LexerError>

Parses a list item, handling both ordered and unordered types

fn is_table_start(&self) -> bool

Checks if the current posisiton is the start of a table

fn parse_table(&mut self) -> Result<Token, LexerError>

Parses a table, handling column alignment

fn get_current_indent(&self) -> usize

Gets the current line’s indentation level

fn is_list_marker(&self, marker: char) -> bool

Checks if the given character at the current position is a list marker A list marker is followed by whitespace (space or tab)