State machines and stateful parsing

Reach for split(',') and CSV looks solved in one line. Then a field contains a comma: the row "a,b",c is meant to hold two fields, a,b and c. Split on every comma and you get three pieces ("a, b", c), the quotes still attached and the first field torn in half.

The fix is to stop treating every comma as a separator. Walk the input one character at a time and track a single piece of state: am I currently inside a quoted field? A comma inside quotes is data; a comma outside quotes is a separator.

This "for each character, update some state, occasionally emit a result" pattern is called a state machine. It comes up in any non-trivial parsing task: JSON, command-line arguments, terminal escape sequences, markup languages.

A skeleton

fn parse(line: &str) -> Vec<String> {
    let mut fields = Vec::new();
    let mut current = String::new();
    let mut in_quotes = false;
    let mut chars = line.chars().peekable();

    while let Some(c) = chars.next() {
        match (c, in_quotes) {
            ('"', false) => in_quotes = true,
            ('"', true) if chars.peek() == Some(&'"') => {
                // Escaped quote inside a quoted field.
                current.push('"');
                chars.next();
            }
            ('"', true) => in_quotes = false,
            (',', false) => {
                fields.push(std::mem::take(&mut current));
            }
            (c, _) => current.push(c),
        }
    }
    fields.push(current);
    fields
}

Two things worth pointing out:

• peekable() lets you look at the next character without consuming it. Essential when one character's meaning depends on the one after it (the "" -> " rule here).
• match on a tuple (c, in_quotes) lets you express each transition as one arm. Easier to read than nested if/else.
• std::mem::take gives you the current string and replaces it with an empty one in a single move. No clone, no temporary.

A note on while let

while let Some(c) = chars.next() { ... } is the loop counterpart of if let from the Option chapter. It keeps running as long as the pattern matches, and stops as soon as it doesn't. Iterators return None at the end, so while let Some(...) is a natural fit when you need more control than a for loop gives you (here we want to call chars.next() again inside the loop body to consume the second ").

Tuple matching like match (c, in_quotes) { ... } is the same idea as the tuples chapter's let (a, b) = pair, just used as a match scrutinee. The arms then pattern-match both elements at once, and the guards (the Result chapter's if chars.peek() == Some(&'"')) do the rest.

The tests in this chapter also lean heavily on raw strings (r#"..."#, introduced in the env-file parser chapter) so the CSV examples can contain literal commas and quotes without an escape forest.

A useful tactic

When stateful parsing gets hairy, write the simple version first (split_once, split(',')) and let the easy tests pass. Then upgrade to the state-machine version for the harder cases. Failing tests give you concrete examples to think against, instead of trying to imagine every edge case up front.

For real CSV in production code, reach for the csv crate; it handles all the corners that this exercise glosses over. But knowing how to write a state machine yourself is a transferable skill.

A first pass: comma-splitting

Before tackling the messy realities of CSV (quotes, escapes, embedded commas), let's handle the trivial case: a line that's nothing but plain values separated by commas, possibly with surrounding whitespace. This is what most "I'll just split on commas" CSV parsers do, and it's also why so many of them break.

Use str::split and str::trim. Collect into a Vec<String>.

Useful from the standard library

• str::split with a ',' argument yields each comma-separated piece as a &str.
• str::trim drops leading/trailing whitespace from each piece.
• str::to_string in a map step turns the borrowed pieces into the owned Strings the return type wants.
• Iterator::collect finishes the chain. The body fits on one line: line.split(',').map(|s| s.trim().to_string()).collect().

Quotes, embedded commas, and escapes

Real CSV is a state machine in disguise. A field can be wrapped in double quotes, in which case any commas inside the quotes are part of the field, not separators. And a literal " inside a quoted field is encoded as "" (two quotes).

Suggested order of attack:

1. Plain a,b,c and simply quoted "a","b","c" (the basic test).
2. Commas inside quoted fields: "a,b",c.
3. Escaped quotes: "a""b",c -> [a"b, c].

Walk the string character by character with a peekable iterator and keep a small in_quotes: bool flag. When you see " while already inside quotes, peek the next char: if it's another ", push a literal " and consume both; otherwise close the field.

Useful from the standard library

• str::chars is the entry point for character-level iteration.
• Iterator::peekable wraps the iterator so you can look ahead one character. Essential for the "" -> " rule.
• Peekable::peek returns Option<&Item> without advancing.
• std::mem::take swaps the current String with a fresh empty one in a single move. Cleaner than current.clone() followed by current.clear().
• A match (c, in_quotes) on the tuple lets you express each state transition as a single arm. Add a guard (if chars.peek() == Some(&'"')) for the escape rule.

Parsing a whole file

With a working line parser, the file-level parser is mostly plumbing: split on newlines, treat the first line as headers, and parse the rest as data rows.

Use str::lines to split: it handles trailing newlines gracefully, so "a,b\n" gives one line, not two.

This step composes on top of parse_csv_line from the previous step. To keep each step independently runnable, the signature is re-declared here as a stub with todo!(). Replace it with your solution from step 4 (or just call into it).

Useful from the standard library

• str::lines yields each line as a &str, stripping \n and \r\n. A trailing newline does not create an empty trailing line.
• Iterator::next on the iterator pulls off the header line; an empty file should return empty headers and rows.
• Iterator::map + parse_csv_line over the remaining lines builds the rows.
• Iterator::collect to materialize both the headers and the rows into Vecs.

Wrapping up the CSV parser

You wrote the easy version of CSV with split and trim, then upgraded to a real state machine that handles quoted fields and escaped quotes, and glued the parsed lines into headers + rows.

What we learned

• Stateful parsing comes up everywhere CSV doesn't (JSON, command lines, terminal escape sequences). The routine is always: walk the input character by character, keep a small flag (or enum) of current state, occasionally emit a result.
• A peekable iterator is the standard tool for "what comes next?" decisions like the "" -> " escape rule. Iterator::peekable costs nothing in practice.
• match (token, state) { ... } over a tuple expresses each state transition in one line. Match guards (if cond) handle the cases where the transition depends on the lookahead.
• std::mem::take(&mut s) gives you the current value and replaces it with Default in one move. Cleaner than clone-then-clear when you're harvesting an accumulator.
• The simple split/trim version is worth writing first. It passes the easy tests and gives you a baseline; the state-machine upgrade then has concrete failing cases to react to.
• For real CSV in production, reach for the csv crate: it handles all the corner cases (BOMs, custom delimiters, escaped newlines inside fields) this exercise glosses over. Writing the parser by hand once is still worth doing for the transferable state-machine technique.