“Knock, knock.”
“Who’s there?”
“Broken state machine.”
“Broken state machine who?”
“Knock, knock.”
CSV, or Comma-Separated Values, is a wonderfully simple file format:
Frank,42,Engineer
Hannah,24,Teacher
Jake,35,Doctor
Each line is a record and each comma separates fields: name, age, and occupation. Just split by commas and you are d...
Joe,30,Artist,Painter,Writer
Oh no...
Contrary to popular belief, CSV is a mischievous pain in the neck. It lures you
into thinking it's just a list of comma-separated fields, but the moment you
reach for split(','), it will rear its ugly head. For example, a row "a,b",c
is meant to hold two fields, a,b and c. Split on every comma and you get
three pieces ("a, b", c), the quotes still attached and the first field
torn in half.
From csv-spec.org:
CSV is not a file format, it is a loose set of guidelines of how to structure tabular data into a plain text string. As such there’s an endless amount of *.csv files floating around which are highly incompatible with each other. The closest thing there is to a specification is RFC 4180.
This should tell you everything you need to know about CSV. And yet! We will attempt to tame this beast and write a parser for a subset of CSV, including quoted fields and escaped quotes.
The trick is to stop treating every comma as a separator. Walk the input one character at a time and track a single piece of state: "Am I currently inside a quoted field?" A comma inside quotes is data; a comma outside quotes is a separator.
This "for each character, update some state, occasionally emit a result" pattern is called a state machine. It comes up in various parsing tasks: JSON, command-line arguments, terminal escape sequences, markup languages, the legacy data format used by your favorite spreadsheet program.
Before writing CSV, consider the following tiny state machine, where we are counting characters outside square brackets.
The input is something like "ab[secret]c" and the output is 3, because there are three characters outside the brackets: a, b, and c. The brackets themselves are not counted, and neither are the characters inside them. (We assume brackets are balanced and never nested.)
fn count_visible(text: &str) -> usize {
let mut hidden = false;
let mut count = 0;
for c in text.chars() {
match (c, hidden) {
('[', false) => hidden = true,
(']', true) => hidden = false,
(_, false) => count += 1,
(_, true) => {}
}
}
count
}
assert_eq!(count_visible("ab[secret]c"), 3);
Here, we track the state before and after each bracket.
Our CSV parser needs a similar distinction between "data" and "syntax", but on top of that needs to collect fields and recognize escaped quotes.
peekable() lets you inspect the next character without consuming it. This
is a convenient way to distinguish an escaped quote from a closing quote.while let Some(c) = chars.next() repeats until the iterator is exhausted.
Unlike a for loop, it lets you call chars.next() inside the body to
consume a second character when needed.(c, in_quotes) lets an arm consider both character and
state. A match guard can add a lookahead condition.std::mem::take moves out a
completed String and leaves an empty one behind, without cloning its
contents.The tests use raw strings so quotes don't need backslash escapes in the Rust source. CSV itself escapes quotes by doubling them.
When stateful parsing gets hairy, write the simple version first (split_once,
split(',')) and let the easy tests pass. Then upgrade to the state-machine
version for the harder cases. Failing tests give you concrete examples to think
against, instead of trying to imagine every edge case up front.
This parser is practice for the state-machine loop, not a complete CSV implementation. For production code, use the
csvcrate, which handles cases we leave out, such as quoted fields containing newlines.
Start with plain values separated by commas. You'll handle quotes, escapes, and embedded commas in the next exercise.
Return the fields as a Vec<String>.
Useful from the Standard Library
str::splitwith a','argument yields each comma-separated piece as a&str.ToString::to_stringin amapstep turns the borrowed pieces into the ownedStrings the return type wants.Iterator::collectfinishes the chain.
&str that splits on a delimiter and gives you an
iterator. Convert each piece to an owned String, then collect the results.
/// Parses a simple CSV line without quotes by splitting on commas.
fn parse_simple_csv_line(line: &str) -> Vec<String> {
line.split(',').map(str::to_string).collect()
}
#[test]
fn test_parse_simple_csv_line() {
let line = "name,age,city";
let fields = parse_simple_csv_line(line);
assert_eq!(fields, vec!["name", "age", "city"]);
}
A field can be wrapped in double quotes, in which case any commas inside the
quotes are part of the field, not separators. And a literal " inside a quoted
field is encoded as "" (two quotes).
Preserve whitespace rather than trimming fields. RFC 4180, section 2 says:
Spaces are considered part of a field and should not be ignored.
Spaces are part of the data, whether the field is quoted or not: a, b becomes
["a", " b"], and " a ",b becomes [" a ", "b"]. Remove the surrounding
quotes, but keep the spaces inside them.
Implement the cases in the order shown by the tests:
a,b,c and simply quoted "a","b","c" (the basic test)."a,b",c."a""b",c -> [a"b, c].Escaped quotes are the tricky part: the same character can be syntax or data. It's worth getting one case working at a time. If a test fails, trace its input on paper and mark which characters belong in the output.
Useful from the Standard Library
str::charsis the entry point for character-level iteration.Iterator::peekablewraps an iterator so you can look ahead one character.Peekable::peekreturnsOption<&Item>without advancing.std::mem::takeswaps the currentStringwith a fresh empty one in a single move. That saves you a copy compared withcurrent.clone()followed bycurrent.clear().
Start with the distinction from the bracket example: does this character act as syntax in the current state? Keep the completed fields separate from the field you're still reading.
For escapes, compare "a",c with "a""b",c. At the quote after a, what
information tells you whether the field is ending? peek borrows the next item
without consuming it. Check which characters your loop has consumed before its
next iteration.
If the last field is missing, trace an input without a trailing comma. What happens to the unfinished field when the iterator runs out?
/// Parses a CSV line with proper quote handling. Handles embedded commas and
/// doubled quotes, as in `"a""b",c`. Preserves whitespace in all fields.
fn parse_csv_line(line: &str) -> Vec<String> {
let mut fields = Vec::new();
let mut field = String::new();
let mut in_quotes = false;
let mut chars = line.chars().peekable();
while let Some(c) = chars.next() {
match c {
'"' if in_quotes => {
// A doubled quote inside a quoted field is a literal quote; a
// lone quote ends the quoted section.
if chars.peek() == Some(&'"') {
field.push('"');
chars.next();
} else {
in_quotes = false;
}
}
'"' => in_quotes = true,
',' if !in_quotes => {
fields.push(std::mem::take(&mut field));
}
_ => field.push(c),
}
}
fields.push(field);
fields
}
#[test]
fn test_parse_csv_line_plain_numbers() {
let line = r#"1,2,3"#;
let fields = parse_csv_line(line);
assert_eq!(fields, vec!["1", "2", "3"]);
}
#[test]
fn test_parse_csv_line_plain_strings() {
// Warm-up: every field is quoted, no commas inside, no escapes. Get this
// passing first; it forces you to enter and exit a quoted field, but
// nothing trickier.
let line = r#""a","b","c""#;
let fields = parse_csv_line(line);
assert_eq!(fields, vec!["a", "b", "c"]);
}
#[test]
fn test_parse_csv_line_quoted() {
let line = r#"name,"age, years","city""#;
let fields = parse_csv_line(line);
assert_eq!(fields, vec!["name", "age, years", "city"]);
}
#[test]
fn test_parse_csv_line_quoted_escaped() {
let line = r#""John ""Johnny"" Doe","25","New York""#;
let fields = parse_csv_line(line);
assert_eq!(fields, vec![r#"John "Johnny" Doe"#, "25", "New York"]);
}
#[test]
fn test_parse_csv_line_preserves_unquoted_whitespace() {
assert_eq!(parse_csv_line("a, b"), vec!["a", " b"]);
assert_eq!(parse_csv_line(" a ,b "), vec![" a ", "b "]);
}
#[test]
fn test_parse_csv_line_preserves_quoted_whitespace() {
assert_eq!(parse_csv_line(r#"" a ",b"#), vec![" a ", "b"]);
assert_eq!(parse_csv_line(r#"a," ""b"" ""#), vec!["a", " \"b\" "]);
}
Your line parser has done the harder work. For the whole file, split on newlines, treat the first line as headers, and parse the rest as data rows.
Assume each record occupies one line; this parser won't handle newlines inside quoted fields.
You'll reuse parse_csv_line from the previous page. To keep this page
independently runnable, its signature is re-declared here as a stub with
todo!(). Paste your earlier solution into the stub.
Useful from the Standard Library
str::linesyields each line as a&str, stripping\nand\r\n. A trailing newline does not create an empty trailing line.Iterator::nexton the iterator pulls off the header line; an empty file should return empty headers and rows.Iterator::map+parse_csv_lineover the remaining lines builds the rows.Iterator::collectgathers the parsed rows into aVec<Vec<String>>.parse_csv_linealready returns the headers as aVec<String>.
content.lines() gives you an iterator over &str lines.next() on the iterator pulls
the first one off; the rest you can map(parse_csv_line).collect().
/// Parses a CSV line with proper quote handling. (Re-stubbed from step 4 so
/// this file compiles on its own.)
fn parse_csv_line(line: &str) -> Vec<String> {
let mut fields = Vec::new();
let mut field = String::new();
let mut in_quotes = false;
let mut chars = line.chars().peekable();
while let Some(c) = chars.next() {
match c {
'"' if in_quotes => {
if chars.peek() == Some(&'"') {
field.push('"');
chars.next();
} else {
in_quotes = false;
}
}
'"' => in_quotes = true,
',' if !in_quotes => {
fields.push(std::mem::take(&mut field));
}
_ => field.push(c),
}
}
fields.push(field);
fields
}
/// Parses a complete CSV file. First line is headers, remaining lines are data.
///
/// Use
/// [`str::lines`](https://doc.rust-lang.org/std/primitive.str.html#method.lines)
/// to split on newlines. `lines()` already handles a trailing `\n` gracefully;
/// it won't yield an empty last line for `"a,b\n"`. Real CSVs often end with a
/// newline, so this is the right tool. Returns (headers, rows).
fn parse_csv_file(content: &str) -> (Vec<String>, Vec<Vec<String>>) {
let mut lines = content.lines();
let headers = lines.next().map(parse_csv_line).unwrap_or_default();
let rows = lines.map(parse_csv_line).collect();
(headers, rows)
}
#[test]
fn test_parse_csv_file() {
let content = "name,age,city\nAlice,30,Boston\nBob,25,Seattle";
let (headers, rows) = parse_csv_file(content);
assert_eq!(headers, vec!["name", "age", "city"]);
assert_eq!(rows.len(), 2);
assert_eq!(rows[0], vec!["Alice", "30", "Boston"]);
}
You started with split, then kept track of quotes so commas inside
a field stayed where they belonged. Once that worked, you reused the line parser
to collect headers and rows.
What We Learned
- The same stateful parsing pattern appears in JSON, command lines, and terminal escape sequences. In each case, read one item, consult the current state, then update the state or emit a result.
- A peekable iterator lets you inspect what comes next without consuming it. We use it to recognize doubled quotes. Another approach is to track an additional "after quote" state.
match (token, state) { ... }over a tuple expresses each state transition in one line. Match guards (if cond) handle the cases where the transition depends on the lookahead.std::mem::take(&mut s)gives you the current value and replaces it withDefaultin one move. You can move the accumulated string into your results without cloning it and then clearing the original.- The simple
splitversion is worth writing first. It passes the easy tests and gives you a baseline; the state-machine upgrade then has concrete failing cases to react to.- Hand production CSV files to the
csvcrate, which handles BOMs, custom delimiters, and newlines inside quoted fields. Keep the state-machine loop for parsers you do need to write yourself.
Want a little more practice? Try the optional Even More CSV Parsing where we put the parser behind a module boundary. These don't count toward course completion.
Extra practice to explore at your own pace. These chapters do not count toward course progress.