We've learned about the horror that is CSV parsing in the State Machines and Stateful Parsing chapter. And since we can't get enough of it, let's do some more parsing!
This time we try to change the separator without breaking quoted fields and then reuse our parser through a small public API. Along the way, we learn about Rust's module system.
- Each exercise is independent and works in the browser (or as a local Rust test file).
- The module task includes a working line parser; it doesn't depend on finishing the delimiter task.
- This bonus chapter does not count toward course completion.
Our little CSV parser is doing well, but one day a spreadsheet export arrives with semicolons instead of commas.
We get a little nervous as we think about how to change the parser without breaking the quoting rules.
It takes a little thought, but we can implement a function,
parse_delimited_line(line, delimiter), that supports any delimiter without
changing the quoting rules. A delimiter inside quotes is still data, and doubled
quotes still mean one literal quote. Commas become ordinary data when the
delimiter is ;.
If you'd like to adapt your earlier quote parser, copy its body into this editor. But you can also work directly from the starter below.
I suggest keeping parse_csv_line as a thin wrapper that chooses a comma as the
separator to preserve our public API. (Imagine that our users rely on this function not to break.)
For this exercise, callers supply a delimiter other than ", \r, or \n.
You can assume that all inputs have balanced quotes, quotes wrap whole fields,
and records occupy one line.
Please preserve whitespace, empty fields, and trailing empty fields. An empty line is one empty field. Rejecting malformed CSV is outside this small challenge.
Before running the tests, predict the fields in "a;""b";c with ; as the
delimiter. Then try a tab or a Unicode character as the delimiter instead.
Does your loop operate on characters or bytes?
fn parse_delimited_line(line: &str, delimiter: char) -> Vec<String> {
let mut fields = Vec::new();
let mut field = String::new();
let mut in_quotes = false;
let mut chars = line.chars().peekable();
while let Some(c) = chars.next() {
match c {
'"' if in_quotes => {
// A doubled quote inside a quoted field is a literal quote; a
// lone quote ends the quoted section.
if chars.peek() == Some(&'"') {
field.push('"');
chars.next();
} else {
in_quotes = false;
}
}
'"' => in_quotes = true,
c if c == delimiter && !in_quotes => {
fields.push(std::mem::take(&mut field));
}
_ => field.push(c),
}
}
fields.push(field);
fields
}
fn parse_csv_line(line: &str) -> Vec<String> {
parse_delimited_line(line, ',')
}
#[test]
fn semicolon_fields_and_literal_commas() {
assert_eq!(parse_delimited_line("a,b;c;", ';'), vec!["a,b", "c", ""]);
}
#[test]
fn delimiter_inside_quotes_is_data() {
assert_eq!(parse_delimited_line(r#""a;b";c"#, ';'), vec!["a;b", "c"]);
}
#[test]
fn escaped_quote_next_to_delimiter() {
assert_eq!(
parse_delimited_line(r#""a;""b";c"#, ';'),
vec!["a;\"b", "c"]
);
assert_eq!(parse_delimited_line("\"\"\"\";\"\"", ';'), vec!["\"", ""]);
}
#[test]
fn tabs_unicode_and_whitespace() {
assert_eq!(
parse_delimited_line(" left \tright ", '\t'),
vec![" left ", "right "]
);
assert_eq!(
parse_delimited_line("é🦀\"a🦀b\"🦀", '🦀'),
vec!["é", "a🦀b", ""]
);
}
#[test]
fn empty_fields_are_preserved() {
assert_eq!(parse_delimited_line("", ';'), vec![""]);
assert_eq!(parse_delimited_line(";;", ';'), vec!["", "", ""]);
}
#[test]
fn comma_wrapper_keeps_quote_and_escape_behavior() {
assert_eq!(parse_csv_line(r#""a,b","c""d","#), vec!["a,b", "c\"d", ""]);
}
This step deliberately reuses the whole-file parser from the state-machine chapter. The new work is experimenting with its module boundary, not inventing another parsing loop.
I've put a working parse_line inside mod csv { ... } below. You can use it
as is, even if you skipped the delimiter exercise. Fill in parse_file by
reusing your earlier file parser and adjusting the helper name to parse_line.
If you haven't written that step, parse the first line as headers and the rest
as rows, then return both. Use the supplied line parser for headers and rows.
Keep the same quoting rules as before. We still assume valid input and don't support newlines inside quoted fields. A few details to watch out for:
I suggest using str::lines() to walk through the input. It handles both Unix
and Windows line endings.
The tests live outside mod csv. Once they pass, try these changes one at a
time. Before each run, predict which call the compiler will accept or reject.
pub from parse_file. Run the tests, inspect the error, then put
pub back.csv::parse_line("a,b") from a test. Compare that call with the one
inside parse_file, then remove the added call.Which parts of the parser do callers need to see? Keeping the line helper
private lets you change it without changing the public API. Being in the same
file doesn't give the outer tests access to private items inside csv.
If you're working locally, try moving the code inside mod csv { ... } into
csv.rs, next to 3_parser_module.rs. Leave the tests where they are, and
replace the module block with:
#[path = "csv.rs"]
mod csv;
The path tells Rust where to find our file, including when you run the exercise
through the chapter's generated main.rs. You don't need to edit main.rs.
Callers still use csv::parse_file, just as before.
If you're using the course's browser editor, keep everything in one file: it only sends that file to the playground.
mod csv {
fn parse_line(line: &str) -> Vec<String> {
let mut fields = Vec::new();
let mut field = String::new();
let mut in_quotes = false;
let mut chars = line.chars().peekable();
while let Some(c) = chars.next() {
match c {
'"' if in_quotes => {
// A doubled quote inside a quoted field is a literal quote;
// a lone quote ends the quoted section.
if chars.peek() == Some(&'"') {
field.push('"');
chars.next();
} else {
in_quotes = false;
}
}
'"' => in_quotes = true,
',' if !in_quotes => {
fields.push(std::mem::take(&mut field));
}
_ => field.push(c),
}
}
fields.push(field);
fields
}
pub fn parse_file(content: &str) -> (Vec<String>, Vec<Vec<String>>) {
let mut lines = content.lines();
let headers = lines.next().map(parse_line).unwrap_or_default();
let rows = lines.map(parse_line).collect();
(headers, rows)
}
}
#[test]
fn public_api_parses_headers_and_rows_with_the_same_rules() {
let (headers, rows) =
csv::parse_file("\"last, first\",note\n\"Doe, Jane\",\"said \"\"hi\"\"\"\n");
assert_eq!(headers, vec!["last, first", "note"]);
assert_eq!(rows, vec![vec!["Doe, Jane", "said \"hi\""]]);
}
#[test]
fn empty_file_and_headers_only() {
assert_eq!(csv::parse_file(""), (vec![], vec![]));
assert_eq!(
csv::parse_file("a,b\n"),
(vec!["a".into(), "b".into()], vec![])
);
}
#[test]
fn blank_interior_line_is_one_empty_field() {
let (headers, rows) = csv::parse_file("a,b\n\n1,2\n");
assert_eq!(headers, vec!["a", "b"]);
assert_eq!(rows, vec![vec![""], vec!["1", "2"]]);
}
#[test]
fn trailing_empty_field_is_preserved_without_an_extra_record() {
let (headers, rows) = csv::parse_file("a,b\n1,\n");
assert_eq!(headers, vec!["a", "b"]);
assert_eq!(rows, vec![vec!["1", ""]]);
}
#[test]
fn multiple_rows_and_crlf() {
let (headers, rows) = csv::parse_file("a,b\r\n1,2\r\n3,4\r\n");
assert_eq!(headers, vec!["a", "b"]);
assert_eq!(rows, vec![vec!["1", "2"], vec!["3", "4"]]);
}
Extra practice to explore at your own pace. These chapters do not count toward course progress.