csv_plus v1.3.0
pub.dev GitHub

Reading a CSV file that is not UTF-8

Excel on Windows does not write UTF-8. Pick the encoding instead of getting replacement characters.

Install#

dart pub add csv_plus
import 'package:csv_plus/csv_plus.dart';

The symptom#

A name comes back with a replacement character instead of an accent, or a price shows a stray symbol where the euro sign should be. That file is not UTF-8. The Excel CSV export on Western European Windows writes Windows-1252, a single byte encoding, and reading those bytes as UTF-8 either throws or substitutes.

Pick the encoding#

Pass a CsvCharset to any of the byte decoders:

import 'package:csv_plus/csv_plus.dart';

final rows = const CsvCodec().decodeBytes(
  bytes,
  charset: CsvCharset.windows1252,
);

Three encodings are built in, with no extra dependency:

  • CsvCharset.utf8, the default. A leading byte order mark is stripped, and a malformed byte is replaced rather than thrown so one bad byte does not lose the file.
  • CsvCharset.latin1, also called ISO-8859-1. Every byte maps to the code point of the same value.
  • CsvCharset.windows1252, the usual Excel output on Windows. It matches Latin-1 except in the 32 slots from 0x80 to 0x9F, which hold the euro sign, curly quotes and dashes instead of control codes.

Which one do I need#

If the file came out of Excel on Windows, try windows1252 first. It is a superset of Latin-1 in practice, so it reads correctly in both cases and is the safer default of the two. Reach for latin1 only when you know the source is strictly ISO-8859-1 and you want the control codes preserved.

The byte order mark#

A UTF-8 byte order mark is stripped whichever encoding you choose, so it never gets glued onto your first column name. That is the bug behind a header that will not match: the key is not name but an invisible prefix followed by name.

When writing, set addBom so Excel opens the result as UTF-8 rather than guessing:

const codec = CsvCodec(CsvConfig(addBom: true));
final bytes = codec.encodeToBytes(rows);

Writing a non UTF-8 file#

Output is always UTF-8. If a downstream tool truly requires a single byte encoding, encode to a string first and convert it yourself with the latin1 encoder from dart:convert.