Large CSV files

Files bigger than memory, processed a row at a time.

Install#

dart pub add csv_plus

import 'package:csv_plus/csv_plus.dart';

Why loading the whole file fails#

Decoding a file into a list holds every row at once, and the in-memory representation is several times larger than the bytes on disk. A CSV of a few hundred megabytes can exhaust the heap long before it finishes. Streaming keeps one row in flight instead.

Stream a file#

import 'package:csv_plus/io.dart';

await for (final row in CsvFile.stream('huge.csv')) {
  process(row);
}

Memory stays flat whatever the file size, because rows are handed to you as they are parsed and released once you are done with them.

Stream anything, not just a file#

A network response or any other byte source works the same way, with backpressure handled for you so a fast producer cannot outrun your processing:

final rows = codec.decoder.bindBytes(byteStream); // Stream<List<int>>

Reading only part of a file#

When you want a sample rather than the whole thing, bound it at decode time instead of reading everything and throwing most of it away:

final codec = CsvCodec(CsvConfig(
  skipRows: 1,
  hasHeader: true,
  maxRows: 1000,
));

maxRows lets the batch decoders stop early, so the rest of the file is never parsed.

Writing large output#

Append as you go rather than building one enormous string:

import 'package:csv_plus/io.dart';

for (final batch in batches) {
  await CsvFile.append('out.csv', batch);
}