Giter Club home page Giter Club logo

csv-stream's Introduction

csv-streamify Build Status

NPM

Parses csv files. Accepts options. No coffee script, no weird APIs. Just streams. Tested against csv-spectrum and used in production.

Works in node 0.10, 0.12 and iojs.

Installation

npm install csv-streamify

Usage

This module implements a simple node stream.Transform stream.

var csv = require('csv-streamify'),
    fs = require('fs')

var fstream = fs.createReadStream('/path/to/file'),
    parser = csv(options /* optional */, callback /* optional */)

// emits each line as a buffer or as a string representing an array of fields
parser.on('readable', function () {
  var line = parser.read()
  // do stuff with data as it comes in

  // current line number
  console.log(parser.lineNo)
})

// AND/OR
function callback(err, doc) {
  if (err) return handleErrorGracefully(err)

  // doc is an array of row arrays
  doc.forEach(function (row) { console.log(row) })
}

// now pump some data into it (and pipe it somewhere else)
fstream.pipe(parser).pipe(nirvana)

Note: If you pass a callback to csv-streamify it will buffer the parsed data for you and pass it to the callback when it's done. This behaviour can obviously lead to out of memory errors with very large csv files.

Options

You can pass some options to the parser. All of them are optional.

The options are also passed to the underlying transform stream, so you can pass in any standard node core stream options.

{
  delimiter: ',', // comma, semicolon, whatever
  newline: '\n', // newline character (use \r\n for CRLF files)
  quote: '"', // what's considered a quote
  empty: '', // empty fields are replaced by this,

  // if true, emit arrays instead of stringified arrays or buffers
  objectMode: false,

  // if set to true, uses first row as keys -> [ { column1: value1, column2: value2 }, ...]
  columns: false
}

Also, take a look at iconv-lite (npm install iconv-lite --save), it provides pure javascript streaming character encoding conversion.

CLI

To use on the command line install it globally:

$ npm install csv-streamify -g

This should add the csv-streamify command to your $PATH.

Then, you either pipe data into it or give it a filename:

# pipe data in
$ cat some_data.csv | csv-streamify
# pass a filename
$ csv-streamify some_data.csv > output.json
# tell csv-streamify to read from + wait on stdin
$ csv-streamify -

Wishlist

  • browser support

If you would like to contribute either of those just open an issue so we can discuss it further. :)

Contributors

Nicolas Hery (objectMode)

csv-stream's People

Contributors

derekdreery avatar jmgunn87 avatar klaemo avatar nicolashery avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.