Comments (7)
My bad, I forgot that the issue was associated with the bom. Version "5.5.6" of csv-parse fixes the issue.
from node-csv.
Any update on this issue?
from node-csv.
Please look at this commit. I have tried to reproduce your error but it seems to run fine, catch the error passed in the "skip" event.
from node-csv.
There is a code that reproduces the issue on my side.
import path from 'path';
import { pipeline } from 'stream/promises';
import { parse as parseCSV } from 'csv-parse';
import { Writable } from 'stream';
import { createReadStream } from 'fs';
async function testRecordsSkip() {
const errors: any = [];
const records: any = [];
const sink = new Writable({
objectMode: true,
write: (_, __, callback) => {
records.push(_);
callback();
},
});
const csvSource = createReadStream(path.join(__dirname, 'test_with_bom.csv'));
const parser = parseCSV({
skip_records_with_error: true,
bom: true,
});
parser.on('skip', function (err) {
errors.push(err);
});
await pipeline(csvSource, parser, sink);
console.log({
records,
errors,
});
}
testRecordsSkip().catch(console.error);
The output is
{
records: [
[ 'id', 'first_name', 'last_name', 'email', 'modified_at' ],
[ '1', 'Ring', 'Grinyov', '[email protected]', '2022-02-14' ],
[ '3', 'Cammi', 'Bendix', '[email protected]', '2022-02-14' ]
],
errors: []
}
Test file is there
test_with_bom.csv
from node-csv.
I tried again with your code and the output is still what we expect:
{
records: [
[ 'id', 'first_name', 'last_name', 'email', 'modified_at' ],
[ '1', 'Ring', 'Grinyov', '[email protected]', '2022-02-14' ],
[ '3', 'Cammi', 'Bendix', '[email protected]', '2022-02-14' ]
],
errors: [
CsvError: Invalid Record Length: expect 5, got 6 on line 3
at Object.__onRecord (file:///Users/david/projects/open-source/csv/csv/packages/csv-parse/lib/api/index.js:364:11)
at Object.parse (file:///Users/david/projects/open-source/csv/csv/packages/csv-parse/lib/api/index.js:247:40)
at Parser._transform (file:///Users/david/projects/open-source/csv/csv/packages/csv-parse/lib/index.js:31:26)
at Transform._write (node:internal/streams/transform:175:8)
at writeOrBuffer (node:internal/streams/writable:392:12)
at _write (node:internal/streams/writable:333:10)
at Writable.write (node:internal/streams/writable:337:10)
at ReadStream.ondata (node:internal/streams/readable:766:22)
at ReadStream.emit (node:events:514:28)
at addChunk (node:internal/streams/readable:324:12) {
code: 'CSV_RECORD_INCONSISTENT_FIELDS_LENGTH',
bytes: 141,
comment_lines: 0,
empty_lines: 0,
invalid_field_length: 0,
lines: 3,
records: 2,
columns: false,
error: undefined,
header: false,
index: 6,
raw: undefined,
column: 6,
quoting: false,
record: [Array]
}
]
}
Note, tested with Node.js versions "v16.13.0", "v18.17.1", "v20.5.1" and "v21.5.0".
from node-csv.
I tried again with your code and the output is still what we expect:
{ records: [ [ 'id', 'first_name', 'last_name', 'email', 'modified_at' ], [ '1', 'Ring', 'Grinyov', '[email protected]', '2022-02-14' ], [ '3', 'Cammi', 'Bendix', '[email protected]', '2022-02-14' ] ], errors: [ CsvError: Invalid Record Length: expect 5, got 6 on line 3 at Object.__onRecord (file:///Users/david/projects/open-source/csv/csv/packages/csv-parse/lib/api/index.js:364:11) at Object.parse (file:///Users/david/projects/open-source/csv/csv/packages/csv-parse/lib/api/index.js:247:40) at Parser._transform (file:///Users/david/projects/open-source/csv/csv/packages/csv-parse/lib/index.js:31:26) at Transform._write (node:internal/streams/transform:175:8) at writeOrBuffer (node:internal/streams/writable:392:12) at _write (node:internal/streams/writable:333:10) at Writable.write (node:internal/streams/writable:337:10) at ReadStream.ondata (node:internal/streams/readable:766:22) at ReadStream.emit (node:events:514:28) at addChunk (node:internal/streams/readable:324:12) { code: 'CSV_RECORD_INCONSISTENT_FIELDS_LENGTH', bytes: 141, comment_lines: 0, empty_lines: 0, invalid_field_length: 0, lines: 3, records: 2, columns: false, error: undefined, header: false, index: 6, raw: undefined, column: 6, quoting: false, record: [Array] } ] }Note, tested with Node.js versions "v16.13.0", "v18.17.1", "v20.5.1" and "v21.5.0".
I replaced your file with the one I shared and it doesn't work, but when I test it with the one from the link you shared, it works.
When I check the files, I see this, meaning that your file doesn't contain a BOM?
alex@undefined:~/WebstormProjects/test$ cat src/411.csv | file -
/dev/stdin: ASCII text
alex@undefined:~/WebstormProjects/test$ cat src/test_with_bom.csv | file -
/dev/stdin: Unicode text, UTF-8 (with BOM) text
from node-csv.
So there is a script that should reproduce an issue on your side
import assert from 'node:assert';
import { Writable } from 'node:stream';
import { parse } from 'csv-parse';
import { finished } from 'stream/promises';
import { Readable } from 'stream';
async function a() {
const errors: any = [];
const parser = parse({
bom: true,
skipRecordsWithError: true,
});
// Create a stream and consume its source
const sink = new Writable({
objectMode: true,
write: (_, __, callback) => callback(),
});
const csv = `\ufeffkey,value
1,value-1
2,value-2
3,"value-3
one more line
and one more line"`;
const csvSource = Readable.from([csv]);
const outStream = csvSource.pipe(parser).pipe(sink);
// Catch records with errors
parser.on('skip', (e) => {
errors.push(e);
});
// Wait for stream to be consumed
await finished(outStream);
// Catch error from skip event
assert.equal(errors.length, 1);
}
a().catch(console.error);
from node-csv.
Related Issues (20)
- Option `to` and `to_line` results in `ERR_STREAM_PREMATURE_CLOSE` HOT 4
- How can I convert json object with array to csv? HOT 1
- Date parse bug HOT 2
- Parse dot notation columns into nested objects HOT 4
- CSV Parse breaks on comment characters that are also in rows HOT 4
- csv-parse fails to parse very large CSV files
- User-defined value generation HOT 5
- support headers as comment HOT 3
- Support parsing of quote characters within quotes HOT 2
- Ignore comment lines for the from_line value
- bug: Fatal JavaScript invalid size error HOT 1
- Error Lines Property Should Return All Lines of error HOT 1
- Taking too much energy and patience to navigate and understand the docs site HOT 1
- Browser Streaming Support HOT 2
- Problems parsing CSV with nested quotes,commas etc HOT 2
- ReferenceError: Buffer is not defined on import in Nuxt3/Vue3 HOT 2
- Support multiple quote characters when parsing HOT 3
- First value of the first row has a leading space. HOT 1
- cvs-parse fails 'randomly' on multi-character column delimiter and row terminator HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from node-csv.