Question: Skip first CSV line about scramjet HOT 8 CLOSED

Have you taken a look at the shift method? If you put it after lines it should do what you want, like this:

  StringStream.from(
    fs.createReadStream('contries.csv')
  )
.lines()
.shift(x => console.log("first line:", x)) // here.
.parse()
.setOptions()
.each()
.run()
.then()

There's also a csv parsing method on StringStream called CSVParse - that may be useful.

As to the docs, we're working on a new website and, subsequently, on better documentation. Would you be so kind to review and give us some tips where we should start improvements? Ideally to: [email protected]

from scramjet.

No problem - yes, in the shift operation you need to pass the number of chunks you want to shift - so the call should be:

   .shift(1, (x) => console.log("Shifted", x)) // an array should appear.

And move it after parse - this will make sure we work on objects, not lines so the bytes won't kick in.

from scramjet.

I will! I'm glad you are working on a new documentation website! 😄
I'll try this right away, sorry I missed it (I saw it but interpreted it for something else)

from scramjet.

Interesting... and what did you think it did? Maybe I'd reword the description somehow?

from scramjet.

I read it "wrong". I read the shift function from the StringStream object.

stringStream.shift(bytes, func)

I was unsure on how to calculate the number of bytes I wanted remove.

Since I started with StringStream as you can see on my other issue, I thought it would override the shift function from DataStream and I wouldn't be able to remove the entire line. I got confused.

from scramjet.

You're right, that doesn't make much sense if you look from this perspective. DataStream..shift operates on whole chunks (objects) - so that's how the line is removed when used after parse.

Ok. I see the point - this needs a major rethink - perhaps shiftBytes should be other than shift for compatibility with DataStream.

We need to consider changing that in v5 - of course a backwards compatible version will be provided for the current v4 line.

from scramjet.

Oh shiftBytes would be a great idea!

Hmm may I ask, in my logic if I do the following, no lines are added to the database, in fact the console.log doesn't even trigger. But if I remove the .shift function it works and all lines are added including the header and...an invisible last line that doesn't appear in the file (may be something I'll look later)

Same code:

async function insert_countries(that) {
  const { StringStream } = require('scramjet')
  const fs = require('fs')
  const path = require('path')

  const client = await that.pg.connect()

  const INSERT_ENTRY =
    'INSERT INTO countries (iso3, iso2, name, dialing_code, dialing_exit, population, area_km2) VALUES ($1, $2, $3, $4, $5, $6, $7);'

  StringStream.from(
    fs.createReadStream(
      '/media/DarkHawk/srv/NodesProjects/___Ackanet/database/table-countries.csv'
    )
  )
    .lines()
    .shift(x => console.log('Skipped', x))
    .parse(line => {
      // Each line need to be properly formated
      const entry = line.split(';') //comma split

      let reconstruction = []

      // The following fields need formating
      reconstruction.push(entry[4])
      reconstruction.push(entry[3])
      reconstruction.push(entry[0])
      reconstruction.push(entry[1])
      reconstruction.push(entry[2])
      reconstruction.push(entry[5] != '' ? parseInt(entry[5]) : null)
      reconstruction.push(entry[6] != '' ? parseInt(entry[6]) : null)
      return reconstruction
    })
    .setOptions({ maxParallel: 1 })
    .each(async reconstruction => {
      console.log('arg:', reconstruction)

      try {
        await client.query(INSERT_ENTRY, reconstruction)
      } catch (err) {
        console.log('Error while adding line...', err)
        // some more logic could be here?
      } finally {
        // client.release()
      }
    })
    .run()
    .then(
      () => {
        console.log('Read entire file.')
        // return resolve(true)
      },
      e => console.log('Error while handling file.', err)
    )
}

Total output: Read entire file.

I'll let this issue closed for now as it may be my code and not an actuel bug since I'm not very familiar yet with the package.

And I wanted to thank you for the time you take answering my questions. Most of the time we get better help in the repo than on Q&A websites.

from scramjet.

Duh, thanks

from scramjet.