Batch issues yet again. about scramjet HOT 7 CLOSED

Ok, good, I'll check the differences over the weekend and I'll try to figure out what went down.

4.23 may rely on a bit older papaparse, but that should not affect the parsing. There may be some things missing around rate and exec, but I don't see you using this.

from scramjet.

@KwangGan if you could post a sample CSV link here so we could try to reproduce the problem.

Two things may be important:

1. The csv should include similar data - like if there's some Thai chars it would be good to provide similar example
2. The length of the csv - if that's smaller than the batch size we should keep it that way.

Here's the overall bugfixing procedure:

1. Isolate the test case reproducing the issue
2. Add the test case to scramjet/test/methods (or if that's multi-method we'll find a better place)
3. Fix the issue.
4. Confirm issue resolution on your end.
5. Bugfix release.

from scramjet.

Thank you for reopen issue

1. I tried decrease batch size to 1 or increase to 500 but no luck.

2. This is my CSV file that contain only EN alphabet.
   https://ybin.me/p/b2a673d1480d1c94#WGKBUy47K9F/ZaRyg5/xEd581xCW9hy+E6VkQOfmCKE=

3. this is my code using multer_s3 and express js for handle request and upload.
   with this code I managed to upload and parse file URL to axios and pipe stream data to Stringstream convert everything to the new Json object and then process stalled at batch function

https://ybin.me/p/4f62a44fa86cb9a7#W2Asc7kv9vX2JQw2D8TqUikJPgwMRiB4Sjyhz73fUxs=

3. here my package.json
   https://ybin.me/p/c96db83f4852f288#LDqRVel2AMDFcBhcbxZG0oqHQv5pc820fNhP1XYF6Dk=

node 10.15

Best Regards

from scramjet.

My solution for now is Downgrade to v4.23 and everything works as expect :D

from scramjet.

@KwangGan I have tested batch in your scenario and I don't think it's our case (I've added some special tests that should cover the case).

I believe your code may be affected by a change in pull which in turn handles the function* slightly different. Can you try a slightly different way of getting the stream?

StringStream
              .from(async function () {
                const response = await axios.get(csv_url, {
                  responseType: 'stream'
                })
                return response.data.pipe(stripBomStream())
              }, { maxParallel: 4 })

I'm not sure if this helps but the answer will lead me somewhere...

from scramjet.

@KwangGan could you create some kind of a repo (or fork of scramjet) that would have a test for you scenario?

from scramjet.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from scramjet.