paigen11 / file-read-challenge Goto Github PK

Hey @paigen11

Thank you for sharing your code from your itnext post

I just tried it on my pc (window 10, node v8.12.0) but neither readFileStream or readFileEventStream were able to process the whole itcont.txt (3,49Go)

How did you execute your test? Have you any clues about why event-stream did worked for me?

First of all, thank you for your post, it was quite good.

For streaming, in every rl.on('line', call you already have the current line, and you are extracting firstNames and formattedTimestamp already.

But rl.on('close' call you are again running loop over firstNames dateDonationCount arrays which is unnecessary.

On your rl.on('line' event you can calculate those values and then you can remove the unnecessary loops in the close event. That way your measurement will be way lower than the previous one.

Just check this out.

readFileStream.js

if (firstHalfOfName.includes(' ') && firstHalfOfName !== ' ') {
	firstName = firstHalfOfName.split(' ')[0];
	firstName.trim();
	firstNames.push(firstName);      
        // make your calculation here, no need to loop over firstNames again
	dupeNames[firstName] = (dupeNames[firstName] || 0) + 1;
} else {
	firstNames.push(firstHalfOfName);
	dupeNames[firstHalfOfName] = (dupeNames[firstHalfOfName] || 0) + 1;
}

[....]

dateDonationCount.push(formattedTimestamp);
// here, eliminate the dateDonationCount.forEach loop in the close event
dateDonations[formattedTimestamp] = (dateDonations[formattedTimestamp] || 0) + 1;

plse give me example upload files big ?