Comments (4)
Hi @tomkeee, we'll be looking into this next week.
from scramjet.
@tomkeee Print operation in every language is slow, if you want to print every line of big file, you have to keep in mind that will drastically slow down your operation. What is a time of execution (processing full file) without printing?
from scramjet.
@Tatarinho
You are right that print operations are quite slow, yet I tried to do a similar operation on my local machine (code below) and the execution time was 55 sec (on Scramjet it was 25min).
import time
lines_number = 0
with open("/home/sirocco/Pulpit/data.csv") as file_in:
start = time.time()
for i in file_in:
if lines_number <1000:
print(f"{lines_number} \n")
elif lines_number > 2000:
print(f"{lines_number} bigger than 2000 \n")
lines_number += 1
print(f"the line_numbers is {lines_number}\n execution time: {time.time()-start}")
from scramjet.
Hmm... so I did some initial digging and was able to run a test with local network and a similar program in node works quite fast, but not as fast as from the disk...
We need to take into account the network connection, but that wouldn't explain 25 minutes.
Could you follow this guide: https://docs.scramjet.org/platform/self-hosted-installation
Then based on this, can you try your program with the data sent to 127.0.0.1
? We'd exclude the network and the platform configuration as the culprit...
from scramjet.
Related Issues (20)
- MultiStream.map(...).mux is not a function HOT 2
- Parsing CSV files of certain arbitrary lengths suddenly stops HOT 7
- StringStream.whenEnd produces UnhandledPromiseRejectionWarning on error HOT 4
- Using scramjet to concate and split data received with socket HOT 10
- DataStream.toJSONArray causes an exception on empty input HOT 5
- Fail when plugin adds a new class twice HOT 1
- I would like to pin this repository to my GH profile HOT 1
- Stateful stream transformation HOT 3
- Class 'PromiseTransform' incorrectly implements class 'Readable' HOT 6
- DataStream.distribute causes ReferenceError when using external function inside HOT 1
- Iterating each chunk with async function with stopping stream HOT 2
- Runner failed error HOT 3
- Cannot stop instance HOT 3
- 'Sequence unpack failed' issue HOT 9
- Instance timeout
- Runner failed error HOT 2
- Drag n drop into 'add sequence' multiple times spams seq list HOT 1
- Add sequence's remove button on the list page
- Getting error Class 'PromiseTransform' incorrectly implements class 'Readable' and 'Writable' HOT 12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scramjet.