j50n / deno-proc Goto Github PK

I have some ideas for reworking the API so it is a lot more intuitive than the current one - and also covers more use cases. This ticket is for the feature branch, as this is going to break stuff, and I want to make sure it is ready before I push it out.

This is probably just allowing the caller to provide a function that examines the exit code and throws or doesn't.

Simple examples of each input/output type. This needs to be completed.

Run some tests and see what is going on.

I fetch the Node.js nightly binary release, extract node executable and run that extracted node executable without using npm or node_modules whatsoever https://github.com/guest271314/download-node-nightly-executable.

I need to find a good example for the README. I would like it to demonstrate piping bytes and processing string data, but be simple enough to read. It would be good if there were an analogous bash script to compare it to. Might be good to use stdin for input as well.

Chained Errors should be optional, as they are a bit harder to deal with in production code, but can be useful during debugging.

When the outputs of a process have been completely consumed, the process and its resources are closed automatically. At this point, we won't need to close them again, so we can remove their references in the associated ProcGroup.

This covers the case where someone uses a single ProcGroup to run lots of processes without closing.

Deno has a "tee" mechanism. Let's embed that into this and make it simple to use.

Current line separators are hard coded to "\n" in some places. These need to be changed to recognize the OS and use the appropriate line ending.

Explain bash --logon -c. Find a good example.

We have support for arrays and AsyncIterables, but not for Iterables. It seems like there should be handlers specifically for Iterable, just for the sake of completeness.

Separation of error propagation logic from data pump seems to be causing a race condition where occasionally one of the error tests will fail. I think I need a redesign to use a queuing strategy. That would probably clean up a lot of incidental complexity as well, but is a lot of rework.

Inspect the Deno source and make sure we are using the correct buffer size for transferring data.

I'd like to be able to shadow-run a process, sending stderr to Deno stderr and stdout to Deno stdout, and no stdin. This is a common use case for running CI/CD processes.

I would also like to be able to decorate the outputs with color and time, and perhaps a label.

Just extend the parameters so environment variables work like they do in Deno.run.

A good evening project.

Check out some standard signals and write some tests. It probably does not work as expected in its current form.

I am currently just inheriting.

Callers may want to suppress, scrape, or redirect. Need to work out various use cases and develop tests for each.

How would I add color and a timestamp on each line of stderr? This is a nice example.

Something is wrong with string Iterables. I am getting 50,000 to 100,000 lines per second peak performance on very small lines. It seems like the overhead per line should be much lower than 200usec per line.

I need to do some profiling to figure out where the performance hit is happening.

To get stderr to be near real time, it needs to be unbuffered.

I need a way to support passing messages from the stdin side of the process so that I can implement persistent worker processes.

Add unbuffered options for Byte Iterable. Extend these to unbuffered string options.

Modify the API to allow no group to be defined. In this case, a global group will be used. This works fine for processes that clean up correctly and for short runs. With that boilerplate gone, it feels a lot more like an extended bash script.

Current error processing is baked in. In a few cases, we don't want to throw an error on non-zero exit codes, and it may be desirable to control the error that is thrown.

This will be implemented as another function you can pass along with the alternate stderr handler function. This does not break backwards compatibility of the API.

Also, note that an error can already be thrown from the stderr handler.

This should all be documented.

I'd like to be able to use map and filter without wrapping with asynciter, and I would also like to be able to run another process from the output of a process.

Currently, you can pass an array of strings to the error handler function, and it will attach it to any error that is raised. This is a design error. It should be any type. We are just passing it along.

I cannot fix this immediately, so I am going to hang on to this and keep an eye on the related ticket for a while.

May be related to Aborting a CompressionStream/DecompressionStream leaks a resource #14212.

I have tried several ways to inject the ExitCodeError of the process into the
stream, including using flush of TransformStream. I settled on converting
the stream to AsyncIterableIterator, throwing the error after the stream had
been fully processed, and then converting all that back to ReadableStream -
because that mechanism is conceptually straightforward and should just work.

The error:

 ERRORS 

Non-zero exit code => ./tests/exitcode-errors/errors-1.test.ts:4:6
error: Leaking resources:
  - A text decoder (rid 5) was created during the test, but not finished during the test. 
    Close the text decoder by calling `textDecoder.decode('')` or `await textDecoderStream.readable.cancel()`.

The error is being suppressed where it occurs using:

sanitizeResources: false,

in the test definition.

Deno has deprecated Deno.run in favor of a new API, Deno.Command, which is in many ways a major improvement over the old API. It is, however, quite a disruptive redesign.

I have been meaning to look at simplifying my own API. Perhaps this is the time.

I am going to attempt a redo with the idea that I can make this a lot easier. Let's see how it goes.

Add a documentation section for short-form.

Also write the code-level comments.

The stderr processing function needs to pass bytes, not strings. This is because passing strings means we are buffering until a linefeed, which works for some output but not others. The default implementation should just write the bytes to stderr as they are read.