alogani / nimgo Goto Github PK

Normally, the test here :
https://github.com/Alogani/NimGo/blob/main/tests/specificcases/stackoverflow1.nim

Should print the stacktrace on stderr. It is not the case on windows

Already done it, but forgot to save this addition, will be easier to add again

Explicit logic for timeouts is both cheaper at runtime and easier to understand than these academic "timeout tasks" that in the end don't work because nobody thought about it anymore.

Continuation of #9

And the alikes for readLine, readChunk, etc.

On posix: Deallocation of virtual memory allocated with mmap by a factor >10 slower than virtual memory allocated with alloc. But memory allocated with alloc is incompatible with mprotect

@Araq Do you have some knowledge of those memory internals that could explain why alloc is so much faster but incompatible with PROT_NONE pages ? That order of magnitude is not negligible

those objects might stay as ref objects:

• GoStreams
• GoSocket (Socket from std/net is a ref)
• GoProc

Coroutines callback environment is protected thanks to https://github.com/Alogani/NimGo/blob/main/src/nimgo/private/safecontainer.nim

However, this is not idiomatic and good Nim. It should use system.protect() and system.dispose(x) instead.

Handling stack memory has multiple aspects into achieving a reliable, secure, and explicit error management.

Growable stack

This might not be easy to implement. So the best solution is certainly to rely on exclusively on having a very big stack of virtual memory. There is still the problematic to support lower end device, so we shall determine the maximum amount of storage to allocate (for example sysconf(SC_PAGESIZE) * (sysconf(SC_PHYS_PAGES) on linux) and to support operating systems that can't support virtual memory (which ones ?)

Protection against overflow

The last allocated page for the stack should be marked as protected:

• m_protect on linux
• VirtualProtect on windows

Error message in case of stackoverflow

When a page of memory is protected, the OS will throw an exception like SIGESEGV on linux. This can be catched but cannot be recovered safely (at least on linux). So SIGSEGV shall be catched to permit to write the stacktrace, the error message. However, SIGEGV from stackoverflow should be distinguished from other SIGSEGV (how ?)

Some resources:

• https://mikemarcin.com/posts/coroutine_a_million_stacks/
• https://rethinkdb.com/blog/handling-stack-overflow-on-custom-stacks

If you have time, don't hesitate to run the test suites and the bench on windows with NimGo and tell me what are the errors messages.

If you know how and it is possible, you can also tell me how to add a windows github hosted runner

Any proposition of fix are also welcome ;-)

when defined(windows):
  {.warning.}
  exit(0)

should be added on all tests that doesn't currently works on windows

Non working modules can be found here : #26

For now, exception inside a coroutine are simply rethrown. But not letting the coroutine finish by itself can induce memory leakage.

This internal behaviour will be fixed in next release by rethrowing exceptions after returning to parent's coroutine

A user in the Nim Forum suggests to rename NimGo for a better name. I think he might be right.

Here is his post https://forum.nim-lang.org/t/11720#75214

I think NimGoroutines shall be a good name. It expresses both the concept of Coroutines, the inspiration of Go in that specific feature (Goroutines), and is not restricted to only IO (because NimGo will also leverage Os Threads in future release).

Please read here the definitions I use before responding to this post. If we don't agree on what we talk, we won't talk about much.

Questions

1. Should the EventLoop and DispatcherLoop be merged together ?
2. If they are separate, should the EventLoop run in its own thread or each thread would run a shared EventLoop ?
3. Should each thread have its own task Dispatcher in addition to global task Dispatcher ?
4. Should it have a fixed number of Threads ? If yes, how many ? If no, determined by what ?
5. What design should the DispatcherLoop have ?

• FIFO for all tasks, except LIFO for last task for better caching/responsivness ?
• FIFO only ?

6. Other considerations ?

Proposition of response

If I read correctly this paper shared from @mratsim : https://assets.ctfassets.net/oxjq45e8ilak/48lwQdnyDJr2O64KUsUB5V/5d8343da0119045c4b26eb65a83e786f/100545_516729073_DMITRII_VIUKOV_Go_scheduler_Implementing_language_with_lightweight_concurrency.pdf

The answers could be :

1. No
2. Separate thread for I/O pool
3. Yes
4. Should have 61 Os Threads (why ?)
5. FIFO with LIFO for last item

However I am wondering how well a single selector can scale. But having multiple selectors by thread is trickier, even if the same fd can be registered in multiple dispatcher (with special considerations for thread safety, but this might be implementation specific).

This is the last major work before work on multithreading (#21) could begin.

The goal is to create a portable API to query asynchronous events. This will likely significantly redesign the public API of the nimgo/eventdispatcher library, and hopefully the API won't require many changes after this update.

• defining the API
• Writing startProcess for posix
• Handle the pseudo terminal specific case for posix
• Add convenience function run
• Implements the window's low level API (help wanted)
• etc?

Because the library won't be strictly async, the keyword named `goAsync might not be appropriate ?

Here are some propositions:

• go
• goTask
• goAsync
• goNim
• launch
• startTask
• coSpawn

Although I highly prefer four spaces ident, this is the norm in Nim and it will facilitate contributions. I have modified my VsCodium settings to help me see them better.

import nimgo, nimgo/gofile

var file = currentSourcePath()
var gf = openGoFile(file)

withEventLoop():
    goAsync proc() =
        echo gostdin.readline()

src/nimgo/eventdispatcher.nim(424) addInsideSelector
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

Coroutine ref object in fact wraps a McoCoroutine struct defined in minocoro.h This struct holds low level information of the coroutine. It uses calloc/free. It should be allocated instead on Nim's side with Nim's alloc/dealloc