darksidesync's Introduction

DarkSideSync

Usecase

I wanted to create a lua binding for a library. The library uses callbacks to report on its async operations. The main issue however was that the library created its own threadpool and the callbacks could be executed on any one of those. And that, sooner or later, will wreak havoc with Lua's single threaded nature.

So while Lua's main thread is doing its Lua thing, it needs to be notified that there is some other thread that tries to deliver some result it needs to handle.

To support this DarkSideSync (DSS) was created, DSS has no use on its own, it will only support other libraries in handling async callbacks.

Main principle

Whenever a background thread has some data to deliver it will call a DSS method to temporarily store it in a queue. Whenever Lua has time to poll for data from this queue, the background library will get a direct call from DSS to deliver its data to Lua.

It supports async callbacks that just deliver data (the callback thread does not get blocked) and async callbacks that also expect a result (in this case the thread gets blocked until the Lua side thread has delivered the results)

There is an optional notification using a UDP packet, which is an easy way to wake up Lua from a select() network method.

Reference

Documentation can be found in the repository and on the GitHub page, a general description and some diagrams are also available.

Positive

It is very generic and cross platform. Because DSS takes care of threads, locks and (optionally) sockets, a library binding for an async library may require no platform specific code and still work cross-platform
It is setup as a separate library, loaded from Lua, no C links. It, sort of, dynamically extends the Lua C api
Supports async callbacks that only deliver data (the callback thread is not blocked)
Supports async callbacks that need a response (the callback thread is blocked until the Lua side response has been delivered)
Supports multiple async background libraries simultaneously
Supports multiple concurrent Lua states
It has been setup with the intend to support multiple versions of the DSS API, so in the future multiple background libraries can use a single DSS library, while talking to different versions of the API
The notification is also platform independent and even network library independent (eg. not bound to LuaSocket) any network library supporting UDP will do (this is the reason for not using file descriptors or pipes)

Negative

Notification using UDP packets requires some overhead, so for a very high number of callbacks it might be better to only use polling

Copyright & License

License is the same as Lua 5.1; MIT license.

The name

Lua = moon, dark side of the moon, externally managed threads that Lua cannot reach, you get it...

Changes

1.0 13-may-2013

Initial released version

darksidesync's People

Contributors

Stargazers

Watchers

darksidesync's Issues

Drop poll() results in table

Instead of having a varargs function (poll() can return any number of results), simply return a single table holding a list of these results.

Because the first results of poll() are fixed and only the latter part is varargs, it is impractical to have to load everything into a table first and then remove the first elements one by one. If the varargs element is dropped into a table, then the poll() number of results will be fixed.

data element is not threadsafe

The library attached pData element used in many api functions, is easy to get and set. But threadsafety has not been taken into account. So for now an application using it must use its own lock when accessing the contents of the element.
This should be incorporated by creating a mutex for each library to handle this. Then the client libraries can be really platform independent.

simplify api: remove setresult()

Currently when a async callback expects a result, then DSS will add a userdatum to the list of results (1st position). The Lua side code is expected to call the setresult() method of that userdatum with the results to return to the async callback.

instead of calling it setresults it could be called __call (meta method) which would effectively turn the userdatum into a callback function. Which is easier to understand for users.

decode function must clear unused elements on stack

when the decode callback is called, and for example, it places 3 elements on the Lua stack, and reports 2 results, then all 3 elements are passed on to Lua.
This usually fails because the first arg must be a lua function, which it then probably is not.

workaround; make sure the decode callback removes any other elements from the stack before returning

Recommend Projects

tieske / darksidesync Goto Github PK