Need help using uvw together with boost::coroutines2 about uvw HOT 28 CLOSED

@abrownsword

If you are willing to collaborate, it would be great.
I'm not actively using the library anymore, so my free time is filled with other projects. Help would be appreciated!! ;-)

from uvw.

Please, attach the error message (if any) to the issue. It crashes tells nothing about it.

from uvw.

Null pointer dereferences, which results in bad memory access at source().get(). No error message.

The code expects loop->run<uvw::Loop::Mode::ONCE>() waits for next fs event, but it won't so that sink(std::move(event)); inside the event handler is not called.

The first awaitEvent for fileReq->open returns successfully. The second one didn't.

The problem may relate to loop->run<uvw::Loop::Mode::ONCE>() but I don't know how to fix it

from uvw.

ONCE is meant for one shot loops to dequeue what's pending or at least give them a tick.
You should use plain run in your case. It hangs forever because of the FsEventHandle you attach to the loop and forget to close. Keep a copy of the handle and close it when you are done.

I'm closing the issue in a few hours for this isn't a problem in uvw.
We can continue discussing here but it doesn't make sense to keep it open.

from uvw.

Thanks for your reply.

It hangs forever because of the FsEventHandle you attach to the loop and forget to close.

But I don't mean to close the fsEventHandle. I want to watch changes in the directory continuously.

I'm closing the issue in a few hours for this isn't a problem in uvw.

It's ok

from uvw.

Well, if you don't want to close the handle, you don't want to stop the loop.
The handle won't work as well when the ONCE tick is returned. It requires a running loop to work, you know that?

from uvw.

I'm not sure that I understand what you mean.

It requires a running loop to work, you know that?

Yes I know it.

The handle won't work as well when the ONCE tick is returned.

Only before the next loop->run. To my understanding, the control will return to libuv when starting a loop. Libuv will choose the event to emit, including FsEventEvent(when detects a change) and FsEvent(when any I/O operation finishes). That's how async programming works, isn't it?

from uvw.

Libuv will choose the event to emit, including FsEventEvent(when detects a change) and FsEvent(when any I/O operation finishes).

libuv works with callbacks actually, it's uwv that turns everything into events.
That being said, what you say happens if and only if you have a running loop.

As long as I can see, the problem is that you need a persistent loop because you want it to run multiple requests one after the other. On the other side, you don't want it for it's blocking.
Probably you should reconsider your design, you can't have the cake and eat it after all. :-)

from uvw.

libuv works with callback actually, it's uwv that turns everything into events.

That's not very easy to use, to be honest. As you can see in awaitEvent, you have to bind two different event handler if you want to handle the error too ( it's a must for solid code ). If one of them is called, you want to erase both of them. In this case, Emitter::once is not very useful.

you want it to run multiple requests one after the other. On the other side, you don't want it for it's blocking.

That's what I want. IO operations for the same file must be run in sequence; but for difference files, they can be handled concurrently. Otherwise synced IO operations are enough.

Probably you should reconsider your design, you can't have the cake and eat it after all. :-)

I don't think there is any design issue. Although I'm not familiar with libuv, I work with nodejs often. It's a common pattern for these requirements

from uvw.

That's not very easy to use, to be honest. As you can see in awaitEvent, you have to bind two different event handler if you want to handle the error too ( it's a must for solid code ). If one of them is called, you want to erase both of them. In this case, Emitter::once is not very useful.

C++ is a typed language.
Alternatives are to ask either for callbacks that accept two parameters (both the error and a potentially invalid packet of data, like the ones in node.js) or for two different callbacks.
In the latter case, a common pattern is to use on for all the callbacks but for the error one for which once is usually fine.

What I mean is that you have a design flaw with your approach targeted to this model.
You are confirming it too. You cannot use libuv as you would use node.js, C and JavaScript are pretty different languages.

Btw, let's try to solve your problem thinking from the libuv side for a moment.
You don't want to get a running loop with a handle for it's blocking and you don't want to close the handle, but it could happen that you have two requests running one after the other of which the former creates the latter.
All what you can do in this scenario is having two shots of your loop as once. In the worst case, the second one will be a no-op.
Back to uvw... The answer is the same.

If you want the cake and eat it, you must find a compromise.

from uvw.

To be clear, your problem is interesting.
It's an use case of libuv I've never seen before (I didn't use it that much probably) and I'm willing to help you.

from uvw.

I'm willing to help you.

Thank you very much

a common pattern is to use on for all the callbacks but for the error one for which once is usually fine.

I dont quite understand what you mean. If you mean once for error and on for others, the error event can be much harder to handle because you can't know which operation leads to the error.

It's an use case of libuv I've never seen before

It's a common use case indeed. Have you written web services? Server will handle to multiple requests concurrently. But code for single request usually has dependencies internally and can be run in sequence only. For example you must open a database before reading records from it

from uvw.

It's a common use case indeed. Have you written web services?

Yeah. I meant the combination of boost coroutines and libuv is something I've never done.

Server will handle to multiple requests concurrently. But code for single request usually has dependencies internally and can be run in sequence only. For example you must open a database before reading records from it

Speaking about your example, a web service, what I don't understand is why you can't keep your loop up and running. I usually do this and that's all. Your description asks for it indeed.
That's why it's interesting and I think it's due to the fact that you uses boost coroutines, that is the use case I've never seen.
The loop is closed. ;-)

It looks like you are using an external tool for the (let me say) web services part and you use internally libuv only to monitor files, am I wrong?

from uvw.

It looks like you are using an external tool for the (let me say) web services part and you use internally libuv only to monitor files, am I wrong?

No, this is the first time that I use libuv. All I want to do is getting familiar with libuv & uvw by writing a demo ;)

from uvw.

:-D

Ok, so, can I ask you why you want to use both libuv and boost coroutine together?
I thought you had constraints for that.
I mean, coroutines and async are two paradigms that I've never seen used together in the same piece of code.

from uvw.

I mean, coroutines and async are two paradigms that I've never seen used together in the same piece of code.

They are often used together because it's a common pattern using coroutines ( actually semi-coroutine or fiber ) to simplify coding with async operations. That is using sync syntax to write async code. To be simpler: KILL callbacks.

auto fileReq = loop->resource<FileReq>();

fileReq->open(path + "/" + event.filename, Flags<FileReq::FileOpen>(FileReq::FileOpen::RDONLY), S_IRUSR);

awaitEvent<FsEvent<FsReq::Type::OPEN>>(loop, fileReq); // wait for `fileReq->open` finishing asynchronously. Code below is blocked before `fileReq->open` is finished but the thread isn't. It can handle other FsEvent events concurrently.

fileReq->stat();
const auto fileSize = awaitEvent<FsEvent<FsReq::Type::STAT>>(loop, fileReq).stat.st_size; // wait for `fileReq->stat` asyncly, no callback.

const auto progress = progresses.emplace(event.filename, 0).first->second;

fileReq->read(progress, static_cast<unsigned int>(fileSize - progress));
const auto readEvent = awaitEvent<FsEvent<FsReq::Type::READ>>(loop, fileReq); // like before
const auto fileContent = std::string(readEvent.data.get(), readEvent.size);
std::printf("[Content appended] %s: %s", event.filename, fileContent.c_str());

fileReq->close(); // like before
awaitEvent<FsEvent<FsReq::Type::CLOSE>>(loop, fileReq);

There are at least javascript, C# ( and other .NET languages ), python, dart, kotlin that implemented a feature called async function to do so in programming language level. There is also a library that implement this pattern using semi-coroutine / generator because async function can be, and internally is, implemented in semi-coroutine.

There is proposal for C++ too. There are even implementations in Visual C++ and clang. And there is even a library that apply the pattern to libuv. But it's still a proposal. Its paper is not stable and gcc doesn't support it yet.

I'm a frontend programmer. I write await things almost every workday. C++ is just my hobby 😄

from uvw.

Well, yeah, coroutines are around since a long time ago. What I'm saying is that I've never seen them used along with libuv and async stuff in general.
I worked only on projects entirely built on top of coroutines or designed around async stuff, but never used them together. It sounds interesting indeed.
Mainly for its kill callbacks part!! 👍
Something to use to feed my mind in the future, thank you for the hints.

Well, that being said it looks like you are more experienced than me in mixing the two paradigms, so I don't know how I can help you actually.
I'm sorry. I don't see how you can use the loop of libuv the way you are trying to do, as I said.

from uvw.

I'm sorry. I don't see how you can use the loop of libuv the way you are trying to do, as I said.

It's enough. Thanks for your reply again.

from uvw.

You're welcome. Next time feel free to contact me directly if you need help instead of opening an issue.

from uvw.

I create a new loop inside FsEvent event handler and it works: https://pastebin.com/mwsxzuE0
Don't know if there is any performance impact though.

Note: I found that FileReq::stat calls uv_fs_fstat internally, thus emits FsEvent<FsReq::Type::FSTAT>, which was unexpected.
I think FileReq::stat should be FileReq::fstat or FileReq::stat should call uv_fs_stat internally and emits FsEvent<FsReq::Type::STAT>.

By the way, do you know something about this problem? It's opened for about 3 days but got no response.

EDIT: I found that two loop would block each other, which resulted in synchronous, completely unusable code. Will find the solution later.

from uvw.

FsReq uses uv_fs_stat, FileReq uses uv_fs_fstat. Functions used under the hood look fine.
It's only a naming problem between stat and fstat? I suspect that changing the name in FileReq would break already existent code.

from uvw.

It's only a naming problem between stat and fstat? I suspect that changing the name in FileReq would break already existent code.

But it's misleading. The name of event and function should match each other.

EDIT: I realized that the code did have a design flow. I have rewritten the code now it works as expected ( unblock I/O operations between different files ). There are lots of ugly hacks because C++ doesn't have GC and true closure. It's better to wait C++ Coroutine TS

from uvw.

This is a very interesting blog post and it would very interesting to see this approach integrated into uvw along side it’s current paradigm: https://blogs.msdn.microsoft.com/vcblog/2017/02/02/using-ibuv-with-c-resumable-functions/

from uvw.

This is a very interesting blog post and it would very interesting to see this approach integrated into uvw along side it’s current paradigm: blogs.msdn.microsoft.com/vcblog/2017/02/02/using-ibuv-with-c-resumable-functions

It has been mentioned in my post ;)

And there is even a library that apply the pattern to libuv.

from uvw.

Sorry, I missed your link. The concept would be substantially more effective if implemented directly in uvw, and leveraging the Coroutines TS (VS & CLang, not sure about gcc).

from uvw.

It would be interesting, I agree. Unfortunately I've not much free time to work on it currently.

uvw is (let me say) under maintenance, that is I'm updating it to the new versions of libuv so as to keep the library up-to-date but I'm not actively developing new features (unless I find companies interested in funded development or users willing to help).
If you want to help me and open a PR with a proof of concept of the work, I can review it and help to finalize the integration.

from uvw.

@abrownsword @CarterLi

from uvw.

Sure. It won't be in the immediate future, but it looks like a fun enough project to warrant some effort. I think it could be a substantial value-add that uvw could offer to distinguish itself further from other networking models. Writing protocol code in "straightline" fashion is very compelling.

The coroutine TS isn't supported by gcc yet (I checked), but it sounds like its likely to make it into C++20, in which case all major compilers should have support for the TS prior to 2020. The Clang and VS implementations look like they're already in good enough shape to do serious work with them. I don't like having Boost as a dependency, so would rather stick with the TS.

from uvw.