Consider using libffi to allow for external calls about miri HOT 16 OPEN

Hey! I've started working on this issue -- here is a design doc of our proposed plan so far: link here. We'd love your feedback/comments on the proposed approach!
@oli-obk @RalfJung

Thanks! :)

from miri.

Previously suggested on IRC based on @cmr's mentions in a different context.
The closure example presents what I think is one of the most valuable features of this library: the ability to JIT a C function which hardcodes extra owned values, a whole closure masquerading in a function pointer.

from miri.

@drbo We've definitely been considering that.

For a bit of context, there are two versions of Miri we'll have in the future. If the Rust teams decide to use a MIR-based constant evaluator, some version of Miri will get incorporated into rustc, and that version will never use libffi because you want to ban arbitrary FFI calls in constant evaluation.

But it would be a lot of fun to add libffi to this external version of Miri. I would probably wait until after any Miri → rustc merging happens, though.

from miri.

Now that Miri is indeed in rustc (wooooo!), any further thoughts on this?

If I understand correctly, FFI would be necessary for us to use Miri as the basis for a Jupyter kernel.

from miri.

@BatmanAoD I'd advise against using miri for such a purpose - have you looked at webassembly?

from miri.

@eddyb I don't actually know what the requirements are for a Jupyter kernel; I was basing my understanding on an old discussion about using rusti instead of Miri for the basis of the kernel.

I can't actually find documentation on how the kernel API works, though (I probably just haven't spent enough time looking). It may be that communication between JupyterLab and the kernel is entirely over HTTP.

In any case: I'm not sure how WebAssembly would solve the problem. The Jupyter kernel is a REPL for a given language; the goal of a Rust kernel would be to permit people to create Jupyter notebooks that would use the Rust language instead of IPython (or any of the other languages that are already supported via kernels). So the need is for a Rust REPL that supports whatever interface JupyterLab uses.

from miri.

@BatmanAoD Oops, I misunderstood "kernel" as "OS kernel", not an IPython-like thing.
Then miri is likely the way to go, and I agree libffi would be needed.

from miri.

@eddyb Yeah, I think the terminology is a bit confusing!

from miri.

This project is almost ready (as in compiling and testing on Travis) again, I have a PR against rustc ready that should fix all the tests. After that we can accept new PRs. So if anyone wants to hack up a prototype using libffi to execute some c functions that we're currently having horrible fake support for (like syscall), that would be totally awesome. Things like mallocare a little harder though, but with a new allocation kind totally doable without touching rustc.

from miri.

@oli-obk I think pointers in general are hard. Various syscalls take pointers (e.g. read and write) while others return them (e.g. mmap) - so malloc is not unique in this difficulty.
It's easier to detect miri pointers being passed to FFI, and do something about it, than it is to handle the other way - I'm thinking maybe miri could be self-aware (e.g. via /proc/self/maps) and allow access via integer pointers to memory regions that would surely not segfault (that would, amusingly, allow corrupting miri itself - which C code could do anyway, so it's not introducing a new issue).

from miri.

@eddyb If it were possible to guarantee that improper writes would segfault, wouldn't that be preferable to corrupting the Miri process without terminating it?

from miri.

@BatmanAoD My point was that integer pointers could be used to corrupt miri itself - regardless of how you handle segfaults, the values for which corruption could happen are generally hard to detect, and pointless anyway, because you're also calling into C, which can do anything anyway.

from miri.

Yea.... I don't think we should care too much about users doing horrible things. If you do care, don't enable the ffi feature ^^

from miri.

Any news on this?
Would love to use miri on some projects that have a lot of FFI calls.

from miri.

libffi is due for a new release. (GHC and probably other projects have had to vendor it which is a pain.) If you do end up using it, please lobby upstream so we can all get a working release!

from miri.

@emarteca that's a very welcome surprise. :-)
Let's make a new issue to specifically discuss your project: #2365

from miri.