rfc: better native bindings about gno HOT 13 OPEN

One concern is that precompilation (I explained better below how that works) would probably have more trouble if they're implemented this way.

Yes probably, let's do it with function without body + Init first. Then we'll see if we can simplify with a only function without body.

Thx for the explanation.

from gno.

Related with #239 and #498

Good idea, but let's create a PoC to make informed decisions for next steps.

from gno.

I've incorporated the latest ideas in the comments into the OP, and also added a lengthier example section showing exactly how I envision both stdlib code and realm code (and precompiled).

Feedback is very welcome. I'll start working on a PoC soon.

from gno.

For someone who has often felt confused by the issues you mention, I really appreciate you took the time to investigate the whole topic and gave us a proposal of improvement!

I like the idea of using the function without body for that. So you can define the body whether in go directly or via the Init function. If you use Init it's because you want to access the context right ?
Maybe it would be nice to allow an alternate function signature that takes the gno.Machine as the first parameter, thus we can remove the injection via Init.
If I retake your example, this would also be valid mapping :

// json.gno
func MarshalJSON(v interface{}) ([]byte, error)

// json.go
import (
  "encoding/json"
  gno "github.com/gnolang/gno/gnovm/pkg/gnolang"
)

func MarshalJSON(m *gno.Machine, v interface{}) ([]byte, error) {
    return json.MarshalJSON(v)
}

But maybe that's a little far fetched and/or wouldn't work properly.

Otherwise, if the native functions are more complex, then the package "$gnoimportpath/gnovm/stdlibs/encoding/json" (for instance) is imported and the native function called.

I still don't really understand why do you mean by that. What the kind of complexity are you referring to?

from gno.

I like the idea of using the function without body for that. So you can define the body whether in go directly or via the Init function. If you use Init it's because you want to access the context right ? Maybe it would be nice to allow an alternate function signature that takes the gno.Machine as the first parameter, thus we can remove the injection via Init.

That idea also passed through my mind. I guess it could work, though I don't have the expertise to know if adding m *gno.Machine as the first argument covers most cases. From what I can see in stdlibs/stdlibs.go, pn is not used inside of the native functions themselves and store should also be available through m.Store where necessary, so I agree with you that it could be a viable solution, but I'd like to have some extra feedback here.

One concern is that precompilation (I explained better below how that works) would probably have more trouble if they're implemented this way. With the Init function, we can instead provide a Gno version and a precompiled version (like we're doing now for instance for std - where references to the std package are replaced with stdshim when precompiled)

Otherwise, if the native functions are more complex, then the package "$gnoimportpath/gnovm/stdlibs/encoding/json" (for instance) is imported and the native function called.

I still don't really understand why do you mean by that. What the kind of complexity are you referring to?

If the matching native function simply refers, for instance, to another Go standard library function, the easiest thing to do when precompiling Gno code is to replace the call to the function in the Gno stdlib to that in the Go stdlib

// stdlibs/strconv/strconv.gno

func Itoa(i int) string

// stdlibs/strconv/strconv.go

func Itoa(i int) string {
	return strconv.Itoa(i)
}

Then when we use strconv.Itoa in gno code, when we precompile it it is replaced with the go stdlib equivalent.

// example.gno
import "strconv"

func Render() string { return strconv.Itoa(11) }

// example.gno.gen.go
import "strconv"

func Render() string { return strconv.Itoa(11) }

However, say we change Itoa to also have other side effects, say a fmt.Println

// stdlbs/strconv/strconv.go

func Itoa(i int) string {
    fmt.Println("Hello!")
    return strconv.Itoa(i)
}

To maintain consistency between precompiled and gno execution, the precompiled version should become this one:

// example.gno.gen.go
import "github.com/gnolang/gno/gnovm/stdlibs/strconv"

func Render() string { return strconv.Itoa(11) }

from gno.

I think that maybe we don't need any precompile step to do, and we can just execute native code when interpreting the code on the VM if we detect that the function called is from the standard library. We will need to give to these functions a specific gas fee too.

from gno.

@ajnavarro we don't do precompile steps in normal executions, but precompilation is a feature supported by cmd/gno and I think we should continue to support.

And yes, indeed while we still need to sort out gas fees for native function it currently works like you said. Except that now everything's in a big file containing all the native functions (mostly stdlibs/stdlibs.go)

from gno.

One thought @tbruyelle -- I just thought about it and your approach may actually be better and the issue I was referring to easily resolvable. We can simply make calls to the functions which use *gno.Machine use nil when precompiled, so ie. this is how I would implement AssertOriginCall

package std

import // ...

func AssertOriginCall(m *gno.Machine) {
  if m == nil {
    panic("std.AssertOriginCall called from precompiled code")
  }
  if len(m.Frames) != 2 {
    m.Panic("invalid non-origin call")
  }
}

Then when we precompile std.AssertOriginCall() in gno code, it gets converted to std.AssertOriginCall(nil).

from gno.

Indeed nice approach !
I never touched the precompile part, do you think it's easy to convert method calls like that ?

from gno.

I'm also not exactly knowledgeable about precompile internals, but starting from an AST tree and pregenerated mappings this should be doable without too much magic :)

from gno.

Leaving a different approach here, just to continue the discussion and checking pros and cons:

using the following example:

Example user of std package, and generated precompiled code

// examples/gno.land/r/r1/r1.gno
package r1

import "std"

func SampleFunction() {
	std.EncodeBech32("r1", [20]byte{})
	std.Hash([]byte(std.GetOrigCaller())
}

// examples/gno.land/r/r1/r1.gno.gen.go (ie. precompiled)
package r1

import (
	"github.com/gnolang/gno/gnovm/stdlibs/std"
)

func SampleFunction() {
	std.EncodeBech32("r1", [20]byte{})
	std.Hash([]byte(std.GetOrigCaller())
}

• std package is implemented in go, so we can use it as a dependency for go precompiled code, and internally on the VM.
• There will be no difference between the go and gno code, so the only thing that will change is the import.
• When code is executed inside the VM, we can use reflection to call functions on specific packages, std in this case.
• No need to keep track of a huge list of methods on the VM, any change to std package will be reflected on VM and generated go code.
• not crossed dependencies VM <-> std package

WDYT?

from gno.

@ajnavarro

One question, from your example: how does std.GetOrigCaller() gather access to *gno.Machine? Especially for the std package, many function calls require access to the VM's data. Otherwise, the proposals look very similar

from gno.

We could call something like std.With(vm).GetOrigCaller() from the VM to make it available. When the code is executed from Go, we can leave a default mocked VM as the default or use any other implementation. But these might be ideas for the future. std.GetOrigCaller() will call under the hood to std.With(&MockVM{}).GetOrigCaller() (or a nil VM) when called from Go, and that call to std.GetOrigCaller() will be translated to std.With(vm).GetOrigCaller() when called from Gno VM using reflection.

What I'm trying to do here is to avoid having code generation, and make everything as straightforward as possible to allow future IDEs to autocomplete and things like that.

Feel free to discard the idea if you feel it doesn't fit well (you have a bigger context here than me). I'm just hoping that this proposal helps to validate the main one.

from gno.