Error: UnableToCreateMemory about wasmer-go HOT 32 CLOSED

I'll give a try to reproduce the error!

from wasmer-go.

Thanks for running it. I'll try on bare metal. VirtualBox is the only common element between the 2 systems on which I tested. (Well, the only one I can think of.) Will get back on this.

from wasmer-go.

Hi @Hywan , we currently encounter the exact same issue in several of our Wasmer based projects.

@zupa-hu isolates the problem nicely with his example code:

package main

import (
	"fmt"
	"encoding/base64"
	wasm "github.com/wasmerio/go-ext-wasm/wasmer"
)

const WASM_BINARY_BASE64 = `
AGFzbQEAAAABBwFgAn9/AX8DAgEABAUBcAEBAQUDAQAQBhECfwBBgIDAAAt/AEGAgMAACw
dHBQZtZW1vcnkCABlfX2luZGlyZWN0X2Z1bmN0aW9uX3RhYmxlAQALX19oZWFwX2Jhc2UD
AApfX2RhdGFfZW5kAwEDc3VtAAAJAQAKCQEHACAAIAFqCw==
`

func main() {
	bytes, err := base64.StdEncoding.DecodeString(WASM_BINARY_BASE64)
	if err != nil { panic(err) }

	module, err := wasm.Compile(bytes)
	if err != nil { panic(err) }
	defer module.Close()

	for i:=0; i<1000000; i++ {
		fmt.Println(i)

		instance, err := module.Instantiate()
		if err != nil { panic(err) }
		instance.Close()
	}
}

I was able to reproduce the issue on my dev laptop Ubunbu WSL vm, and using three separate Hetzner cloud server instances: a clean Ubuntu 18.04, a Debian 4.19.98-1, and a Fedora 31 based image.

Installed the latest go and the latest go-ext-wasmer on each (clean and up to date) instance (using go get github.com/wasmerio/go-ext-wasm/wasmer).

Each machine gives the exact same result:

...
32717
32718
32719
32720
32721
32722
32723
32724
32725
32726
panic: Failed to instantiate the module:
    link error: unable to create memory: UnableToCreateMemory

goroutine 1 [running]:
main.main()
	/root/goProject/test.go:27 +0x24c
exit status 2

Would be super if you would be able to help us out with this issue!

from wasmer-go.

Hi @Hywan, thanks for looking into this once more. That is so strange - I seem unable not to reproduce the issue. Did you run the test on OS-es running within a VM? I can still give you access to a cloud instance, if you want - I am currently on Wasmer slack, so I could chat with you there - or I could just e-mail you login info.

Another option would be to think about what might be different between your setup and @zupa-hu and my setup. Or what we might do differently from you - we are doing clean installs like this:

#82 (comment)

from wasmer-go.

I'm sorry but I still can't reproduce your error :-/. I don't know how I can help you… Can you tell me what's your OS and arch please?

from wasmer-go.

If you want, I can give you access to a clean cloud server instance - just ping me at [email protected] - or currently at Wasmer's slack :)

from wasmer-go.

Just wanted to add that I'm also super keen on getting this worked out.

Thanks @robinvanemden on your input. Your suggestion with the cloud server instance is superb. @Hywan, if you could try to install it there, then we will either have a working setup or you would have a means to reproduce it. Progress either way. I really hope you can give it a go.

from wasmer-go.

I tried with macOS, Ubuntu and Debian (amd64 every time), with no success :-/.

from wasmer-go.

Contact me on Slack. We will try to investigate on this. I don't like having an unreproducible bug. Thanks for your patience!

from wasmer-go.

It took me 45s to execute your code (in a test), but it actually worked.

…
999974
999975
999976
999977
999978
999979
999980
999981
999982
999983
999984
999985
999986
999987
999988
999989
999990
999991
999992
999993
999994
999995
999996
999997
999998
999999
--- PASS: TestFoo (45.70s)

What version do you have? Also, what's your OS?

from wasmer-go.

Hm, interesting.

I'm running

• debian stretch 9.9
• go version go1.13.4 linux/amd64
• latest github.com/wasmerio/go-ext-wasm - 05371a4

Let me try on another machine.

from wasmer-go.

I tried on different notebook, on a freshly installed Ubuntu 18.04 that runs in VirtualBox. (Note: the other install also ran in VirtualBox.) I changed the attached memory as well to make sure but it did not affect the results.

To make it easier to reproduce, I embedded the wasm binary into the code as well:

package main

import (
	"fmt"
	"encoding/base64"
	wasm "github.com/wasmerio/go-ext-wasm/wasmer"
)

const WASM_BINARY_BASE64 = `
AGFzbQEAAAABBwFgAn9/AX8DAgEABAUBcAEBAQUDAQAQBhECfwBBgIDAAAt/AEGAgMAACw
dHBQZtZW1vcnkCABlfX2luZGlyZWN0X2Z1bmN0aW9uX3RhYmxlAQALX19oZWFwX2Jhc2UD
AApfX2RhdGFfZW5kAwEDc3VtAAAJAQAKCQEHACAAIAFqCw==
`

func main() {
	bytes, err := base64.StdEncoding.DecodeString(WASM_BINARY_BASE64)
	if err != nil { panic(err) }

	module, err := wasm.Compile(bytes)
	if err != nil { panic(err) }
	defer module.Close()

	for i:=0; i<1000000; i++ {
		fmt.Println(i)

		instance, err := module.Instantiate()
		if err != nil { panic(err) }
		instance.Close()
	}
}

I was running the above with go run hello.go. (I also tried with go build and then running, no difference.)

Can you confirm you were running the same wasm binary that I attached?

I'd appreciate any ideas on what to try.

from wasmer-go.

How much memory does the vbox have that you are running this in?

from wasmer-go.

Originally 6GB, I upped it to 8GB earlier, now to 10GB. No difference. I watched the memory usage of the process and it was 0.2% at the time it crashed, so that's definitely not the issue. Also, it crashes just below 2^15. I'm not sure how much memory is assigned at each instantiation, but I can't help to notice that 2^15 * 64KB memory page size gives 2GB.

Okay I just ran a test on this. When I keep all instances in memory, at 32700 instances the process uses ~144MB memory, meaning ~4.4KB / instance. Well, I don't feel any smarter now.. :)

from wasmer-go.

Tested with your WebAssembly module, and it still works. Here is the data I collected:

$ /usr/bin/time -l go test -test.v test/instance_test.go > /dev/null
        0.74 real         0.33 user         0.36 sys
  53809152  maximum resident set size
         0  average shared memory size
         0  average unshared data size
         0  average unshared stack size
     42717  page reclaims
         4  page faults
         0  swaps
         0  block input operations
         0  block output operations
         0  messages sent
         0  messages received
        18  signals received
        66  voluntary context switches
     25350  involuntary context switches

The test has eat 51.32MiB of memory.

From my understanding, it relates to an issue between your system and your memory. It fails around here, https://github.com/wasmerio/wasmer/blob/master/lib/runtime-core/src/memory/dynamic.rs#L37-L46, which is using the system API.

from wasmer-go.

Okay, I tried on bare metal, same issue. It's very suspicious that I have a systemic issue here. Here is the list of steps I have taken. I'm doing my best to provide the right level of detail. I hope I don't miss anything.

• Download and install ubuntu (ubuntu-18.04.3-desktop-amd64.iso)
• apt-get update
• apt-get upgrade
• apt-get install git
• apt-get install build-essential
• wget & untar go binary 1.13.4
• create and enter go project dir
• export CGO_ENABLED=1; export CC=gcc;
• go get github.com/wasmerio/go-ext-wasm/wasmer
• create file ./src/demo/demo.go with the go source code above
• go run demo.go

Output:

...
32704
32705
32706
32707
32708
32709
32710
32711
panic: Failed to instantiate the module:
    link error: unable to create memory: UnableToCreateMemory

goroutine 1 [running]:
main.main()
	/home/zupa/p/wasm/src/demo/demo.go:27 +0x25c
exit status 2

I suspect I'm not building with the right toolchain?

from wasmer-go.

Honestly, I've no idea at that point. Since I'm not able to reproduce (even with a virtual machine on Ubuntu), I don't know how I can help.

But most importantly, why do you need to generate such a large number of Wasm instances?

from wasmer-go.

Okay, got it, thanks for trying to help!

I'm building a platform that lets 3rd party devs run their own code. It's basically a website builder. Apps are tiny, some pages execute in the order of 100 apps, needing 100 instances per pageload. This adds up quickly. I have also found that adding delays don't help.

Can you confirm that I need nothing special for the build to work correctly? Is running apt-get install build-essential before-hand enough?

from wasmer-go.

If go get works, it means that everything is installed correctly. There is nothing special.

from wasmer-go.

Ok, thanks!

from wasmer-go.

Super! If it would help to have access to, for example, an Ubuntu cloud instance to reproduce, I would be happy to share login info to one. I presume it is not accidental that 32726 is near C's short signed int max of 32767 here?

from wasmer-go.

Hi @Hywan , I attempted to solve the issue with C.free(), but the result of this attempt remains the same: UnableToCreateMemory. My code:

package main

/*
#include <stdio.h>
#include <stdlib.h>
*/
import "C"
import (
	"encoding/base64"
	"fmt"
	wasm "github.com/wasmerio/go-ext-wasm/wasmer"
	"reflect"
	"runtime"
	"strings"
	"unsafe"
)


func main() {
	bytes, _ := base64.StdEncoding.DecodeString(`AGFzbQEAAAABBwFgAn9/AX8DAgEABAUBcAEBAQUDAQAQBhECfwBBgIDAAAt/AEGAgMAACwdHBQZtZW1vcnkCABlfX2luZGlyZWN0X2Z1bmN0aW9uX3RhYmxlAQALX19oZWFwX2Jhc2UDAApfX2RhdGFfZW5kAwEDc3VtAAAJAQAKCQEHACAAIAFqCw==`)
	module, _ := wasm.Compile(bytes)

	instance, _ := module.Instantiate()

	fmt.Println("@@@@@@@@@@@@@@@@")
	InspectStruct(instance, 0)
	fmt.Println("@@@@@@@@@@@@@@@@")

	instance.Close()

	fmt.Println("@@@@@@@@@@@@@@@@")
	InspectStruct(instance, 0)
	fmt.Println("@@@@@@@@@@@@@@@@")

	for i:=0; i<40000; i++ {
		runWasm(&module)
		fmt.Println("*** ", i)
		runtime.GC()
	}
}

func runWasm(module *wasm.Module) {

	instance, err := module.Instantiate()
	if err != nil { panic(err) }

	runtime.SetFinalizer(&instance, finalizer)
}

func  finalizer(instance *wasm.Instance) {
	fmt.Printf("### a finalizer has run for %+v\n", instance)

	instance.Close()

	M := reflect.ValueOf(*instance).FieldByName("Memory")
	mp := reflect.Indirect(M).FieldByName("memory").Pointer()
	C.free(unsafe.Pointer(mp))

	//ip := reflect.Indirect(reflect.ValueOf(*instance)).FieldByName("instance").Pointer()
	//C.free(unsafe.Pointer(ip))

	instance = &wasm.Instance{
		Exports: nil,
		Memory:  nil,
	}
}

func InspectStructV(val reflect.Value, level int) {
	if val.Kind() == reflect.Interface && !val.IsNil() {
		elm := val.Elem()
		if elm.Kind() == reflect.Ptr && !elm.IsNil() && elm.Elem().Kind() == reflect.Ptr {
			val = elm
		}
	}
	if val.Kind() == reflect.Ptr {
		val = val.Elem()
	}

	for i := 0; i < val.NumField(); i++ {
		valueField := val.Field(i)
		typeField := val.Type().Field(i)
		address := "not-addressable"

		if valueField.Kind() == reflect.Interface && !valueField.IsNil() {
			elm := valueField.Elem()
			if elm.Kind() == reflect.Ptr && !elm.IsNil() && elm.Elem().Kind() == reflect.Ptr {
				valueField = elm
			}
		}

		if valueField.Kind() == reflect.Ptr {
			valueField = valueField.Elem()

		}
		if valueField.CanAddr() {
			address = fmt.Sprintf("0x%X", valueField.Addr().Pointer())
		}

		fmt.Printf("%vField Name: %s,\t Field Value: %v,\t Address: %v\t, Field type: %v\t, Field kind: %v\n",
			strings.Repeat("\t", level),
			typeField.Name,
			address,
			typeField.Type,
			valueField.Kind())

		if valueField.Kind() == reflect.Struct {
			InspectStructV(valueField, level+1)
		}
	}
}

func InspectStruct(v interface{}, level int) {
	InspectStructV(reflect.ValueOf(v), level)
}

Maybe the export map is not fully freed? Or might this be an Rust based issue after all? I did find a related issue in the Wasmer repo, so I referenced the current issue there as well.

from wasmer-go.

Sure!

I tested the code on my laptop's Ubunbu WSL VM, and on three separate Hetzner cloud server instances: clean Ubuntu 18.04, Debian 4.19.98-1, and Fedora 31 based images, all 64 bits.

On each I just install the OS, go1.14.2 linux/amd64, run "go get github.com/wasmerio/go-ext-wasm/wasmer" and the code above.

from wasmer-go.

Out of curiosity, which which OS/arch are you using?

from wasmer-go.

Hi @Hywan!

We tried to both contact you with @robinvanemden on Slack yesterday but as you did not respond I checked your account's past comments and I see zero activity. Maybe we contacted you in the wrong slack channel or the wrong account?

from wasmer-go.

Checking :-).

from wasmer-go.

Hi @Hywan! Not sure if I missed any communication between you; did you get a chance to test drive it on the Hetzner instance @robinvanemden has set up?

from wasmer-go.

Hi @zupa-hu - the Hetzner instance is still up, but as far as I can see, @Hywan has not yet logged in. But maybe this issue will be resolved by #140 (comment) - fingers crossed.

from wasmer-go.

Thanks for the update @robinvanemden! Interesting - fingers crossed then! :)

from wasmer-go.

I set up a new machine and tried again, without any success so far. But I've restarted the rewrite today, and I believe it will solve your issue since Wasmer has changed in depth.

from wasmer-go.

@Hywan is that machine a cloud instance as well? Maybe you can give us access to it so we can see how you have done it? I'm pretty sure we do something differently if it works for you but doesn't for us.

Anyways, I understand if the new release fixes this soon there is little point in investigating this. Is there an ETA?

Then @robinvanemden can kill the instance? Or do you still plan to test drive it there?

from wasmer-go.

I just tested this again with the last wasmer-go release and can't reproduce the bug.

I'm closing this issue for now but feel free to reopen if needed.

from wasmer-go.