Say we have a module declared like this:

[[module]]
route = "/hello"
module = "my.wasm"

It would be nice for the module to be able to claim several sub-routes, and have those mapped automatically. For example, the module may want to add a /hello/foo path that maps to the function foo() and a /hello/bar path that maps to bar().

We could create a reserved function name that WAGI can call to register subpaths. For example, we could map the _routes() function for this. The expectation is that the function would return a TOML document that described its routes:

[[route]]
path = "/foo"
entrypoint = "foo"


[[route]]
path = "/bar"
entrypoint = "bar"

The module's base path (/hello) would then be combined with the _route()'s relative paths to form absolute paths (/hello/foo and /hello/bar). This would prevent a module from usurping another module's namespace. (And would be easier to debug)

The TOML structure makes it possible for us to add new dynamic configuration in the future. But I think the above represents the minimal.

Now, when it comes to generating the lookup tables, WAGI could take a few strategies:

• It could generate the full path lookup table every time that it loaded. This is obviously heavy in cost.
• It could call _routes() only when a prefix was matched. So if the client sends /hello/foo/bar/baz, WAGI would do a prefix match on its known routes, find that a module matched the prefix, and then invoke the _routes() call on the module to find out if /foo/bar/baz matched one of the subpaths

Of course, neither of these approaches preclude caching.

Right now, we pass any extraneous HTTP headers set by the module. Are there headers that we should block for security or performance reasons?

Right now, we only remove the following HTTP headers from the ones that are injected into the environment:

• HTTP_AUTHORIZATION
• HTTP_CONNECTION

The specification notes that other security-sensitive headers should also be removed. What headers should be removed?

The static fileserver demo (wagi-fileserver) is returning 500 errors for all paths under static. This appears to be a recent issue e.g. it works on commit da17547 (PR 96).

It constantly surprises me and I'm not sure whether it's intentional or an oversight (or, and perhaps this is more likely, I'm doing something wrong).

If your Bindle server is not localhost:8080, you currently have to specify that on every module as bindle_server. It would be good if WAGI supported the usual environment variable, or some other option.

Currently, all of the logs for all of the things go to STDOUT. Probably, we want logs per module to be split out into their own streams

Right now, if you want to pass environment variables to the guest module, you do so on the CLI like this:

$ wagi -e FOO=1 -e BAR=2

We could support a Docker-style (or Heroku style) .env file that had a simple format like this:

FOO=1
BAR=2

I think this could be done simply with an existing Rust crate.

I think the default behavior would be to try to load a $(PWD)/.env file, but with a CLI flag to override. Though if we are reluctant to load by default, we could just add an --env-file=$PATH option.

We could possibly add a CRUD API instead of just using modules.toml.

Modules built from swift-wasm do not execute. The error is: Err(Exited with i32 exit status 71)

Example Hello World code:

print("content-type: text/plain\n\n");
print("Hello WAGI!\n");    

Right now, the 500 errors are completely uninformative and we print the error to the system's STDOUT. We should log the error and give a way to correlate the log message to the error response.

Since we don't have networking, our best way to share information between two modules would be to share virtual files across multiple modules. That might require a new top-level object in the Wagi config.

Nice project.

Wondering about "In the future, as WASI matures, we will relax the restrictions on outbound networking."

Could you explain what this implies? Thxs.

This depends on a new release based on deislabs/wasi-experimental-http#71.

I am pointing Wagi to a remote Bindle, and I expected that after the first pull, the module to be cached. However, it seems that Wagi is always pulling from the remote.

(the issue exists regardless of whether the Wasmtime cache is used or not)

Below is Wagi responding to two identical consecutive requests:

$ RUST_LOG=wagi=debug wagi --bindle-server https://bindle.deislabs.io/v1 --bindle hippos.rocks/helloworld/2.0.0  --cache examples/cache.toml --listen 0.0.0.0:3000
Jun 30 18:37:01.665  INFO wagi: Starting server addr=0.0.0.0:3000
No log_dir specified, using temporary directory /tmp/.tmpyYh1Nq for logs
Jun 30 18:37:01.666  INFO wagi: Loading bindle name="hippos.rocks/helloworld/2.0.0"
Jun 30 18:37:01.923 DEBUG wagi::runtime::bindle: Loaded modules from the default group (parcels that do not have conditions.memberOf set) default_modules=1
Jun 30 18:37:01.945 DEBUG route_instantiation:load_cached_module{cache=/tmp/.tmpZ387eR module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}: wagi::runtime: module cache miss. Loading module from remote.

Jun 30 18:37:14.125 DEBUG run_wasm{entrypoint="_start" use_tls=false uri=/ module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}:module instantiation: wagi::runtime: Overriding port port="3000"
Jun 30 18:37:14.125 DEBUG run_wasm{entrypoint="_start" use_tls=false uri=/ module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}:module instantiation: wagi::runtime: Overriding port port="3000"
Jun 30 18:37:14.125  INFO run_wasm{entrypoint="_start" use_tls=false uri=/ module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}:module instantiation: wagi::runtime: Using log dir log_dir=/tmp/.tmpyYh1Nq/8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1
Jun 30 18:37:14.128 DEBUG run_wasm{entrypoint="_start" use_tls=false uri=/ module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}:module instantiation:load_cached_module{cache=/tmp/.tmpZ387eR module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}: wagi::runtime: module cache miss. Loading module from remote.


Jun 30 18:37:18.399 DEBUG run_wasm{entrypoint="_start" use_tls=false uri=/ module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}:module instantiation: wagi::runtime: Overriding port port="3000"
Jun 30 18:37:18.399 DEBUG run_wasm{entrypoint="_start" use_tls=false uri=/ module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}:module instantiation: wagi::runtime: Overriding port port="3000"
Jun 30 18:37:18.399  INFO run_wasm{entrypoint="_start" use_tls=false uri=/ module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}:module instantiation: wagi::runtime: Using log dir log_dir=/tmp/.tmpyYh1Nq/8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1
Jun 30 18:37:18.403 DEBUG run_wasm{entrypoint="_start" use_tls=false uri=/ module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}:module instantiation:load_cached_module{cache=/tmp/.tmpZ387eR module=parcel:hippos.rocks/helloworld/2.0.0#6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26}: wagi::runtime: module cache miss. Loading module from remote.

Edit: it seems to keep re-writing the same file in the module cache:

$ ls -al bindle-cache/
total 1732
drwxr-xr-x 2 pi pi    4096 Jun 30 18:45 .
drwxr-xr-x 4 pi pi    4096 Jun 30 18:44 ..
-rw-r--r-- 1 pi pi 1762539 Jun 30 18:46 6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26

$ sha256sum bindle-cache/6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26
6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26  bindle-cache/6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26

# wait until next request, see the file has been overwritten:

$ ls -al bindle-cache/
total 1732
drwxr-xr-x 2 pi pi    4096 Jun 30 18:45 .
drwxr-xr-x 4 pi pi    4096 Jun 30 18:44 ..
-rw-r--r-- 1 pi pi 1762539 Jun 30 18:47 6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26
$ sha256sum bindle-cache/6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26
6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26  bindle-cache/6f88f066fd7fdde440ff3bbc5b7c0e3ef9d2fc0c7b7a98122b5c0f6fc75bed26

We need to switch over to the Apache license before we can look at donating Wagi to a foundation. It's probably better to do this earlier rather than later.

This will bring WAGI in line with our other projects and allow us to have some good telemetry if needed for integration with other systems

We recently added support for building and running Wagi on Windows (#60) -- but because of the platform-dependent Makefile, you cannot make run on Windows.

We should either add a separate target for Windows, or move the logic to create the directories and certificates in a build.rs script.

Note that setting the log environment variables still needs to be in the Makefile, and that is going to have to be platform dependent.

Now value f SCRIPT_NAME does not start with '/', such as 'wagi-deploy/rust-wasi.wasm' , but from https://datatracker.ietf.org/doc/html/rfc3875#section-4.1.13 the SCRIPT_NAME should start with '/' if not NULL

The syntax is the same as for PATH_INFO (section 4.1.5)

      SCRIPT_NAME = "" | ( "/" path )

   The leading "/" is not part of the path.  It is optional if the path
   is NULL; however, the variable MUST still be set in that case.

Wagi should be able to serve TLS/HTTPS requests.

• It should be configurable from the CLI
• It should also set SERVER_PROTOCOL to https for guest modules

Right now, the module's STDERR is sent directly to the process's STDERR. Probably we should indirect that and other messages to a log that can be independently configured.

Starting Wagi multiple times on Windows (not checked linux or macos) can result in the following output and wagi not listening on the port specified in -l arg

[2021-07-06T14:15:31Z INFO wagi] Setting default host to localhost:3000
thread 'main' panicked at 'error binding to 127.0.0.1:32768: error creating server listener: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)', C:\Users\sdavies.cargo\registry\src\github.com-1ecc6299db9ec823\hyper-0.14.2\src\server\server.rs:65:13

Notes:

Wagi does not terminate when this panic occurs
The port that it complains about does not seem to be in use (but has been previously)
Seems like after some time (TIME_WAIT?) port becomes available.

We should be able to pull oci://example.com/foo/bar:latest into Wagi.

Right now, WAGI loads the WASM module at request time. Big WASM modules can take seconds to load.

There is probably a way to cache or preload modules to speed this up. wasm3 interprets rather than JIT-ing, and that might be substantially faster in this case (though I don't think wasm3 currently has WASI support).

Currently, the content type header in docs and examples is Content-Type: text/plain.
We should update it to include the encoding as well - Content-Type: text/html; charset=UTF-8

wagi --env is good for global env vars for all modules, but for some cases, I need to inject some secret vars for a Wasm module only, not for all modules. I found the example from https://github.com/deislabs/wagi-examples/tree/main/env_wagi

[[module]]
# Mount this to example.com/env
route = "/env"
# Point to the directory that 'cargo build' wrote the WASM file
module = "/path/to/env_wagi/target/wasm32-wasi/release/env_wagi.wasm"
# Mount the source code of this repo
volumes = { "/" = "/path/to/env_wagi" }
# Set an environment variable
environment.TEST_NAME = "test value"

But it does not work on 0.3.0, any clue?

pub struct Module {
    /// The route, begining with a leading slash.
    ///
    /// The suffix "/..." means "this route and all sub-paths". For example, the route
    /// "/foo/..." will match "/foo" as well as "/foo/bar" and "/foo/bar/baz"
    pub route: String,
    /// The path to the module that will be loaded.
    ///
    /// This should be an absolute path. It must point to a WASM+WASI 32-bit program
    /// with the read bit set.
    pub module: String,
    /// Directories on the local filesystem that can be opened by this module
    /// The key (left value) is the name of the directory INSIDE the WASM. The value is
    /// the location OUTSIDE the WASM. Two inside locations can map to the same outside
    /// location.
    pub volumes: Option<HashMap<String, String>>,
    /// The name of the function that is the entrypoint for executing the module.
    /// The default is `_start`.
    pub entrypoint: Option<String>,
    /// The URL fragment for the bindle server.
    ///
    /// If none is supplied, then http://localhost:8080/v1 is used
    pub bindle_server: Option<String>,

    /// List of hosts that the guest module is allowed to make HTTP requests to.
    /// If none or an empty vector is supplied, the guest module cannot send
    /// requests to any server.
    pub allowed_hosts: Option<Vec<String>>,

    /// Max http concurrency that the guest module configures for the HTTP
    /// client. If none, the guest module uses the default concurrency provided
    /// by the WASM HTTP client module.
    pub http_max_concurrency: Option<u32>,
}

I keep getting Error: Not all routes could be built: Invalid id: InvalidId with bindle: modules.

It seems like this is caused by

I believe that changing bindle_cache_key(uri) to bindle_name fixes it.

Today Brendan suggested a feature that I think would be really cool:

He suggested "dyanmic overrides packaged as part of the request." He's thinking along the lines of Launch Darkly or A/B testing frameworks.

In the simplest case, you could merely select which module to run based on host headers. But Brendan suggested taking it even further and being able to specify a module in an HTTP header and have that loaded in the request (think like a Bindle reference where the runtime could verify the integrity of the package before loading).

Probably, we can start with loading conditionally off of a Bindle + HTTP header or A/B flag cookie. Then we could work out this second part.

The goal of this feature is to take some files from a bindle and serve them via a Wasm module in the bindle. The target should be a wasm module that handles static file serving in a predictable and configurable way.

General overview:

• One bindle can have more than one Wasm module
• One module may serve out static assets
  • Ideally, we would be able to write a pretty good static file serving module and re-use it all over the place
• Assets can be clearly identified as static assets that should be served at an understood path.

A Bindle with Static Assets and Multiple Modules

Here is an example invoice.toml with two modules and some files that should be served statically, along with another file that should never be served statically. I have omitted a bunch of boilerplate for brevity.

bindleVersion = "1.0.0"

# The top-level data
[bindle]
name = "example.com/static"
version = "1.0.0"

[[group]]
name = "static-files"
satisfiedBy = "allOf"

[[group]]
name = "private-files"
satisfiedBy = "allOf"

# An example of a custom app
[[parcel]]
label.name = "myapp.wasm"
label.mediaType = "application/wasm"
conditions.requires = ["private-files"]
features.wagi.path = "/app"   # From the hippofacts

# An example module that just serves static files
[[parcel]]
label.name = "static.wasm"
label.mediaType = "application/wasm"
conditions.requires = ["static-files"]
features.wagi.path = "/static/..."  # Note the wildcard /..., which matches all subpaths

# These three parcels are static files to be served by static.wasm
[[parcel]]
label.name = "img/penguin.jpg"
label.mediaType = "image/jpeg"
conditions.memberOf = ["static-files"]
features.wagi.file = true   # Generated from hippofacts, designates this as a member of the `files = []` array in hippofacts

[[parcel]]
label.name = "img/cassowary.jpg"
label.mediaType = "image/jpeg"
conditions.memberOf = ["static-files"]
features.wagi.file = true

[[parcel]]
label.name = "css/style.css"
label.mediaType = "text/css"
conditions.memberOf = ["static-files"]
features.wagi.file = true

# This asset is private, is mounted to myapp.wasm, and can't be served via static.wasm
[[parcel]]
label.name = "super/secret/data.txt"
label.mediaType = "text/plain"
conditions.memberOf = ["private-files"]
features.wagi.file = true

(See comments on deislabs/hippo-cli#15 for how this would be generated from hippofacts)

Serving the Files

The bindle above is mounted to some URI paths:

• http://example.com/app/ is mapped to myapp.wasm
• http://example.com/static/ and all subpaths are sent to static.wasm

When WAGI encounters the features.wagi.file = true flag, the parcel is considered a file and is mounted as a file, attached to any modules that require this parcel.

In the bindle above, the mayapp.wasm module would have one file available to it, effectively mounted to the path /super/secret/data.txt. It would be able to access that data however it saw fit.

The static.wasm module has the following paths available, mounted by Wagi:

• /img/cassowary.jpg
• /img/penguin.jpg
• /css/style.css

The static.wasm module effectively can translate HTTP paths to static file paths. Here's basic pseudocode:

var path_info = env.PATH_INFO            // Example: /static/img/penguin.jpg
var prefix = env.X_MATCHED_ROUTE  // Example: /static

var file = path_info.trim_prefix(prefix)  // Gives us /img/penguin.jpg
var mime = guess_mime_by_extension(file) // gives us image/jpeg

// Obviously, we'd want to do something more memory-efficient
var data = fs.read_file(file)

// Print a content-type header, an empty line, then the data
stdout.println("Content-Type: " + mime)
stdout.println("")
stdout.print(data)

Right now, a module is entirely buffered, and then its contents are length-checked and sent back to the client. This is fine for small files, but would be both slow and resource intensive for larger responses. We can probably figure out a way to stream that content back.

This will break from the CGI spec, but that is okay since the spec was pre-HTTP/2

following modules.toml file in Readme or examples directory leads to de-serialization issue
suru@linux-vm:~/Documents/wagi$ target/debug/wagi
=> Starting server
Error: missing field modules at line 6 column 1

Now value of SERVER_PROTOCOL is http or https, but from https://datatracker.ietf.org/doc/html/rfc3875 the SERVER_PROTOCOL should be alike HTTP-Version = "HTTP" "/" 1*digit "." 1*digit, such as HTTP/1.1 .

The following option should be added to the Wagi CLI:

-e NAME=VALUE

This should be supported for multiple options (-e foo=bar baz=lurman). It should also support quoted values: -e foo="test -e".

This should parse the input into name/value string/string pairs, which will be passed to every running module as environment variables.

For example, if two a.wasm and b.wasm are both running, both should receive all of the name/value pairs sent using -e.

There is currently no support for setting the maximum number of concurrent requests a guest module can perform.
Since the HTTP library supports it, we should expose it as a field in the module config as well:

https://github.com/deislabs/wasi-experimental-http/blob/6a6226ead5ae2335ccde0a7e9a0b4a128d46249f/crates/wasi-experimental-http-wasmtime/src/lib.rs#L296-L298

Right now, the Router has some configuration info that gets passed to the Module in the form of long parameter lists on functions. We should probably encapsulate that information in some way.

Example: https://github.com/deislabs/wagi/blob/main/src/runtime/mod.rs#L137

I started playing around with some of the example repos that are available, specifically the AssemblyScript one, and was wondering if there was a way to pull data from the body of a request like the example below or would you need to send the data in via query parameters?

curl -d 'hello world' -H 'Content-Type: text/plain' http://localhost:3000/as-hello

It looks like the CONTENT_LENGTH is updated correctly, but I dont see an env var available for request body data.

The following headers are not completely implemented:

• AUTH_TYPE
• REMOTE_USER

For discussion, see deislabs/hippo-cli#25

It appears that Hyper does not always set this information on the URL object it sends. So sometimes defaults are inserted when it seems like actual values should be available.

The current OCI support added in #39 only allows pulling anonymously from OCI registries.
There should be a way to pass credentials to authenticate to a private registry.

According to the CGI spec, the PATH_TRANSLATED env var was a path to a file on the filesystem. We could use PATH_TRANSLATED to hold the path info with the prefix chopped off. E.g. if the route matches is /foo/... and the PATH_INFO is /foo/bar/baz, then the PATH_TRANSLATED would be /bar/baz.

Since this is a frequently needed transformation, it is probably a good idea to do it.

If we are worried about overloading a CGI term, we could use something like X_SUBPATH or something

Right now we buffer the entire HTTP payload before sending to the client. That is really bad performance for things like images. We should instead be able to send chunked data to the client.

It would be great to be able to do:

[[module]]
domain = "my.example.com"
path = "/"
module = "foo.wasm"

And be able to support host name matching.

Early on, we write Wagi to do multi-tenant hosting. So we had a number of features like virtual hosting that were designed to be strict and to reduce the possibility of accidentally loading the wrong site (information disclosure).

But our use case is rapidly evolving toward using a single Wagi instance per app. And in this case, it makes more sense to delegate the host mapping to Nomad or other services, and not have to explicitly configure a Wagi app. Rather, only Wagi itself would know about the host name, and that information would be supplied once for all endpoints hosted on that Wagi instance.

Should we remove virtual hosting, disallowing modules from declaring their own host name?

👍 = Yes
👎 = No

Wagi currently supports reloading the modules.toml file dynamically with /_reload. However, it has introduced a few problems:

• There is now race condition when writing modules files
• The aysnc code has gotten very complicated, with strange things happening at strange places
• This does not work well with Bindle and OCI

So it is probably good to just remove this.

I had a modules.toml that I assumed would not care about where it was served from - it specified routes and handlers but no hosting.

I ran my WAGI server on localhost:32768 and pointed it at this modules.toml. The routes failed to load, as far as I can tell because they did not match the default_host of localhost:3000.

Am I misunderstanding? Is this expected behaviour? It surprised the heck out of me.

On the plus side, it did give us much better tracing of the handler mapping logic.

The documentation points to the following examples:

• env_wagi: Dump the environment that WAGI sets up, including env vars and args.
• hello-wagi-as: AssemblyScript example using environment variables and query params.

Neither of which are publicly available.

The current implementation of Wagi only uses the first free wasm module from a bindle.

It should instead do the following:

• Support multiple modules
• Support mapping volumes to modules, where a volume is a parcel in a group required by the module
• Support mapping module configuration options (host, route, etc) from inside of the bindle using features

The first section in Wagi module output is a set of headers. But there are specific headers that are supported. We need to document all of the.

I believe they are:

• location
• content-type
• status

But there might be others.