Future of HTTP High Level API Design about web-server-frameworks HOT 19 OPEN

Another topic for the agenda that can easily live inside an issue is a wish list of features we would like to have based on our experience as maintainers of webframeworks. For example a nice onFinished API, simple observability, and so on.

The question I have is, do we want to use the regularly scheduled meeting for this? Thursday, August 6⋅6:00 – 7:00am PST?

The regular meeting is fine for me, with a slight preference for the 5-6pm UTC time.

from web-server-frameworks.

@wesleytodd async (req) => { status, headers, body } is growing on me. It’s similar to undici.

from web-server-frameworks.

PS: @ronag is there a good way to ping you (without co-opting an unrelated thread)? I just returned comments on the gist and I don't think it does notifications so I didn't @ mention.

It does notifications 😄. You can also reach me by mail (it's on my GH profile).

from web-server-frameworks.

cc @nodejs/web-server-frameworks ☝️

from web-server-frameworks.

Hey, so last night I was just messing around reading docs and thinking and I made a gist. I posted on twitter and @bengl responded with a link to a different take.

I think we need to discuss the above issues, but one thing I really liked about the gist format is that it give concrete examples for us to discuss. Do folks like this idea and think it would be good for us all to take small ideas to share with the group? I feel like this would be a decent format for discussion and brainstorming. Anyway, feel free to post ideas you have here as I think the more ideas we have to pull from the better.

from web-server-frameworks.

PS: @ronag is there a good way to ping you (without co-opting an unrelated thread)? I just returned comments on the gist and I don't think it does notifications so I didn't @ mention. There is also the question of setting up some time with James to go over some of the lower level apis, but I was not sure how to reach out to you.

from web-server-frameworks.

Not that this is by any means a valid sample of the community, but I did an informal twitter poll which has interesting results and enough answers to make it worth considering. https://twitter.com/wesleytodd/status/1303048777687855105?s=20

from web-server-frameworks.

Another gist to reference: https://gist.github.com/Rich-Harris/4c061058176bb7f914d229d5c2a5d8ce

from web-server-frameworks.

Sorry, I missed that meeting. Can anyone fill me on why a middleware pattern should be part of the HTTP high-level API design and not the responsibility of the frameworks?

from web-server-frameworks.

The gist is really just me playing with what all layers might look like. I for sure do not intend to imply that a middleware pattern should land in core. Sorry if that is how the above gist's came across. The way I often work is to see if things make sense together in a wholistic way, so my framework.js file is exploring what the top level framework api would look like, and http-next would be a middle tier api. Does that help clear it up?

from web-server-frameworks.

yup 🙂

from web-server-frameworks.

Big thumbs up!! Someone in the twitter thread said this, and I like it:

a server handler is a mapper. It takes an input (the request) and produces an output.

Obviously there are details we would have to work out (like what can body be, same for headers), but I think on the whole this is the cleanest api I have seen.

One of the driving reasons Express has been so popular for so long is it's design philosophy is to stay as close to the core api's as it can while still adding value. I think that the above api would enable this same design philosophy for express or its inheritor to implement the "return a response" api and still remain close to the "core" approach.

The frameworks can layer on top (while still following the spirit of the api) things like () => 404 to generate the rest of the response based on the status code. Another nice framework api for the middleware piece would be that any middleware which does not return a response like object does not send a response, instead continues the chain.

from web-server-frameworks.

Great discussion! (And greetings from https://loopback.io land 👋🏻 😄 )

I'd like to point few important benefits of the proposed syntax async (req) => { status, headers, body }.

When writing Express-style middleware, there are few things that are difficult to accomplish:

• Execute code after the route handler has finished, e.g. to print a log entry containing the HTTP response code and how long it took to handle the request - see the code in morgan for an example. Packages like on-headers and on-finished abstract the low-level complexity away, but the middleware author must be aware of them.
• Transform the response body written by the route handler, e.g. to apply gzip compression - see the code in compression for an example, it's effectively replacing writable-stream-related methods on the res object.

The proposed syntax makes these two tasks much easier to implement:

function addLogging(originalHandler) {
  return async function(req) {
    const start = Date.now();
    const result = await originalHandler(req);
    console.log('%s %sms', result.status, Date.now() - start);
    return result;
  }
}

function addCompression(originalHandler) {
  return async function(req) {
    const result = await originalHandler(req);
    // decide if we want to apply compression
    // modify result.headers and result.body as needed
    return result;
  }
}

I have a concern about the property name status though. I find it a bit ambiguous - is it referring to status code (200), status message (OK) or an object holding both ({code: 200, message: 'OK'})? Personally, I prefer to use statusCode for clarity.

from web-server-frameworks.

How do we handle trailers + push?

from web-server-frameworks.

Packages like on-headers and on-finished abstract the low-level complexity away, but the middleware author must be aware of them.

They also rely on monkeypatching core, which is a significant issue for core maintainability.

from web-server-frameworks.

@wesleytodd

This would be very elegant and probably sufficient for 90% of use cases:

middleware(async ({ 
  headers: Object|Array, 
  body: AsyncIterator,
  trailers: Promise,
  push: PushFactory
}) => ({ 
  headers: AsyncIterator|Object|Array,
  body: AsyncIterator,
  trailers: Promise
}))

headers with AsyncIterator would allow the case for informational headers.

No special classes required.

Missing the following features:

• Trailers
• Informational headers
• Push

Would it be a viable option for frameworks such as express to not support some or all of the above missing features?

Trailers could be added like so (although I don't like it):

middleware(async ({ 
  headers: Object, 
  body: Readable, 
  trailers: Promise<Object> 
}) => ({ 
  headers: Object, 
  body: Readable,
  trailers: Promise<Object> 
}))

Informational headers:

middleware(({ 
  headers: Readable<Object>, 
  body: Readable<Buffer>, 
  trailers: Readable<Object> 
}) => ({ 
  headers: Readable<Object>, 
  body: Readable<Buffer>, 
  trailers: Readable<Object>
}))

Which could be unified into:

middleware(Readable<Headers|Readable|Trailers> => Readable<Headers|Readable|Trailers>)

from web-server-frameworks.

Just thinking out loud:

middleware (async function * (src) {
  const headers = await src.next()

  yield headers

  const body = await src.next()

  yield body

  const trailers = await src.next()

  yield trailers
})

middleware (async function * ({ headers, body, trailers }) {
  for await (const h of headers) {
    yield h
  }

  yield info

  yield info

  for await (const b of body) {
    yield b
  }

  for await (const t of trailers) {
    yield t
  }
})

from web-server-frameworks.

So to expand out on the proposal above and give it some "real" (but very oversimplified) code for how you might write a minimal express like (but with response's returned) "framework" on top:

https://gist.github.com/wesleytodd/e5642c0d39fa71bdebf8ef31ddbd5e40#file-simple-framework-src-js

So the underlying protocol things would be hidden away, and for the body, we would totally avoid streams at the higher levels (unless the user decided to pass a stream of their own making as body when creating the response).

from web-server-frameworks.

Also, to address the generator function approach (which I think would make a great framework api, but not a great lower level), I think the layer we have laid out would enable building that api on top of it really simply. I don't think this decision would block any of the major frameworks from adopting it under the hood.

from web-server-frameworks.