Exposing datagram acknowledgements about datagram HOT 38 CLOSED

I may be wrong, but this seems very implementation specific, but IMO flow control is unnecessary so long as datagrams are indicated to the application layer inline (no buffering).

For instance, MsQuic only acknowledges a packet with a datagram frame if it's able to successfully process it (datagram frame processing here). Additionally, "processing" a datagram frame includes indicating it up (inline indication here) to the application layer. There is absolutely no buffering at the datagram frame layer in MsQuic; so flow control is not a really issue.

It's possible the application layer then does some buffering and would require flow control, but that's completely up to the application layer. If there is a scenario where the application layer may drop data (due to buffering constraints or something else) it should be up to the application layer to decide if the peer needs to know that information; and if so, inform them appropriately at their layer.

from datagram.

I agree with @nibanks's earlier comment that this sounds very implementation-specific. Our implementation also would not drop any datagram frames that have been received and ack'ed, since we process the data immediately and send it inline.

I'd almost prefer to just have a SHOULD NOT drop datagram frames if they've been received and ack'ed. This will certainly influence API design, and means that you'll have a rougher time if you just try to imitate a UDP socket buffer.

from datagram.

+1 to #15 (comment) . There is a process hop between the QUIC code and the JavaScript code, and the QUIC code needs to send ACKs quickly (especially if that packet contains a STREAM frame next to a DATAGRAM frame) so it's simply not possible to wait on JavaScript before sending the ACK. You could buffer datagrams indefinitely after ACking them to ensure that the application gets them, but that goes against the entire philosophy of DATAGRAMs by increasing latency unboundedly. "SHOULD NOT drop DATAGRAM frames" sounds completely unrealistic to me.

from datagram.

There are two questions here:

• who owns the buffers -- QUIC, or the application; and
• what are the application's reliability requirements.

I would argue that for maximizing generality of a DATAGRAM receiver implementation -- so that multiple application behaviors are possible -- you want the application to own these buffers. Whether these buffers are implemented in QUIC or at the application or somewhere in between doesn't matter; it is meant for the application.

For an application that wants some notion of reliability, there are two choices in how a receiver manages its buffers:

• it signals delivery out of the buffers and/or loss (ACKs/NACKs/gaps);
• it signals buffer size explicitly to avoid any loss (flow control)

An app that sits on top of QUIC DATAGRAMs thus has two choices:

• use explicit Application ACKs to signal receipt (and gaps or NACKs to signal loss), since the sender might have caused the application buffer to overflow;
• use explicit flow control to disallow application buffer overflowing. Doing this has the nice property that QUIC ACKs can then be treated as application ACKs, since application flow control will ensure that this data does not get dropped at the receiver between QUIC and the app.

In either case, it's a decision that is both application- and implementation-dependent. An app that cannot know the buffer size will have to rely on using explicit application ACKs. An app that is ok with data loss can allow loss to happen, and may or may not signal this back. An app that wants to avoid buffer overflow at the application buffers can reuse QUIC ACKs as application ACKs but will need to ensure that application flow control is in place.

Implementations ought to expose DATAGRAM QUIC ACKs to the application, so that applications have a choice of using that signal. However, the application is ultimately responsible for using them correctly.

from datagram.

Adding text to clarify that receipt of an ACK for a packet containing a DATAGRAM frame does not mean that the application got the frame sounds like the right solution here.

from datagram.

I suggest that we remove the recommendation that datagram acknowledgements can be exposed to the application, and possibly even prohibit it.

from datagram.

Alternatively, this problem could be avoided if there was explicit flow control for datagrams. This would avoid a full datagram buffer and may have other benefits.

from datagram.

Adding flow control to DATAGRAM seems like a problem for use cases that don't want that, like tunneling of UDP-based transports that provide their own flow control

from datagram.

Adding flow control to DATAGRAM seems like a problem for use cases that don't want that

Surely that could be addressed by specifying arbitrarily large or infinite flow control limits? Would still complicate the spec, which seems unfortunate.

IMO flow control is unnecessary so long as datagrams are indicated to the application layer inline

That's a really cool solution. It's not a small constraint on the receiver's implementation, but it's certainly a case where we forbidding application-visible ACKs would be a harmful restriction.

from datagram.

Inline datagram processing is a cool idea and removes the need for flow control.

Unfortunately, it's the sender that exposes acknowledgements to the application, but it's the receiver that would need to process and acknowledge datagrams inline. The sender would not be able to assume the receiver uses a particular implementation.

from datagram.

The capability to do inline indications on the receive side could be negotiated with the transport parameter. "Hey, I receive datagrams, and I don't buffer them." Then the sender side can decide how they want to handle ACKs based on that state. Though, it's not great that they still have to handle the scenario of "I do buffer, and I might throw them away" so long as any peer they communicate with might buffer.

Personally, I'd just like to say that we require guaranteed app delivery for acknowledged QUIC packets with datagram frames, and then let QUIC implementations sort it out, but I realize that's not easily done for most implementations without significant refactoring. But if we don't do this, I do see it as a major bummer that buffering on the receiver side could effectively make QUIC layer acknowledgements worthless to the app (which IMO was the best advantage to this solution).

from datagram.

My only other thought would be to have a new frame (e.g. DATAGRAM_DROPPED) that a receiver sends if it drops DATAGRAM frame; but to accurately identify which that were dropped, each datagram would need a unique identifier and that further complicates all this. So I definitely wouldn't prefer this solution.

from datagram.

"Hey, I receive datagrams, and I don't buffer them."

Or, in more explicit terms, "My ACKs guarantee delivery of datagram frames to the application."

it's not great that they still have to handle the scenario of "I do buffer, and I might throw them away"

Is this any different from handling the scenario of a peer that doesn't support datagram frames at all?

from datagram.

Is this any different from handling the scenario of a peer that doesn't support datagram frames at all?

Yes, there is a difference. If the peer doesn't support datagram frames at all, the protocol might not work at all, and the app layer will just reject the connection outright. If the app supports datagram frames, but it's implementation may or may not buffer. In that case, depending on the app protocol, it may or may not then need additional app layer acknowledgements or accounting.

For instance, perhaps some game protocol that has a fire-and-forget "current player state" datagram payload could care less about acknowledgements, so it doesn't care if buffering drops a frame. All it needs is datagram support.

from datagram.

I'd almost prefer to just have a SHOULD NOT drop datagram frames if they've been received and ack'ed. This will certainly influence API design, and means that you'll have a rougher time if you just try to imitate a UDP socket buffer.

I would also prefer this approach.

from datagram.

For an implementation that doesn't expose the contents of QUIC packets to an application, exposing the ACKs has little value. The flow control issue is then a little moot. I would push back on any strict requirements as mentioned, it presupposes that implementations will provide a single style of processing or API.

SHOULD NOT seems acceptable because I would drop them based on some criteria and the text probably encourages me to document this aspect.

from datagram.

In the context of WebTransport, datagrams are processed at the Javascript level. Browsers are understandably not going to execute and block on inline Javascript callbacks for every received datagram.

from datagram.

If the app supports datagram frames, but it's implementation may or may not buffer. In that case, depending on the app protocol, it may or may not then need additional app layer acknowledgements or accounting.

Why can't the app layer reject the connection outright if the peer's transport parameters indicate that it does not support reliable acknowledgement for datagrams, just as it would if they indicate no support for datagrams at all? An app which doesn't care about ACKs obviously wouldn't do so, but those that do, can. Or they could apply app-layer mitigations, but that's more options, not fewer.

from datagram.

For an implementation that doesn't expose the contents of QUIC packets to an application, exposing the ACKs has little value.

@LPardue You don't need to expose how a datagram frame buffer from the app is put in a QUIC packet in order to give them information back about the receipt (or not) of an acknowledgement of their data. MsQuic allows the app to give a context pointer along with the buffer they send. We then indicate acknowledgement or suspected loss information back to the app for that context (event here).

Browsers are understandably not going to execute and block on inline Javascript callbacks for every received datagram.

@kixelated and @DavidSchinazi I totally agree that it's unreasonable to expect every layer above QUIC in the WebTransport scenario to do inline delivery, but that isn't the only scenario, and having that one scenario's requirement for buffering shouldn't make us throw our hands up and say no scenarios will be able to use a zero buffering approach. Protocols directly on top of QUIC should be able to do whatever they want.

The "SHOULD NOT drop DATAGRAM frames" purely at the QUIC layer is not unrealistic IMO. What the layers on top of QUIC do are totally up to them to define and implement. WebTransport datagrams processed at the Javascript layer will need application layer acknowledgements if the protocol requires info on exactly which datagrams were delivered.

from datagram.

Thanks Nick, that makes more sense. I do wonder though how useful it acks are if you're talking to my endpoint who might, in some conditions, drop the occasional DATAGRAM on the floor.

from datagram.

@nibanks I was thinking of this in terms of WebTransport since that was really top of mind. I agree that different applications have different needs, and some might be able to do this inline ACKing more than others. That said, I'm really not sure what the point of this SHOULD NOT is, it's like saying "an IP router SHOULD NOT drop IP packets" - most router implementors know that they shouldn't drop packets unless they have a good reason such as having nowhere to put the packet.

from datagram.

it's like saying "an IP router SHOULD NOT drop IP packets"

@DavidSchinazi I do agree somewhat with that sentiment, but it's a bit different because the transport is saying "Yes, I got the packet and was able to successfully process it.". If it's possible/allowable for the transport to then just drop it on the floor, is there really any reason to acknowledge datagram frames at all, beyond just another RTT estimate?

I just feel it would be a big shame that we have all this ACK machinery built into QUIC, but it gets wasted on datagrams. Not only that, it has to be rebuilt completely by the app to duplicate all the work QUIC already does.

from datagram.

@nibanks the way I see it, there are only two ways to perform queueing when you have one input and one output:

• option 1: you buffer things indefinitely forever
• option 2: you don't

I really don't see any other way. I think you're suggesting that transports should buffer DATAGRAMs infinitely if the application is busy, and that doesn't sound like a good idea to me.

from datagram.

@janaiyengar the way I read your summary of the issue (which is great, btw. Thanks!) it seems that not only do ACKs need to be exposed to the application, but a receiver can't drop acknowledged datagrams without informing the application, right? Otherwise, how would the app be able to reliably implement it's own flow control along with QUIC ACKs? A sender who receives an ACK of a datagram would assume that datagram is taking up buffer on the peer application; but if QUIC dropped it before it received that buffer, then both application sides would get out of sync in their view of the FC window. The sender could eventually stall out because of this, right?

from datagram.

"Instant processing" model is really nice, and makes sense if you're in the same process as the QUIC stack, but across a privilege separation boundary (web app/browser, userspace/kernel, etc), this is not feasible*, so I don't think this should be allowed at least by default.

* this is technically possible, and it is in fact how WebSocket API works; if your web app is too slow at processing incoming WebSocket messages, your server will just crash a browser tab remotely by overflowing the task queue, so most people agree this was a big mistake.

from datagram.

After reading through this thread I may have missed it, but has anyone considered simply having the spec say something along the lines of:

"If an ACK frame has been sent and then due to implementation specific buffering or resource constraints it is unable to process the corresponding DATAGRAM frame, the application MUST be notified of the locally dropped frames so that it has the option to support additional signaling if necessary."

This will ensure that only for any implementation where this case is possible, the application developer/operator will be made aware of such a possibility and can decide how best to handle it either through configuration or custom application logic.

from datagram.

What could a reasonable application do with that notification, aside from assert that it doesn't happen?

from datagram.

What could a reasonable application do with that notification, aside from assert that it doesn't happen?

Many things are possible with the notification, the ability to recognize when the dropped packets are not due to network congestion could be used to do custom throughput signaling back to the sender that incorporates other local signals such as queue depth or processing rate.

The underlying implementation might also expose dynamic controls on the cache or queue size that the application can adjust. It could also back off other memory or processor heavy operations in order to prioritize capacity to not drop future packets.

By ensuring that the ACK signaling only indicates congestion it isolates the responsibility of additional flow control signaling to just implementations that support it, only having to support notifying applications in such an event where frames are dropped locally.

from datagram.

Dropping datagrams ought to be similar to resetting a stream. It is pretty much the same semantic.

from datagram.

@huitema can you elaborate? Resetting a stream is a non-recoverable situation for a stream, whereas dropping datagrams is recoverable.

from datagram.

Resetting streams is defined as an application triggered message: application says it cannot proceed with the stream and provides an application defined code and reason. I think the application saying "I am dropping some of the datagrams that I received" has similar semantics.

from datagram.

I see. It would make me sad to turn a recoverable error into an unrecoverable one though, because some applications may not want that.

from datagram.

Yes, that's where the analogy with reset breaks. The datagram dropped message should be informative, not change the state of the connection or the datagram streams.

from datagram.

The difficulty I have with this concept is that I don't know what I'd expose to the application. My datagram implementation basically keeps a fixed-size queue of frames. So if I dropped some all I can tell an application is that something got dropped. Adding more information about what was dropped increases complexity.

The library already doesn't expose information about packets to the application, and so far things have worked out OK.

from datagram.

I tend to agree that there isn't much a QUIC stack that queues datagrams can do here. The best it could do it indicate how many it dropped, but that's likely not enough. If that data needs to be retransmitted then (at the app layer) they would likely need more detailed info (such as an app identifier for each dropped packet).

I think the best the draft can do here is mention that if the QUIC stack drops datagrams frames and also acknowledges them, then those acks cannot be leveraged by the application layer. It might be nice to have an additional TP that allows behavior this to be negotiated, but it's not strictly necessary.

from datagram.

" If that data needs to be retransmitted then (at the app layer) " ... then the app needs to run a transport on top of datagrams. Or use regular streams.

from datagram.

+1, if you need reliability then datagrams are not the droids you're looking for

from datagram.

The more I think on this, the more it just seems similar to the existing notion that we have with QUIC streams; transport delivery != application end-to-end delivery & processing. It's like email read-reciepts: just because I opened the email doesn't mean I read it, digested it and have a response.

The doc might benefit from doing a better job of explaining this notion and the interop concerns that might arise. For instance, stating that it is possible for receiving implementations to acknowledge DATAGRAM-bearing packets while providing no guarantee that the DATAGRAMS are presented to the application. Valid reasons are flow control or any other arbitrary reason the receiver decides. Application's that desire end-to-end loss-detection need to be aware of this possibility and build measures that accomodate it. As Jana points out, there are some signals an implementation MAY chose to provide to make that easier. If they don't, applications can vote with their feet.

from datagram.