reusing flow identifiers is racy about draft-ietf-masque-h3-datagram HOT 9 CLOSED

If your HTTP/3 code is closely coupled with your QUIC code, this can be solved by looking at QUIC packet numbers. If we take the CONNECT-UDP example, receiving a RESET_STREAM frame for the CONNECT-UDP stream ensures that the peer will not send subsequent DATAGRAM frames with this flow ID. If you record the packet number of the packet that carried the RESET_STREAM frame, you can then drop any DATAGRAM frames with that flow ID whose packet number is lower. That solves both problems you've described.

from draft-ietf-masque-h3-datagram.

If your HTTP/3 code is closely coupled with your QUIC code, this can be solved by looking at QUIC packet numbers.

It is not, my implementation separates the two layers.
In the design of QUIC transport, we went a long way to avoid looking at the packet number at all. If this design principle applied to the transport, it should even more so apply to applications built on top of QUIC.

If you record the packet number of the packet that carried the RESET_STREAM frame, you can then drop any DATAGRAM frames with that flow ID whose packet number is lower.

That only applies for the server side, doesn't it? The client still wouldn't know if a packet it receives was sent before or after the ID was reassigned.

from draft-ietf-masque-h3-datagram.

It is not, my implementation separates the two layers.

This feature is optional, so you could choose to not implement it. Or are you proposing a different mechanism that works?

In the design of QUIC transport, we went a long way to avoid looking at the packet number at all. If this design principle applied to the transport, it should even more so apply to applications built on top of QUIC.

I agree that we shouldn't violate this design principle in the mandatory parts of the spec, but doing so in optional features seems acceptable to me.

That only applies for the server side, doesn't it? The client still wouldn't know if a packet it receives was sent before or after the ID was reassigned.

This isn't really about client or server, but more about who sent the RESET_STREAM frame. The receiver of RESET_STREAM can key on the packet number that carried that frame, whereas its peer can key on the packet number of the first acknowledgement for that RESET_STREAM.

from draft-ietf-masque-h3-datagram.

This feature is optional, so you could choose to not implement it.

How so? How do I prevent the peer from reusing a flow identifier?

Or are you proposing a different mechanism that works?

Actually, I'm not quite sure I understand the issue that @martinthomson raised in #10. I see flow identifiers as analogues to stream IDs. For stream IDs, we never even considered reusing them, even though that wouldn't have required a layer violation in that case.
Why are the 3 additional bytes (I'm assuming you're using fewer than 1073741823/2 identifiers on a typical connection, otherwise it's 7) too expensive in one case, but not the other?

from draft-ietf-masque-h3-datagram.

How so? How do I prevent the peer from reusing a flow identifier?

Why would you want to prevent the peer from reusing a flow identifier? The peer ensures the security properties of reuse.

Actually, I'm not quite sure I understand the issue that @martinthomson raised in #10. I see flow identifiers as analogues to stream IDs. For stream IDs, we never even considered reusing them, even though that wouldn't have required a layer violation in that case.
Why are the 3 additional bytes (I'm assuming you're using fewer than 1073741823/2 identifiers on a typical connection, otherwise it's 7) too expensive in one case, but not the other?

Because the payload of DATAGRAM frames needs to fit in a single QUIC packet, whereas STREAM frames allows splitting the payload across packets. When you're tunneling, this means that the overhead impacts your MTU when using DATAGRAMs.

from draft-ietf-masque-h3-datagram.

I'd like to raise the attention towards the idea that reliably supporting reuse of flow identifiers can become quite complicated when in large-scale installations.

Consider a large scale point of presence, where a large number of servers are sharing the same public IPs:

                                                   ^            
                                                   |            
                                                   |    Higher  
                                                   |  protocols 
+--------------------------------------------+     |            
|            Application cluster             |     |            
+--------------------------------------------+     |            
                                                   |            
+-----------+                    +-----------+     |            
|+---------+|                    |+---------+|     |            
||   H3    ||                    ||   H3    ||     |            
|+---------+|                    |+---------+|     |            
|+---------+|  x 100             |+---------+|     |            
||  QUIC   ||                    ||  QUIC   ||     |            
|+---------+|                    |+---------+|     |            
|+---------+|      GRE-style     |+---------+|     |            
||   LB    |<------forwarding---->|   LB    ||     |            
|+---------+|                    |+---------+|     |            
+-----------+                    +-----------+     |            
                                                   |            
+--------------------------------------------+     |    Lower   
|          Advertised IP addresses           |     |  protocols 
+--------------------------------------------+     |            
                                                   v 
                                                   

In such scenario, multiple datagrams that share the same flow identifier may be served by several distinct physical servers. In order for these servers to be able to route the datagrams in a consistent way, the flow id allocation service must be able to provide some sort of consistency guarantees: when the flow identifier expires, any server that may serve the flow needs to agree that the flow has indeed expired. In other words, there is a consensus problem lurking behind the scenes.

While there are several ways to achieve such consensus, all share some propagation delay. Clients don't have a good way to estimating such delay (except for the observation that it's unlikely to be significantly shorter than the retransmit timeout).

It may be worth at least mentioning that reusing the flow identifiers rapidly is not recommended.

(edit: changed the wording that has mistakenly projected an argument as a known fact).

from draft-ietf-masque-h3-datagram.

Hi @oesh , this is an interesting scenario but I'm not sure I understand it: are you saying that multiple servers can send the same datagram flow ID on the same connection? How does that work? Given that QUIC uses incremental packet numbers as the encryption nonce, you're going to need a synchronization point in the server-to-client datapath anyway, or am I misunderstanding something?

from draft-ietf-masque-h3-datagram.

Hi @DavidSchinazi, I was probably unclear. It is also possible that I am missing something. Let me try to clarify my comment.

EDIT: I have reformatted / rephrased and clarified the comment. Again, I may be missing something, and will be glad to learn more in this case.

The flow allocation service SHOULD also provide a mechanism for applications to indicate 
they have completed their usage of a flow identifier and will no longer be using that flow identifier,
this process is called "retiring" a flow identifier. 

I think the main point of my comment - if the flow id is known to the applications, then it would be hard to communicate the retirement of the flow ID. In particular, in some situations it would be impossible for an endpoint to infer that a particular flow id has been successfully "retired" by the peer endpoint, because the application can retain the state associated with that flow id.

from draft-ietf-masque-h3-datagram.

Maybe the draft isn't clear: the idea is that the application is responsible for knowing that the flow ID is no longer in use, and ensuring that the application on the peer also knows that it is no longer in use, before retiring it - the application tells the flow allocation service to retire, the flow allocation service doesn't do it on its own.

from draft-ietf-masque-h3-datagram.