Response format specification: f parameter and content negotiation about ogcapi-features HOT 40 CLOSED

Clemens as I had a discussion about this topic at today's WFS/FES SWG teleconference and this is what we concluded ... We need to cover two basic content negotiation use cases:

• client-to-server
• embedded links/handcrafted links for debugging/hypermedia

The first use case, client to server, is normatively covered in the HTTP specification via the Accept header.
For the second use case, the question arose about whether we even need to cover it normatively in the core at all. The thinking is that a server is free to define additional parameters and extension mappings/endpoints outside the core so an implementer is free to implement their preference in their server's OpenAPI description. Furthermore, for the hypermedia case anyway, the hrefs are typically opaque with the rel and type parameters providing the necessary context. So, Clemens and I thought that an informative treatment of this use case in the core would be sufficient. If a better consensus arises, we can amend the standard normatively to reflect that consensus.
We plan to disccus the topic further at the SWG teleconference this Thursday @ 10:00am EST and all are welcome to participate there. The connection details can be found here: https://www4.gotomeeting.com/join/906574757
Of course, comments and participation, are also welcome here ...

from ogcapi-features.

Just in case both the Accept header and Accept parameter are specified, the Accept parameter should be ignored.

The only practical issue with this is that the extremely common use of the ?Accept= parameter — messing around in a browser to experiment or get queries right — doesn't actually work. Because browsers will normally send an Accept: header. eg. my Chrome sends the following for this issue URL:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

This would mean the ?Accept= parameter is never used unless you're not in a browser — in which case you might as well set the Accept: header IMO.

from ogcapi-features.

Agreement on 2018-02-01: Reduce the core to refer to the HTTP standard for content negotiation. Add a discussion that to have links to alternate representations the server needs to provide representation-specific URIs for resources.

Examples for implementation mechanisms should go to the Guide (this thread provides input). Some brief informative content may also be needed in the standard (in notes).

from ogcapi-features.

Since passing different response formats is possible, I would expect an error message in that case. I'm afraid that otherwise, errors are harder to trace.

OK, just realized that the server would respond with HTTP code 406

from ogcapi-features.

The f-parameter should be treated like an override. This is useful in instances where you may want to override the default Accept-Header for a particular client. For example, you want to see the JSON representation in a browser which is expecting HTML.

from ogcapi-features.

Yes, it is also very useful for clickable links. However, using /buildings/abc.json instead of /buildings/abc?f=json has the same effect and is cleaner (no override necessary and no conflict with the HTTP spec, which does not have a mechanism to override the content negotiation rules).

from ogcapi-features.

👍

from ogcapi-features.

I have two issues with using extension as an override mechanism.
(1) You cannot convey version or profile information using extensions and for some formats such as GML that can be important. This forces the client to read the stream and then figure out what it got.
(2) You use a different value to request a particular format using an extension then you would using the Accept header (i.e. application/gml+xml; version=3.2 -vs- .gml). The benefit of the f override is that I can use the exact same value as I would in an accept header.
Accept: application/gml+xml; version=3.2 -vs- f=application/gml+xml; version=3.2.

from ogcapi-features.

The extension is not an override mechanism. The OpenAPI definition will state the media type that you get back, so there is no guessing what .gml returns.

So far, the idea was (also in the WFS 2.x REST API) to use f with short codes like json, basically the same values as used in an extension. I think, using media types as values for f would be error prone. Just as an example, f=application/gml+xml; version=3.2 would not work as the + would become a space character when decoded. And it makes the URLs less hackable, which I think is a benefit.

from ogcapi-features.

Ah ... ok, that helps for sure. As for F, I am not sure that was the intent in the WFS 2.X REST binding (see Table 38). With regard to encoding the value, I don't think that it is a big deal to expect code to property encode parameter values; they have to do that anyway for other tokens a URL that might include special characters ... no?

from ogcapi-features.

+1 to extensions over the 'f' parameter.

from ogcapi-features.

Ok, I have thought about this issue and this is my conclusion.

I think that the gold standard for format/content negotiation should be the HTTP "Accept" header. I believe everyone is in agreement with this.

However, in cases where we want to embed a link the Accept header will not work and so we are considering using the "f" query parameter or extensions like .json or .gml to request a specific format.

I don't like either of these. The problem with extensions is that they are ambiguous. What does ".gml" mean? Does it mean GML version 2, version 3, GML SF level 0, etc. I know that OpenAPI lets you map an extension to a specific mime type but even that is not satisfactory. First because the use of OpenAPI is optional and whatever else might be used may not have a mechanism to map extesnions in a similar way. Second, even with the extension mapping mechanism, you end up having to map multiple extensions to cover all the versions of GML (for example) that your WFS might offer (i.e. .gml2, .gml3, .gml3sf, etc.)

I don't like the use of the "f" parameter, although I like it better than extensions, because the semantics are different than the HTTP Accept header and that just complicates things.

So, I propose that we define a new parameter named "Accept" that behaves exactly like the HTTP Accept header. Just like the header, a client can use the parameter to specify a list of acceptable media types for the response (see: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html). I am not going to get into all of the processing details here but some stuff I remember off the top of my head from the HTTP specification and the Richardson RESTful API book are include: I exepect that the value of the Accept parameter would need to be URL-encoded but that is generally true for query parameters anyway. If no value is specified for the Accept parameter, then the client is declaring that any media type is an acceptable response and the server can generate whatever output format is its default. If an Accept value is specified and no suitable format can be found then the server should throw a 406 exception. Just in case both the Accept header and Accept parameter are specified, the Accept parameter should be ignored.

Finally, since someone implementing a WFS 3.0 will have to include code to handle the Accept header anyway, it makes sense to mostly resuse that code to handle an Accept parameter.

from ogcapi-features.

Having two ways of specifying a value required by the server makes it a DRY problem for the standard. I would pick one and go with it. FWIW you can usually specify headers in browser based fetches, I think.

from ogcapi-features.

As I said in my previous comment I think that the preferred content negotiation mechanism should be the Accept header. I think that the proposed "f" parameter or extensions or my proposed "Accept" parameter is to handle the case where you are using an embedded link (e.g. in a web page). Actually, I don't think that you can set the Accept header for a link in a web page ... but I am not 100% sure. However, Robert's comment is making me think that it should be the other way around ... that is that if the Accept header and the Accept parameter are both specified then the parameter should take precedence. Thoughts?

from ogcapi-features.

from ogcapi-features.

I think Robert is talking about using the browser window as a development tool for playing around with the api. While nice, I think the main use case of an api is for developers to provide applications, while the browser is an application that works in a certain way for users, and it expects to deal with certain formats, and gml is not among them. I would recommend to use a development tool to examine requests and responses when developing, and to use browsers for application delivery. (Caveat: devtools).

There's nothing that prevents an implementation from adding (optional) parameters of its choosing, right?

from ogcapi-features.

Actually, I don't think that you can set the Accept header for a link in a web page ... but I am not 100% sure

Correct.

While nice, I think the main use case of an api is for developers to provide applications, while the browser is an application that works in a certain way for users, and it expects to deal with certain formats, and gml is not among them. I would recommend to use a development tool to examine requests and responses when developing, and to use browsers for application delivery.

That's certainly a defensible approach to take. But if you choose that there's no real reason for an ?Accept= parameter at all. Personally virtually every developer I know uses browsers for web api exploration — whether it's a "proper" development tool or not. It's not that they can't make it work, it just reduces friction during the discovery/exploration/prototyping phase.

One alternative is to have HTML-based API explorers (fetch() & XmlHttpRequest can both send Accept: headers), but I don't think that should be part of a specification.

As @pvretano alludes to, without ?Accept= (or per-format URL suffixes like /{fid}.csv) you can't have a simple hyperlink to any specific-format data or query (eg. "Download the CSV data for the last 30 days"), because it would come in a server-selected format. And also precludes any HTML response pages from having links to alternative formats (eg. "get this as: JSON; CSV; KML; GeoPackage") without using additional Javascript. eg. see the existing spec wiki pages

from ogcapi-features.

As it has been pointed out, I see two main uses for a f/accept parameter:

• to be able to create clickable links that return the response in a certain format;
• to enable users, mainly developers, to hack the URLs in a web browser.

I understand @pvretano's concern about extensions (".json") being ambiguous, but on the other hand for me using media types as values is error prone and more difficult to edit in a browser address line. It is more error prone because it requires encoding/decoding and more than a string comparison as servers will need to normalise the media type strings after decoding before a string comparison.

Maybe an option could also be that we simply remove the parameter from the core, but add a permission that servers may add a server-specific parameter (f/accept) or paths (extensions) that may overrule the Accept header. A server-specific approach should be fine for both use cases as the clickable links are generated by the server anyhow and the human can easily determine the approach taken by the server.

from ogcapi-features.

Accept/Content-type > URL parameter > URL path.suffix, because you get further away from standards as you go from left to right, as far as message semantics goes.

If you are designing an API, URL parameters are well supported, even by HTML forms, links, browser URL bar, etc. Note that you can't prevent a browser from sending the Accept header, when you hack the URL with path suffix extension or parameter. Browsers are programmed from a position of hubris, in that they think they understand everything on the Web, and so they list everything that they understand in their Accept: header, expecting that they, or the script that is running, will know how to deal with what comes back. It's a fair assumption, I suppose. In any case, the server has to prioritize the query parameter if you want the url to be hackable via the API.

If you are designing a hypertext format for use in a/the Web, there practically is no other choice than message Accept and Content-Type metadata, which is where we're at in MapML design, ICYMI.

FWIW we implemented all of three these methods in the now-defunct GeoGratis API. I found the URL bar query parameters (e.g. alt=xml) to be most useful, I guess, as we never did implement a proper 'hypermedia' client (though we could have, I think).

from ogcapi-features.

@prushforth - I do not understand

URL parameter > URL path.suffix, because you get further away from standards as you go from left to right, as far as message semantics goes.

Why do you think that including the format information in the query part of the URI is closer to the standard than to include the format information in the path part of the URI?

I find that including it in the path has the advantage that you can clearly state in the OpenAPI definition that

• http://example.com/myapi/someresource may return a number of media types and the returned media type will be determined by content negotiation,
• http://example.com/myapi/someresource.html will always return text/html (or a 406 depending on the Accept header),
• etc.

I.e., this seems perfectly inline with the TAG finding.

On the other hand, if you use a query parameter, this is more obscure as the server has to ignore the priority listed in the Accept header, if the parameter is provided.

from ogcapi-features.

http://example.com/myapi/someresource may return a number of media types and the returned media type will be determined by content negotiation,
http://example.com/myapi/someresource.html will always return text/html (or a 406 depending on the Accept header),
etc.
I.e., this seems perfectly inline with the TAG finding.

I was thinking that URL parameters are supported by HTML forms, but I guess that is trivial.

I guess URL parameter === URL path.suffix is more correct, but neither is in line with the finding (content type metadata isn't looked for in the URL). Not that that matters - if you are designing an API you are explicitly disregarding some constraints of Web architecture. My point is mostly that the server has to prioritize the URL over the header, and you are specifying a custom content negotiation mechanism where server has to look at two places in the request and then decide which to honour. Instead of the standard saying that, maybe the standard should say it's ok if you (the server developer) want to do that.

from ogcapi-features.

@pvretano the ambiguity is a good point.

I think that servers should be allowed to implement either the f-parameter or
the extension parameter if the returned representation is unambiguous.

For the extensions, this could mean, that if there are multiple representations,
the service could provide multiple resources for each one. So there could be endpoints like

/buildings/{fid}_gmlsf0.json:
  get:
    responses:
      content:
        application/gml+xml;version=3.2;profile=http://www.opengis.net/def/profile/ogc/2.0/gml-sf0


/buildings/{fid}_gmlsf2.json:
  get:
    responses:
      content:
        application/gml+xml;version=3.2;profile=http://www.opengis.net/def/profile/ogc/2.0/gml-sf2


/buildings/{fid}.html:

A compromise would be, to combine a parameter with the endpoints to force a
specific encoding. In this case, I propose not to use the full media type, but
to reference the requirement class (see #9) somehow with an optional parameter, like:

/buildings/4711.xml?encodingClass=gmlsf0

Of course, that does not make sense for all combinations of req classes and
extensions, but this would prevent the need for passing full mime types in the URL.

from ogcapi-features.

My point is mostly that the server has to prioritize the URL over the header, and you are specifying a custom content negotiation mechanism where server has to look at two places in the request and then decide which to honour.

I don't think so. The approach with a URI per media type plus a "generic" one does not require a custom content negotiation mechanism and does not prioritise the URI over the header. See the example in the original comment.

from ogcapi-features.

I can see that the API will work ok, per the URL xor content-type approach. What happens if I specify .foo, where foo is not a known extension, does the server return a 400 or a 406?

from ogcapi-features.

Every extension must be listed in the definition. As there is no endpoint with the .foo extension, the server should respond with a 404 error.

from ogcapi-features.

What is returned depends on the API definition and the request.

If the API definition specifies that /path/resource.foo will always return the media type application/vnd.foo (if status = 200) then the server should return status 406, if the Accept header is, for example, just text/html. But the server should return the foo-representation and status 200, if the Accept header is missing or covers the foo media type.

from ogcapi-features.

Ah, thanks @jonherrmann. I missed the "not a known extension" part.

from ogcapi-features.

@cportele @jonherrmann I think 404 is an ok response, but I think the Accept header should be ignored when the URL .suffix mechanism is used.

from ogcapi-features.

from ogcapi-features.

@prushforth The ".json" is still part of the path and neither RFC 3986 nor RFC 7230 assigns any special meaning to a dot in the path (except the "." and ".." cases), so I do not see why a special rule is necessary / needed for URIs where the path ends in ".xxx".

That said, HTTP 1.1 allows that servers "return responses which are not acceptable according to the accept headers sent in the request." I.e., a server may still return JSON for a ".json"-URI, even if the Accept header asks for other media types only.

from ogcapi-features.

@cportele I think we're more or less agreeing now, so I'll stop there. Thanks for referring me to the op, which I may have forgotten.

from ogcapi-features.

👍 I agree @prushforth

from ogcapi-features.

A few observations ---

1. The accept header is typically used to inform a server of the data formats that the client software can accept in the response. It is part of the protocol and not usually an expression of the users' desired format.
2. File name extensions are used to associate a file with the default software package to be used with that file. It is a kludge to get around the lack of ELF support in MS-DOS. Not a firm foundation to build on.
3. Users may want the returned data to comply with a specific version of a specific profile of a specific standard. Neither the accept header nor file extension can support that.
4. The requested data may be returned out of band. In that case the user requested format may not be one of the options allowed in the accept header.
5. The request may not return a file (ex. streaming sensor data). So there may not be an appropriate file extension.
6. With the exception of the accept header (part of the HTTP protocol) this is too complex a topic to be addressed in the core.

from ogcapi-features.

6. With the exception of the accept header (part of the HTTP protocol) this is too complex a topic to be addressed in the core.

I don't agree here... every implementation (client and server) will need to deal with this, and if it's not addressed at all then IMO we're inviting incompatibility.

3. Users may want the returned data to comply with a specific version of a specific profile of a specific standard. Neither the accept header nor file extension can support that.

This is a pretty common use of the Accept header for REST APIs...? eg. Github

5. The request may not return a file (ex. streaming sensor data). So there may not be an appropriate file extension.

AFAIK Core doesn't deal with streaming, so shouldn't any streaming extension specs deal with it themselves?

from ogcapi-features.

From RFC 7231 "The Accept header field can be used by user agents to specify response media types that are acceptable."
Note that these are the media types acceptable to the user agent (browser). Not the media type desired by the user for this specific operation.
I agree that the accept header is commonly used for this purpose, and that it may be sufficient for the core. I don't agree that this approach can scale to more complex scenarios.
In a related concern, I'm always leery of allowing the user to overwrite parameters which are critical to the correct operation of the protocol.

from ogcapi-features.

from ogcapi-features.

Note that these are the media types acceptable to the user agent (browser). Not the media type desired by the user for this specific operation.

I do not read the HTTP spec to be that strict. For example, when the server does not make the decision and returns multiple options (reactive negotiation), the decision may be taken by the agent (any client, not just the browser) or the user:

Selection of alternatives might be performed automatically by the user agent or manually by the user selecting from a generated (possibly hypertext) menu.

This is not really different from the case where the user states his preferences before the request is sent.

from ogcapi-features.

3. Users may want the returned data to comply with a specific version of a specific profile of a specific standard. Neither the accept header nor file extension can support that.

By the way, this is something the W3C Data Exchange WG is looking at (see also the draft RFC): https://w3c.github.io/dxwg/ucr/#Conneg

from ogcapi-features.

It's important to distinguish between the user agent (software) and the user (wetware).

The user agent has a set of formats which its' program logic can handle. Those formats are advertised in the accept header.

The user wants the data to comply with a specific format or schema. It does not have to be a format that the user agent can process. The user agent only has to successfully deliver the data.

Consider the case where the user agent supports XML and the user wants GML 3.2.
IF the user agent sets the accept header for XML and the server replies with a content type header of XML, then all is good.
IF the user overwrites the accept header with GML 3.2 and the server replies with content type GML 3.2, what does the user agent do?

Can it be made to work? perhaps. Is it optimal? I don't think so.

from ogcapi-features.

Where a format doesn't support embedded <link rel="alternate" type="application/geojson" href="...">, the link header could be used, with the suggested value of the Accept header in the @type field of that header. The service could use whatever means it desired to provide the URLs for such links. Maybe this has been mentioned before, but it just occurred to me. :-)

from ogcapi-features.