old user preferences containing "%" break PUT on /api/0.6/user/preferences about openstreetmap-website HOT 6 CLOSED

I cannot reproduce your issue locally:

Verifying the previous request:

I also checked the exact same requests on https://master.apis.dev.openstreetmap.org/, and it's also working as shown above.

Presumably, you're sending your request with a Content-Type application/x-www-form-urlencoded or something similar, in which case the request fails with HTTP 400.

from openstreetmap-website.

Thanks for looking into this! It was in fact simply a bug in overpass turbo, which is now fixed with tyrasd/overpass-turbo@8db5c43. 🙇

from openstreetmap-website.

Also, the issue in #4001 is a completely different one. There's a missing "force string to be treated as UTF-8" in the character validation rules.

Encoding::CompatibilityError: incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)

API threw unexpected Encoding::CompatibilityError exception: incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
openstreetmap-website/app/validators/characters_validator.rb:6:in `match?'
openstreetmap-website/app/validators/characters_validator.rb:6:in `validate_each'

from openstreetmap-website.

Presumably, you're sending your request with a Content-Type application/x-www-form-urlencoded or something similar, in which case the request fails with HTTP 400.

I was investigating this yesterday, and came to the same conclusion. In fact, it's easily reproduced in our test suite, since the put requests fail with the same 400 response if you add a % character:

       xml = "<osm><changeset>" \
-            "<tag k='created_by' v='osm test suite checking changesets'/>" \
+            "<tag k='created_by' v='osm test suite checking changesets%'/>" \
             "</changeset></osm>"

Failure:
Api::ChangesetsControllerTest#test_create [/home/andy/src/openstreetmap-website/test/controllers/api/changesets_controller_test.rb:82]:
Creation of changeset did not return success status

Started PUT "/api/0.6/changeset/create" for 127.0.0.1 at 2024-01-11 09:52:24 +0000
  
Rack::QueryParser::InvalidParameterError (invalid %-encoding ('created_by' v='osm test suite checking changesets%'/></changeset></osm>))

If we add a "Content-Type: application/xml" header to the put request in the test suite, then that stops Rack from treating it as url-encoded payload, and correctly treats it as XML.

So the first step is as @mmd-osm explains - the client application should set the correct content-type header, and then it will hopefully work fine.

The second step is to open the discussion about whether we should set a default content-type for the API. This is one of these things were someone has built their client application, and it's worked fine without it, until someone hits an edge case and it breaks. Should we treat all requests to the XML api as application/xml by default, to avoid this from happening? Or should we start throwing errors for incorrect content types, so that developers find this out during testing rather than user reports year(s) later?

from openstreetmap-website.

So in this particular case, overpass turbo is indeed sending a request with Content-Type application/x-www-form-urlencoded, although the code explicitly requested Content-Type text/xml earlier on. I haven't investigated in more detail why this is happening. I'd say the issue would have to be fixed on overpass turbo side.

Should we treat all requests to the XML api as application/xml by default, to avoid this from happening? Or should we start throwing errors for incorrect content types, so that developers find this out during testing rather than user reports year(s) later?

Throwing errors for unsupported content types sounds like the better approach. I'm a bit worried that such a change might cause too much breakage. Since we're not mentioning the Content-Type header even once on the OSM API 0.6 wiki page, people might use whatever value that happens to work somehow.

An aggregated log with e.g. User-Agent and Content-Type for a given endpoint might be useful to better understand how much of a problem this really is.

from openstreetmap-website.

Thanks everybody, you are indeed correct -- I was only using copy as cURL as produced by overpass-turbo trying to determine why it worked for everybody else but not me, so I was also using incorrect Content-Type: application/x-www-form-urlencoded.

I can confirm that the overpass-turbo fix above inspired by your comments works fine and solves overpass-turbo issue, so I'm closing this.

from openstreetmap-website.