ronnypfannschmidt / prance Goto Github PK
View Code? Open in Web Editor NEWResolving Swagger/OpenAPI 2.0 and 3.0 Parser
License: Other
Resolving Swagger/OpenAPI 2.0 and 3.0 Parser
License: Other
chardet v4 is finally out, and the setup.py chardet~=3.0
blocks co-habitation with it.
Within my JSON I have a $ref to a relative file path but when parsing get the error:
Exception caught: (<class 'swagger_spec_validator.common.SwaggerValidationError'>)
Exception message: unknown url type: ./schema/ChangePasswordEntity.json
Extract from swagger.json
{
"in": "body",
"name": "ChangePasswordEntity",
"required": true,
"schema": {
"$ref": "./schema/ChangePasswordEntity.json"
}
}
Code:
from prance import ResolvingParser
from prance.util import fs, formats
...
parser = ResolvingParser(file)
contents = formats.serialize_spec(parser.specification, filename)
fs.write_file(filename, contents)
Swagger spec supports recursion, and schemas that have recursive references are perfectly valid — no python library seems to support this though. RefResolver
has a built-in __recursion_protection
, but it protects against all recursion.
Would it be possible to allow it to add an option for recursion depth limit instead, and allow recursion but only until a certain depth limit is reached, and fail silently with a warning?
I'm creating app with Flask, Connexion and Swagger. I want split swagger yaml file in multilpe files.
I have followed this reply:
spec-first/connexion#254 (comment)
When I execute my app appears the exception. This exception is raised by the function _validate_openapi_spec_validator on class BaseParser, line 235.
validate_v2_spec(self.specification)
specification is a dict.
This is the yaml with the reference:
swagger: "2.0"
info:
title: "Document Repository Service API"
version: "1.0"
basePath: /v1.0
paths:
$ref: ./resolutions.yaml
And this is the yaml referenced:
/resolutions/{id}:
get:
x-swagger-router-controller: controllers.resolution_controller
operationId: get_resolution
parameters:
- name: id
in: path
description: Resolution ID
type: string
required: true
- name: format
in: query
type: string
enum: [xml, json]
required: false
default: xml
- name: encoding
in: query
type: string
enum: [utf-8, original]
required: false
default: utf-8
responses:
200:
description: OK
404:
description: A resolution with the specified ID was not found.
Hey, @jfinkhaeuser, you probably using compile
but should use convert
in a readme.
# Convert spec
$ prance convert path/to/swagger.yml path/to/openapi.yml
# Convert spec
$ prance compile path/to/swagger.yml path/to/openapi.yml
Originally we replaced chardet with ICU because chardet had some issues detecting UTF-8. But in some environments, (e.g. google cloud engine), ICU might not be available, so chardet is still a better choice than nothing at all.
As stated in #69, it would be great if we could get some more context into error messages. One way way it might work is if we get file/line information from the YAML/JSON parsers and annotate the specs with something that the validators can ignore, then use those annotations in error messages.
With pyyaml
, we should be able to get this information out of buffers. With JSON, it might be trickier at first glance - worst case we have to use some dependency.
Suggestions are welcome. This issue is more of a reminder to myself to find what we can do here.
A have a simple openapi definition prepared to present the problem, with a sinlge Polish letter: ł.
openapi.yaml:
openapi: 3.0.2
info:
title: title
version: '1.0'
paths:
/something/:
post:
summary: summary
requestBody:
content:
application/json:
schema:
type: string
responses:
'400':
description: description
content:
application/json:
schema:
type: object
properties:
info:
type: array
items:
type: string
enum:
- ł
That snippet compiles as expected:
% prance compile --strict openapi.yaml all-in-one-openapi.yaml
Processing "openapi.yaml"...
-> Resolving external references.
Validates OK as OpenAPI 3.0.2!
Output written to "all-in-one-openapi.yaml".
But I wanted to spit this definition into parts. So I prepared two another files:
openapi-main.yaml:
openapi: 3.0.2
info:
title: title
version: '1.0'
paths:
/something/:
$ref: 'openapi-part.yaml'
openapi-part.yaml:
---
post:
summary: summary
requestBody:
content:
application/json:
schema:
type: string
responses:
'400':
description: description
content:
application/json:
schema:
type: object
properties:
info:
type: array
items:
type: string
enum:
- ł
And now my problem arises:
% prance compile --no-strict openapi-main.yaml all-in-one-openapi.yaml
Processing "openapi-main.yaml"...
-> Resolving external references.
Traceback (most recent call last):
File "/home/xxx/.virtualenvs/prance/bin/prance", line 8, in <module>
sys.exit(cli())
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/cli.py", line 103, in command_invoke
original_invoke(ctx)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/cli.py", line 220, in compile
__validate(parser, name)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/cli.py", line 54, in __validate
parser.parse()
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/__init__.py", line 134, in parse
self._validate()
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/__init__.py", line 271, in _validate
resolver.resolve_references()
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/resolver.py", line 79, in resolve_references
self.specs = self._resolve_partial(self.parsed_url, self.specs, ())
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/resolver.py", line 175, in _resolve_partial
recursions)))
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/resolver.py", line 117, in _dereferencing_iterator
ref_value = self._dereference(ref_url, obj_path, next_recursions)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/resolver.py", line 139, in _dereference
contents = _url.fetch_url(ref_url, self.__reference_cache)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/url.py", line 215, in fetch_url
result = parse_spec(content, url.path, content_type = content_type)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/formats.py", line 205, in parse_spec
result, ctype, ext = parse_spec_details(spec_str, filename, **kwargs)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/formats.py", line 180, in parse_spec_details
result = parser(spec_str)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/formats.py", line 67, in __parse_yaml
return yaml.safe_load(six.text_type(spec_str))
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/yaml/__init__.py", line 162, in safe_load
return load(stream, SafeLoader)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/yaml/__init__.py", line 112, in load
loader = Loader(stream)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/yaml/loader.py", line 34, in __init__
Reader.__init__(self, stream)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/yaml/reader.py", line 74, in __init__
self.check_printable(stream)
File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/yaml/reader.py", line 144, in check_printable
'unicode', "special characters are not allowed")
yaml.reader.ReaderError: unacceptable character #x0082: special characters are not allowed
in "<unicode string>", position 432
In this case I can do some workarounds. Eg. remove three dashes (---) from the beginning of openapi-part.yaml file, which are not obligatory in such a situation, but it desn’t work in all my real cases. It’s also really amazing that I can leave those dashes in the file and remove its third line (summary:) and it works fine, as well.
I can't discover any rules in this behavior.
I use prance==0.16.2 with python 3.7.5.
When YAML is failed in parsing, can we get a list of errors/warnings of the yaml file? It would be helpful if line number and error message are given for every parse error.
find . -iname '*.py' | grep -Ev 'setup|rdf4|tool' | xargs -P4 -I{} python3.8 -Wall -m py_compile {}
./tests/test_util_path.py:56: SyntaxWarning: "is" with a literal. Did you mean "=="?
assert result is 666
This enhances #10.
None
.Swagger reads ref file with encoded url but prance through errr file not found error.
$ref:'#/paths/1api1%7Bname%7D~1data
Error:
Cannot resolve reference "file:///
If RESOLVE_INTERNAL
is unset, internal references within an included file should not be resolved. Discovered while investigating #77
{
"openapi": "3.0.0",
"info": {
"title": "Test API",
"version": "1"
},
"paths": {
"/endpoint": {
"post": {
"requestBody": {
"$ref": "_schemas.json#/$defs/Body"
},
"responses": {}
}
}
}
}
And _schemas.json
:
{
"$id": "_schemas.json",
"$schemas": "https://json-schema.org/draft/2019-09/schema#",
"$defs": {
"Something": {
"type": "object"
},
"Body": {
"content": {
"application/json": {
"schema": {
"$ref": "#/$defs/Something"
}
}
}
}
}
}
Internal reference to $defs/Something
is resolved.
See #77
In YAML you can write strings without quotes. For some specific cases you are forced to use quotes, for example when you put digits, but you want them to be of type string.
But when you write schema and set type of variable to string - when you provide example
you expect that it will also be of type string. But it's converted to the python as integer.
Some schema in requestBody
for example:
type: object
properties:
some_variable:
type: string
example: 1111111111 # This should be string
required: true
This should be:
{'some_variable':'1111111111'}
type: object
properties:
some_variable:
type: string
example: 1111111111
required: true
Result:
{'some_variable': 1111111111}
parse with no errors
components:
schemas:
topo_info:
type: object
properties:
id:
type: string
description: Topo ID
name:
type: string
description: Topo name
child_topos:
type: array
description: Child topo
items:
- $ref: '#/components/schemas/topo_info'
The parser falls to endless loop . I set recursive limit to 100 , it donot work .
File "/api_spec/prance/util/resolver.py", line 175, in _resolve_partial
recursions)))
File "/api_spec/prance/util/resolver.py", line 102, in _dereferencing_iterator
ref_value = self.__reclimit_handler(self.__reclimit, ref_url)
File "/api_spec/prance/util/resolver.py", line 16, in default_reclimit_handler
'resolve "%s"!' % (limit, parsed_url.geturl()))
ResolutionError: Recursion reached limit of 100 trying to resolve file
Are there any reasons to require the very specific version of this library?
I just ran into a situation where I cannot use prance
together with docker-compose
because the later outlaws both the 2.21
and 2.18
(I believe they've encountered bugs in those versions). This is while your package requires specifically those versions (the 2.18
is the one before the present requirement).
I don't really see in your project any usage of requests
that would have worked differently with older versions, why did you choose to require this particular version?
We'd need travis-ci/travis-ci#6865 to be fixed. Or plenty of time to play around with different versions, which will break again in future.
It is expected that the $ref which contains '~1' should be correctly parsed
a.yaml
paths:
/api/v2/vms:
$ref: ./b.yml#/paths/~1api~1v2~1vms
..........................
other thing
b.yaml
paths:
/api/v2/vms:
get:
operationId: list_vm
description: get all
............................
other thing
raise ResolutionError: Cannot resolve reference actualy
I'm trying to enable breaking up yaml swagger specs into smaller files in a set of connexion projects. Currently, connexion does not support the use of relative file references (see issue here). So I investigated prance as a solution.
Technically, it works, in the sense that it enables us to build a fully-resolved spec from a separated spec by resolving the references. However, it is very slow. Ideally, we would like to just load and resolve the swagger files when the application boots, which is normally instantaneous. One project, which has the following swagger:
takes about 20 seconds to resolve the spec on my fancy, expensive laptop. Obviously, this is unacceptable for anything but a way to "compile" a spec which is just loaded each time you boot. Also, it's just plain weirdly slow.
Profiling ResolvingParser
, I see 3097572 calls (30,124,069 entries total) to dpath.path.paths
, which seems like a lot. Poking around a bit in prance.util.resolver.RefResolver
with my debugger, it looks to me like the following is going on:
path/to/file#/definitions/Something
we do some processing and then walk a doc tree, even though this is the same value every time.resolve_references
changes loop, the duplication mentioned in the comment is substantial; 659 pairs vs 174 unique pairs. In some cases, a particular key is set 100 times.The ratio of wall time spent before the biggest changes loop to after the biggest changes loop is about 1:10, so (1) may not be the most pressing issue, but it's still a good couple of seconds, and it may be possible to cut that to milliseconds with some caching. The big problem is changes loop, where deduping the changes list is more significant than the comment makes out.
Fortunately, it's pretty easy to improve this substantially by looping over uniq_changes = dict(changes)
. Tests pass, although I haven't looked at the test suite to determine whether this actually tells us that nothing has gone wrong. It's still kind of slow, but I'd have to think much harder about whether all the dutil.set
ops can be speeded up in any way, and if so whether this is feasible. For example, it looks like dutil.set
is designed to operate in batches, so maybe all the ops which set various path keys to a particular resolved reference could be batched into a single operation. That way we do one dutil.set
for all the references to some/path#/definitions/Foo
. I don't know enough about dutil.set
to know whether this would mean a nontrivial speedup, a trivial speedup, or a slowdown.
Anyway, questions:
I get this error trying to do:
from prance import ResolvingParser
p = ResolvingParser('swagger.yaml')
The yaml file is valid according to editor.swagger.io and http://bigstickcarpet.com/swagger-parser/www/index.html
I have no idea where in the swagger file the error is. A line number or similar would be helpful
I had to rollback to 0.9.0 as the new release doesn't parse my swagger. The errors look like:
ipdb> prance.ResolvingParser(filepath)
*** prance.SwaggerValidationError: 'paths':
- '/person/alias':
- 'post':
- 'parameters':
- 'referenceObject':
- 'required':
- '$ref':
- 'This value is required'
- 'parameterObject':
- 'required':
- 'name':
- 'This value is required'
- 'in':
- 'This value is required'
I can attempt to create a small test case for this if necessary, but I won't get to it for a week or so.
The util.url.fetch_url
and util.url.fetch_url_text
functions are supposed to have an optional cache ("Mapping cache: An optional cache. If the URL can be found in the cache, return the cache contents.", as per the documentation)
However, in the current implementation there will always be a cache, which I believe was not the intention.
The root cause is using {}
as default for the cache
argument; rather than re-instantiating a new dictionary every time the function is called, this re-uses the same dictionary between function calls.
See e.g. https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments for details
A simple fix would be:
def fetch_url(url, cache = None, encoding = None):
cache = cache or {}
URL contents should not be cached unless the cache
argument is provided.
URL contents get cached if no cache
argument is provided.
Given a file test.yaml
with the following contents:
openapi: 3.0.2
info:
title: Test
version: '0.0.1'
and the following program:
import os
from prance.util.url import absurl
from prance.util.fs import abspath
from prance.util.url import fetch_url
url = absurl("test.yaml", abspath(os.getcwd()))
fetch_url(url)
we get the following output:
{'openapi': '3.0.2', 'info': {'title': 'Test', 'version': '0.0.1'}}
Change the file to e.g.
openapi: 3.0.2
info:
title: Foobar
version: '0.0.1'
and then re-run the fetch_url(url)
part. I would expect the output to change, but you still get:
{'openapi': '3.0.2', 'info': {'title': 'Test', 'version': '0.0.1'}}
I would like to find the endpoint in my spec file according to a given URL.
Given /networks/my-net/rbac
, I would like prance to give me the spec for the endpoint listed under the minimal example.
paths:
/networks/{network}/rbac:
I am not aware of a feature in prance that would allow this.
Providing some context which part of my 113k line file failed.
prance validate v1.16.4.json
Processing "v1.16.4.json"...
-> Resolving external references.
Traceback (most recent call last):
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/bin/prance", line 8, in <module>
sys.exit(cli())
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/cli.py", line 108, in command_invoke
original_invoke(ctx)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/cli.py", line 197, in validate
__validate(parser, name)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/cli.py", line 59, in __validate
parser.parse()
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/__init__.py", line 138, in parse
self._validate()
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/__init__.py", line 294, in _validate
resolver.resolve_references()
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 96, in resolve_references
self.specs = self._resolve_partial(self.parsed_url, self.specs, ())
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 207, in _resolve_partial
changes = dict(tuple(self._dereferencing_iterator(base_url, partial, (),
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 133, in _dereferencing_iterator
ref_value = self._dereference(ref_url, obj_path, next_recursions)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 190, in _dereference
value = self._resolve_partial(ref_url, value, recursions)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 207, in _resolve_partial
changes = dict(tuple(self._dereferencing_iterator(base_url, partial, (),
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 133, in _dereferencing_iterator
ref_value = self._dereference(ref_url, obj_path, next_recursions)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 190, in _dereference
value = self._resolve_partial(ref_url, value, recursions)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 207, in _resolve_partial
changes = dict(tuple(self._dereferencing_iterator(base_url, partial, (),
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 133, in _dereferencing_iterator
ref_value = self._dereference(ref_url, obj_path, next_recursions)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 190, in _dereference
value = self._resolve_partial(ref_url, value, recursions)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 207, in _resolve_partial
changes = dict(tuple(self._dereferencing_iterator(base_url, partial, (),
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 133, in _dereferencing_iterator
ref_value = self._dereference(ref_url, obj_path, next_recursions)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 190, in _dereference
value = self._resolve_partial(ref_url, value, recursions)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 207, in _resolve_partial
changes = dict(tuple(self._dereferencing_iterator(base_url, partial, (),
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 112, in _dereferencing_iterator
ref_url, obj_path = _url.split_url_reference(base_url, refstring)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/url.py", line 120, in split_url_reference
parsed_url = absurl(reference, base_url)
File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/url.py", line 56, in absurl
parsed = parse.urlparse(url)
File "/usr/lib64/python3.8/urllib/parse.py", line 372, in urlparse
url, scheme, _coerce_result = _coerce_args(url, scheme)
File "/usr/lib64/python3.8/urllib/parse.py", line 124, in _coerce_args
return _decode_args(args) + (_encode_result,)
File "/usr/lib64/python3.8/urllib/parse.py", line 108, in _decode_args
return tuple(x.decode(encoding, errors) if x else '' for x in args)
File "/usr/lib64/python3.8/urllib/parse.py", line 108, in <genexpr>
return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'dict' object has no attribute 'decode'
wget https://raw.githubusercontent.com/kubernetes/kubernetes/v1.16.4/api/openapi-spec/swagger.json
prance validate swagger.json
I may have be missing something which is already possible, so apologies if this is case.
I'd like to be able to have a reference to a specific part of the 'parent' file become an internal reference when the 'child' file is resolved.
root/parent.yaml
Child:
$ref: './subfolder/child.yaml'
Parent:
...
root/subfolder/child.yaml
something:
$ref: '../parent.yaml#/Parent
Would resolve to:
compiled.yaml
Child:
something:
$ref '#/Parent'
Parent:
...
I've tried to produce this effect with the partial resolution option, but have not been able to get it to work. Applying only RESOLVE_FILES will result in a crash, as the references isn't actually an internal reference. Running without RESOLVE_FILES will fully resolve the reference.
No deprecation warnings produced when prance is used
I use Prance in my OpenAPI / Connexion app like so
def get_bundled_specs(main_file):
parser = prance.ResolvingParser(str(main_file.absolute()), lazy=True, strict=True)
parser.parse()
return parser.specification
This deprecation warning is output:
<snip>/.venv/lib/python3.7/site-packages/prance/util/path.py:122: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
if not isinstance(obj, collections.MutableMapping): # pragma: nocover
Simply run the app.
I can use this as a library only with no dependency on click
My CLI (which uses Click 7.0) is unable to use this as a library as there is a hard dependency for Click 6.7. It be nice if this could be used as a library only without any CLI specific dependencies (and not requiring a fork).
This is more of an question on whether is this even possible. It applies to a state when resolve_types does not include RESOLVE_INTERNAL. The motivation for this is that it is not possible to have recursive schema definitions in referenced local files. Like:
{
"$defs": {
"NestedObject": {
"propertyNames": {
"minLength": 1
},
"additionalProperties": {
"$ref": "#/$defs/NestedObject"
}
}
}
}
Local references in external files are kept local. They are injected to the main specification and converted to the new path.
{
"openapi": "3.0.0",
"info": {
"title": "Test API",
"version": "1"
},
"paths": {
"/endpoint": {
"post": {
"requestBody": {
"content": {
"application/json": {
"schema": {"$ref": "_schemas.json#/$defs/Parameter"}
}
}
},
"responses": {}
}
}
}
}
External file with schema definitions.
{
"$id": "_schemas.json",
"$schemas": "https://json-schema.org/draft/2019-09/schema#",
"$defs": {
"Something": {
"type": "object"
},
"Parameter": {
"properties": {
"something": {
"$ref": "#/$defs/Something"
}
}
}
}
}
Local references in external files are dereferenced. But how? They’d need to be somewhere into the composed specification. Something like:
{
...
"paths": {
"/endpoint": {
"post": {
"requestBody": {
"content": {
"application/json": {
"schema": {"$ref": "#/x-prance-refs/_schemas.json%23%2F%24defs%2FSomething"}
}
}
},
...
}
}
},
"x-prance-refs": {
"_schemas.json#/$defs/Something": {
"type": "object"
}
}
}
Here is a simple script exposing what is dereferenced:
from prance import ResolvingParser
from prance.util.resolver import RESOLVE_FILES
def _main():
parser = ResolvingParser("_openapi.json", resolve_types=RESOLVE_FILES)
parser.parse()
print(parser.specification["paths"]["/endpoint"]["post"]["requestBody"]["content"]["application/json"]["schema"])
if __name__ == "__main__":
_main()
While using the validate
command to combine specs by dereferencing is a very nice emergent property of prance, it may be useful making it explicit.
Effectively, using validate --output-file
should raise a deprecation warning, and a new compile
command should cover the same functionality.
The compile
command should also include automatic conversion of 2.0 to 3.0 as per #17 if and only if mixed specs are provided on the command line. Additionally, the command should try to actually merge specs, instead of just overwriting the same file over and over.
Given a YAML file which paths.{path}.get.responses
has a response code expressed as a number,
two things could happen:
I have a message like this:
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "server.py", line 21, in <module>
app.add_api(get_bundled_specs(Path('specifications/openapi.yaml')))
File "server.py", line 13, in get_bundled_specs
parser.parse()
File "/usr/local/lib/python3.5/dist-packages/prance/__init__.py", line 134, in parse
self._validate()
File "/usr/local/lib/python3.5/dist-packages/prance/__init__.py", line 275, in _validate
BaseParser._validate(self)
File "/usr/local/lib/python3.5/dist-packages/prance/__init__.py", line 172, in _validate
validator(parsed)
File "/usr/local/lib/python3.5/dist-packages/prance/__init__.py", line 225, in _validate_openapi_spec_validator
raise_from(ValidationError, type_ex)
File "<string>", line 5, in raise_from
prance.ValidationError: expected string or bytes-like object
This message is very misleading. For a long file, it took me a considerable amount of time to find the problem.
Create a YAML file like this:
openapi: 3.0.0
info:
description: My API
version: "1.0.0"
title: My Swagger API
paths:
/people:
get:
operationId: controllers.people.read_all
tags:
- people
summary: Retrieves people
description: Retrieves people
parameters:
- name: length
in: query
schema:
type: integer
description: Number of people to get from database
required: false
- name: offset
in: query
schema:
type: integer
description: Offset from beginning of list where to start gathering people
required: false
responses:
200:
description: Successful
Make sure the response code is integer, not string.
I expect:
{$ref: "http://foo.com/bar.yaml#/Object"}
https://github.com/teamdigitale/api-starter-kit/blob/master/openapi/simple.yaml.src
Processing "https://gist.githubusercontent.com/ioggstream/c0adc0e37f3a35b2fc97c5d5238ebdda/raw/91c353b5432cbba61cbb479f82f7d835c99df3be/simple.yml"...
-> Resolving external references.
Traceback (most recent call last):
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/bin/prance", line 11, in <module>
sys.exit(cli())
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/cli.py", line 103, in command_invoke
original_invoke(ctx)
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/cli.py", line 182, in validate
__validate(parser, name)
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/cli.py", line 54, in __validate
parser.parse()
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/__init__.py", line 133, in parse
self._validate()
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/__init__.py", line 267, in _validate
resolver.resolve_references()
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/resolver.py", line 72, in resolve_references
self.specs = self._resolve_partial(self.parsed_url, self.specs, ())
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/resolver.py", line 164, in _resolve_partial
recursions)))
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/resolver.py", line 106, in _dereferencing_iterator
ref_value = self._dereference(ref_url, obj_path, next_recursions)
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/resolver.py", line 146, in _dereference
value = self._resolve_partial(ref_url, value, recursions)
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/resolver.py", line 171, in _resolve_partial
path_set(partial, list(path), value, create = True)
File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/path.py", line 116, in path_set
raise KeyError('Cannot set with an empty path!')
KeyError: 'Cannot set with an empty path!'
prance compile --backend=openapi-spec-validator https://gist.githubusercontent.com/ioggstream/c0adc0e37f3a35b2fc97c5d5238ebdda/raw/91c353b5432cbba61cbb479f82f7d835c99df3be/simple.yml
This code can resolve references (but does not validate it). Maybe there's something you can reuse.
https://github.com/teamdigitale/api-starter-kit/blob/master/scripts/openapi_resolver.py
Hi!
First of thank you for this project - it is exactly what I need!
However, I discovered a bug that occurred when the response code of a method in the Swagger file is interpreted as an integer.
The error is expected string or bytes-like object
.
Example:
swagger: '2.0'
info:
title: Test API
version: "0.1"
produces:
- application/json
basePath: /api/v1
paths:
/test:
post:
operationId: test_api.api.fromurl.post
responses:
200: #<---
description: Returns something
If I use "200"
instead, it works.
I discovered that the bug actually happens in jsonschema._utils
, where a regex fails to run over the integer. Strangely, other packages that I previously used to interpret Swagger (like connexion) also depend on the jsonschema
package and they work fine.
I tested this on Python 3.6.1 on OS X 10.11.6. The version of jsonschema
is 2.6.0 and of prance
0.5.1
I would really appreciate it, if you could investigate this bug!
I have a swagger file, from which I would like to extract a JSON schema object for the response to each API endpoint. JSON schema doesn't like the allOf
magic we've used, so it would be nice if this can be automatically expanded out with an option to prance
.
I've half hacked together a script to do this for our case, but I feel like this feature ought to sit in prance itself. What do you think? Is this difficult to implement? (I haven't read the source yet)
The CLI tests test_convert_defaults
and test_convert_output
fail without network.
In RPM packaging, all tests which require a network have to be disabled as the RPMs are built in an isolated VM to ensure they are reproducible.
As this two tests are verifying CLI behaviour, it would be a good idea if they didnt rely on the network.
A new command should convert OpenAPI 2.0 specs to 3.0, as well as possible. In code, this should be functionality to run on a parser's spec, so we can use it on the BaseParser prior to resolving references.
Tests would require that converted specs validate as 3.0.
ResolvingParser
won't dereference local references when resolve_types
option is set to prance.util.resolver.RESOLVE_FILES
.
openapi: "3.0.0"
info:
title: ''
version: '1.0.0'
paths: {}
components:
schemas:
SampleArray:
type: array
items:
$ref: '#/components/schemas/ItemType'
ItemType:
type: integer
ResolvingParser
resolves internal references.
Assume that minimal spec file is in D:/test.yaml
.
import prance
import prance.util.resolver
parser = prance.ResolvingParser(url='D:\\test.yaml', resolve_types=prance.util.resolver.RESOLVE_FILES)
parser.specification
{'openapi': '3.0.0', 'info': {'title': '', 'version': '1.0.0'}, 'paths': {}, 'components': {'schemas': {'SampleArray': {'type': 'array', 'items': {'type': 'integer'}}, 'ItemType': {'type': 'integer'}}}}
I guess, problem is here. Prance trying to compare str
and ParseResult
objects.
Consider creating a changelog so that it would be possible to easily determine roughly which changes have been made between the released versions.
Given data (as YAML or JSON, or in code as nested structures), validate it against schemas in a spec. Well, basically against anything that can live in the components object (3.0)
I wish a parameter for a partial resolution of items, eg:
components
#/components/schemas/...
...
schemas:
A:
properties:
a: string
b:
$ref: https://foo.com/#B
...
should become
...
schemas:
A:
a: string
b:
$ref: '#/components/schemas/B'
B:
<all b stuff>
...
Full resolution is done
https://github.com/jfinkhaeuser/prance/blob/master/tests/specs/symlink_test is supposed to be a symlink, but in the sdist uploaded to PyPI it isnt a symlink.
Currently this code:
``
from .util.exceptions import raise_from
try:
try:
validate_v3_spec(self.specification)
self.__set_version(BaseParser.SPEC_VERSION_3_PREFIX, spec_version)
except TypeError as type_ex: # pragma: nocover
raise_from(ValidationError, type_ex)
except JSEValidationError as v3_ex:
try:
validate_v2_spec(self.specification)
self.__set_version(BaseParser.SPEC_VERSION_2_PREFIX, spec_version)
except TypeError as type_ex: # pragma: nocover
raise_from(ValidationError, type_ex)
except RefResolutionError as ref_ex:
raise_from(ValidationError, ref_ex)
``
Produces wrong validation messages when there is an error in openapi schema 3.0.0
It returns openapi 2.0.0 messages instead of 3.0.0. After removing fallback correct error messages are shown.
It should use fallback only when openapi version was not specified. I could produce patch, but I don't know what is your preferred way to solve this problem. This fallback is for a reason there?
The readme says openapi-spec-validator is Python 3 only while it also supports Python 2.
Apologies for not following the template but this is an odd situation that doesn't really fit.
Quite rightly, in commit a2ba2c8, which went into 0.14.0, you bumped the version of PyYaml to ">4.0.0" which should include the safe secure behaviour for loading yaml. However there's been a fair amount of drama over in pyyaml's repo which you may have missed.
As far as I can tell from this ticket the maintainer got cold feet about the release of 4.x.x as it was a backwards incompatible change to load. They seem to have abandoned the 4.x.x releases up on PyPi.
The new release will be called
5.1
and I'll write up a "PyYAML 5.1 Release Plan" issue when the time is right.There seems to be some suggesting that people use one of the failed
4.2bx
releases to get #74 behavior. This is a bad idea. 3.13 is the current supported release. I could delete the 4.2b-s from PyPI but I haven't. I almost certainly will after 5.1 goes out.
Seems a bit crazy to me, but that's how things stand right now.
This is causing me issues in my project using prance as pipenv is unable to resolve the requirement for a 4.x version of PyYAML as no officially released version of that package exists.
I am able to tell pipenv to consider pre-releases as well, but unfortunately that's a global option meaning that I'll get pre-releases for everything which causes more issues. It'd obviously be better if pipenv was able to more specifically target enable pre-releases (this is tracked in an issue on pipenv).
I am pretty sure I can work around that limitation in pipenv, but even if I do, given the maintainer seems to have abandoned the 4.x release, I'm not sure it's such a good idea.
For the time being I'll probably freeze prance to "<0.14" until the issue is resolved.
That being said, if possible it seems like a good idea to go back to the old PyYaml version for the time being and it'd be awesome to see a new release of prance with that so I can unfreeze again.
Thanks for all the good work!
I am using connexion and prance with openapi and a recursive json schema that looks like:
"information": {
"type": "object",
"properties": {
"object": {
"type": "array",
"items": {
"anyOf": [
{
"$ref": "#/definitions/object"
},
{
"$ref": "#/definitions/information"
}
]
}
},
"feature": {
"type": "array",
"items": {
"$ref": "#/definitions/feature"
}
}
}
}
where the object can be a simple object or a nested information
I have seen issue #55 and tried to set a max recursion limit together with an handle that return None but I get a prance.ValidationError
following is the traceback
traceback (most recent call last):
File "C:\code\test\test_openapi_json_schema_validation\env\lib\site-packages\prance_init_.py", line 224, in _validate_openapi_spec_validator
validate_v3_spec(self.specification)
File "C:\code\test\test_openapi_json_schema_validation\env\lib\site-packages\openapi_spec_validator\shortcuts.py", line 7, in validate
return validator_callable(spec, spec_url=spec_url)
File "C:\code\test\test_openapi_json_schema_validation\env\lib\site-packages\openapi_spec_validator\validators.py", line 48, in validate
raise err
openapi_spec_validator.exceptions.OpenAPIValidationError
and here is the code
def recursion_limit_handler_none(limit, refstring, recursions):
return None
def get_bundled_specs(main_file: Path) -> Dict[str, Any]:
parser = prance.ResolvingParser(str(main_file.absolute()),
lazy=True,
recursion_limit=1,
recursion_limit_handler=recursion_limit_handler_none,
backend='openapi-spec-validator')
parser.parse()
return parser.specification
app = connexion.FlaskApp(name)
app.add_api(
get_bundled_specs(Path("api.yaml")),
strict_validation=True,
validate_responses=True,
resolver=connexion.RestyResolver("cms_rest_api"))
It is not clear to me whether it is a problem due to the recursive json, that might not be supported or I made it wrong, or is due to my poor understanding on issue #55 and how to use it
Any help it is appreciated!!
Before I work on PR wanted to ask...
Local # references expanded. Instead of this 'paginator': {'$ref': '#/definitions/Paginator'} I'd expect the {ref} dict to be replaced with the actual definition. Or am I just not understanding how this is suppose to work?
ERROR in "spec.yaml" [ResolutionError]: Recursion reached limit of 1 trying to resolve "file:///spect.yaml#/definitions/V1Fooer"!
Hi there, and first off thanks for this resolver!
I'm maintaining another library that relies on Prance as a derferencer. It works well in most cases, but I think I encountered a bug. I'm testing our library (https://github.com/snok/drf-openapi-tester) against various specs, and now I'm trying to test it against the kubernetes swgger 2.0 docs:, which can be found here: https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json
It seems that this spec cannot be resolved at present because instead of a url what is passed to the following function is a dictionary {type: string}
:
def absurl(url, relative_to = None):
....
# prance.util.url lines 49-59
parsed = url
if not isinstance(parsed, tuple):
from .fs import is_pathname_valid
if is_pathname_valid(url):
from . import fs
url = fs.to_posix(url)
try:
parsed = parse.urlparse(url)
except Exception as ex:
from .exceptions import raise_from
raise_from(ResolutionError, ex, 'Unable to parse url: %s' % (url,))
I got to this point in debugging it.
According to the intellij debugger absurl
was called with two parameters:
url={'type': 'string'},
relative_to=ParseResult(scheme='file', netloc='', path='/', params='', query='', fragment='/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps')
If you want I can add a json copy of the kubernetes spec to the repo and have it fail in a test if that would help :).
strict=False
should work with ResolvingParser when operations are defined in a separate file.
$ prance validate --no-strict bad_spec.yaml
Processing "bad_spec.yaml"...
-> Resolving external references.
Validates OK as OpenAPI 3.0.3!
bad_spec.yaml
openapi: 3.0.3
info:
title: OpenAPI spec to demonstrate stringify reference issue
description: |
This file is references one GET operation.
and should validate fine with prance using `strict=False` but currently breaks.
version: '1.0'
paths:
/bad:
$ref: 'bad_get.yaml'
bad_get.yaml
get:
responses:
200:
description: Success
Validation fails:
$ prance validate --no-strict bad_spec.yaml
Processing "bad_spec.yaml"...
-> Resolving external references.
ERROR in "bad_spec.yaml" [ValidationError]: expected string or bytes-like object -- Strict mode enabled (the default), so this could be due to an integer key, such as an HTTP status code.
Create the files above (bad_spec.yaml
and bad_get.yaml
) and then run:
prance validate --no-strict bad_spec.yaml
Note that this works fine:
$ openapi-spec-validator bad_spec.yaml
OK
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.