Giter Club home page Giter Club logo

prance's Introduction

License PyPI Python Versions Package Format Package Status

Logo

Prance provides parsers for Swagger/OpenAPI 2.0 and 3.0 API specifications in Python. It uses openapi_spec_validator, swagger_spec_validator or flex to validate specifications, but additionally resolves JSON references in accordance with the OpenAPI spec.

Mostly the latter involves handling non-URI references; OpenAPI is fine with providing relative file paths, whereas JSON references require URIs at this point in time.

Usage

Installation

Prance is available from PyPI, and can be installed via pip:

$ pip install prance

Note that this will install the code, but additional subpackages must be specified to unlock various pieces of functionality. At minimum, a parsing backend must be installed. For the CLI functionality, you need further dependencies.

The recommended installation installs the CLI, uses ICU and installs one validation backend:

$ pip install prance[osv,icu,cli]

Make sure you have ICU Unicode Library installed, as well as Python dev library before running the commands above. If not, use the following commands:

$ sudo apt-get install libicu-dev python3-dev # Ubuntu/Debian
$ sudo dnf install libicu-devel python3-devel # Fedora

Command Line Interface

After installing prance, a CLI is available for validating (and resolving external references in) specs:

# Validates with resolving
$ prance validate path/to/swagger.yml

# Validates without resolving
$ prance validate --no-resolve path/to/swagger.yml

# Fetch URL, validate and resolve.
$ prance validate http://petstore.swagger.io/v2/swagger.json
Processing "http://petstore.swagger.io/v2/swagger.json"...
 -> Resolving external references.
Validates OK as Swagger/OpenAPI 2.0!

Validation is not the only feature of prance. One of the side effects of resolving is that from a spec with references, one can create a fully resolved output spec. In the past, this was done via options to the validate command, but now there's a specific command just for this purpose:

# Compile spec
$ prance compile path/to/input.yml path/to/output.yml

Lastly, with the arrival of OpenAPI 3.0.0, it becomes useful for tooling to convert older specs to the new standard. Instead of re-inventing the wheel, prance just provides a CLI command for passing specs to the web API of swagger2openapi - a working internet connection is therefore required for this command:

# Convert spec
$ prance convert path/to/swagger.yml path/to/openapi.yml

Code

Most likely you have spec file and want to parse it:

from prance import ResolvingParser
parser = ResolvingParser('path/to/my/swagger.yaml')
parser.specification  # contains fully resolved specs as a dict

Prance also includes a non-resolving parser that does not follow JSON references, in case you prefer that.

from prance import BaseParser
parser = BaseParser('path/to/my/swagger.yaml')
parser.specification  # contains specs as a dict still containing JSON references

On Windows, the code reacts correctly if you pass posix-like paths (/c:/swagger) or if the path is relative. If you pass absolute windows path (like c:\swagger.yaml), you can use prance.util.fs.abspath to convert them.

URLs can also be parsed:

parser = ResolvingParser('http://petstore.swagger.io/v2/swagger.json')

Largely, that's it. There is a whole slew of utility code that you may or may not find useful, too. Look at the full documentation for details.

Compatibility

Python Versions

Version 0.16.2 is the last version supporting Python 2. It was released on Nov 12th, 2019. Python 2 reaches end of life at the end of 2019. If you wish for updates to the Python 2 supported packages, please contact the maintainer directly.

Until fairly recently, we also tested with PyPy. Unfortunately, Travis isn't very good at supporting this. So in the absence of spare time, they're disabled. Issue 50 tracks progress on that.

Similarly, but less critically, Python 3.4 is no longer receiving a lot of love from CI vendors, so automated builds on that version are no longer supported.

Backends

Different validation backends support different features.

Backend Python Version OpenAPI Version Strict Mode Notes Available From Link
swagger-spec-validator 2 and 3 2.0 only yes Slow; does not accept integer keys (see strict mode). prance 0.1 swagger_spec_validator
flex 2 and 3 2.0 only n/a Fastest; unfortunately deprecated. prance 0.8 flex
openapi-spec-validator 2 and 3 2.0 and 3.0 yes Slow; does not accept integer keys (see strict mode). prance 0.11 openapi_spec_validator

You can select the backend in the constructor of the parser(s):

parser = ResolvingParser('http://petstore.swagger.io/v2/swagger.json', backend = 'openapi-spec-validator')

No backend is included in the dependencies; they are detected at run-time. If you install them, they can be used:

$ pip install openapi-spec-validator
$ pip install prance
$ prance validate --backend=openapi-spec-validator path/to/spec.yml

A note on flex usage: While flex is the fastest validation backend, unfortunately it is no longer maintained and there are issues with its dependencies. For one thing, it depends on a version of PyYAML that contains security flaws. For another, it depends explicitly on older versions of click.

If you use the flex subpackage, therefore, you do so at your own risk.

Compatibility

See COMPATIBILITY.rst for a list of known issues.

Partial Reference Resolution

It's possible to instruct the parser to only resolve some kinds of references. This allows e.g. resolving references from external URLs, whilst keeping local references (i.e. to local files, or file internal) intact.

from prance import ResolvingParser
from prance.util.resolver import RESOLVE_HTTP

parser = ResolvingParser('/path/to/spec', resolve_types = RESOLVE_HTTP)

Multiple types can be specified by OR-ing constants together:

from prance import ResolvingParser
from prance.util.resolver import RESOLVE_HTTP, RESOLVE_FILES

parser = ResolvingParser('/path/to/spec', resolve_types = RESOLVE_HTTP | RESOLVE_FILES)

Extensions

Prance includes the ability to reference outside swagger definitions in outside Python packages. Such a package must already be importable (i.e. installed), and be accessible via the ResourceManager API (some more info here).

For example, you might create a package common_swag with the file base.yaml containing the definition

definitions:
  Severity:
    type: string
    enum:
    - INFO
    - WARN
    - ERROR
    - FATAL

In the setup.py for common_swag you would add lines such as

packages=find_packages('src'),
package_dir={'': 'src'},
package_data={
    '': '*.yaml'
}

Then, having installed common_swag into some application, you could now write

definitions:
  Message:
    type: object
    properties:
      severity:
        $ref: 'python://common_swag/base.yaml#/definitions/Severity'
      code:
        type: string
      summary:
        type: string
      description:
        type: string
    required:
    - severity
    - summary

Contributing

See CONTRIBUTING.md for details.

Professional support is available through finkhaeuser consulting.

License

Licensed under MIT. See the LICENSE.txt file for details.

"Prancing unicorn" logo image Copyright (c) Jens Finkhaeuser. Made by Moreven B. Use of the logo is permitted under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.

prance's People

Contributors

a0barth avatar avara1986 avatar cigano avatar dradetsky avatar elemental-lf avatar glutexo avatar goldziher avatar jfinkhaeuser avatar kianmeng avatar lafrech avatar logicplace avatar mboutet avatar meteozond avatar ninoles avatar pre-commit-ci[bot] avatar ronnypfannschmidt avatar samdammers avatar sloria avatar sondrelg avatar tahmidefaz avatar tirkarthi avatar warllama avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

prance's Issues

endless loop when parse recursive $ref

Expected Behaviour

parse with no errors

Minimal Example Spec

components:
  schemas:
    topo_info:
      type: object
      properties:
        id:
          type: string
          description: Topo ID
        name:
          type: string
          description: Topo name
        child_topos:
          type: array
          description: Child topo
          items:
            - $ref: '#/components/schemas/topo_info'

Actual Behaviour

The parser falls to endless loop . I set recursive limit to 100 , it donot work .

File "/api_spec/prance/util/resolver.py", line 175, in _resolve_partial
    recursions)))
  File "/api_spec/prance/util/resolver.py", line 102, in _dereferencing_iterator
    ref_value = self.__reclimit_handler(self.__reclimit, ref_url)
  File "/api_spec/prance/util/resolver.py", line 16, in default_reclimit_handler
    'resolve "%s"!' % (limit, parsed_url.geturl()))
ResolutionError: Recursion reached limit of 100 trying to resolve file 

Steps to Reproduce

Environment

  • OS: linux
  • Python version: 2.7
  • Swagger/OpenAPI version used: 3
  • Backend: openapi-spec-validator

@jfinkhaeuser

CLI tests test_convert_defaults and test_convert_output fail without network

The CLI tests test_convert_defaults and test_convert_output fail without network.

In RPM packaging, all tests which require a network have to be disabled as the RPMs are built in an isolated VM to ensure they are reproducible.

As this two tests are verifying CLI behaviour, it would be a good idea if they didnt rely on the network.

Extend Partial Resolution

I may have be missing something which is already possible, so apologies if this is case.

I'd like to be able to have a reference to a specific part of the 'parent' file become an internal reference when the 'child' file is resolved.

Minimal Example Spec

root/parent.yaml

Child: 
    $ref: './subfolder/child.yaml'
Parent:
    ...

root/subfolder/child.yaml

something:
    $ref: '../parent.yaml#/Parent

Would resolve to:
compiled.yaml

Child: 
    something: 
        $ref '#/Parent'
Parent: 
    ...

Actual Behaviour

I've tried to produce this effect with the partial resolution option, but have not been able to get it to work. Applying only RESOLVE_FILES will result in a crash, as the references isn't actually an internal reference. Running without RESOLVE_FILES will fully resolve the reference.

@jfinkhaeuser

strict=False does not work for referenced files

Expected Behaviour

strict=False should work with ResolvingParser when operations are defined in a separate file.

$ prance validate --no-strict bad_spec.yaml
Processing "bad_spec.yaml"...
 -> Resolving external references.
Validates OK as OpenAPI 3.0.3!

Minimal Example Spec

bad_spec.yaml

openapi: 3.0.3

info:
  title: OpenAPI spec to demonstrate stringify reference issue
  description: |
    This file is references one GET operation.
    and should validate fine with prance using `strict=False` but currently breaks.
  version: '1.0'

paths:
  /bad:
    $ref: 'bad_get.yaml'

bad_get.yaml

get:
  responses:
    200:
      description: Success

Actual Behaviour

Validation fails:

$ prance validate --no-strict bad_spec.yaml
Processing "bad_spec.yaml"...
 -> Resolving external references.
ERROR in "bad_spec.yaml" [ValidationError]: expected string or bytes-like object -- Strict mode enabled (the default), so this could be due to an integer key, such as an HTTP status code.

Steps to Reproduce

Create the files above (bad_spec.yaml and bad_get.yaml) and then run:

prance validate --no-strict bad_spec.yaml

Note that this works fine:

$ openapi-spec-validator bad_spec.yaml
OK

Environment

  • OS: Mac OS X 10.14.6
  • Python version: 3.7.3
  • Swagger/OpenAPI version used: OpenAPI 3.0.3
  • Backend: openapi-spec-validator

@jfinkhaeuser

Swagger Resolution Errors

Expected Behaviour

Minimal Example Spec

Actual Behaviour

Steps to Reproduce

Environment

  • OS: maxOf
  • Python version: 3.9
  • Swagger/OpenAPI version used: swagger 2.0
  • Backend: (flex, swagger-spec-validator or openapi-spec-validator) using the resolver directly

Hi there, and first off thanks for this resolver!

I'm maintaining another library that relies on Prance as a derferencer. It works well in most cases, but I think I encountered a bug. I'm testing our library (https://github.com/snok/drf-openapi-tester) against various specs, and now I'm trying to test it against the kubernetes swgger 2.0 docs:, which can be found here: https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json

It seems that this spec cannot be resolved at present because instead of a url what is passed to the following function is a dictionary {type: string}:

def absurl(url, relative_to = None):
....
# prance.util.url lines 49-59
  parsed = url
  if not isinstance(parsed, tuple):
    from .fs import is_pathname_valid
    if is_pathname_valid(url):
      from . import fs
      url = fs.to_posix(url)
    try:
      parsed = parse.urlparse(url)
    except Exception as ex:
      from .exceptions import raise_from
      raise_from(ResolutionError, ex, 'Unable to parse url: %s' % (url,))

I got to this point in debugging it.

According to the intellij debugger absurl was called with two parameters:

url={'type': 'string'},
relative_to=ParseResult(scheme='file', netloc='', path='/', params='', query='', fragment='/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps')

If you want I can add a json copy of the kubernetes spec to the repo and have it fail in a test if that would help :).

@jfinkhaeuser

Add new compile command

Expected Behaviour

While using the validate command to combine specs by dereferencing is a very nice emergent property of prance, it may be useful making it explicit.

Effectively, using validate --output-file should raise a deprecation warning, and a new compile command should cover the same functionality.

The compile command should also include automatic conversion of 2.0 to 3.0 as per #17 if and only if mixed specs are provided on the command line. Additionally, the command should try to actually merge specs, instead of just overwriting the same file over and over.

'ResolutionError: Cannot resolve reference' when parse $ref which has '~1'

Expected Behaviour

It is expected that the $ref which contains '~1' should be correctly parsed

Minimal Example Spec

a.yaml

paths:
  /api/v2/vms:
    $ref: ./b.yml#/paths/~1api~1v2~1vms
    ..........................
    other thing

b.yaml

paths:
  /api/v2/vms:
    get:
      operationId: list_vm
      description: get all 
      ............................
      other thing

Actual Behaviour

raise ResolutionError: Cannot resolve reference actualy

Steps to Reproduce

  1. parse yaml which has $ref with '~1'

Environment

  • OS: linux
  • Python version: 2.7
  • Swagger/OpenAPI version used: 3.0.0
  • Backend: openapi-spec-validator

@jfinkhaeuser

Prance depends on a now abandoned version of pyyaml

Apologies for not following the template but this is an odd situation that doesn't really fit.

The situation, as far as I can tell.

Quite rightly, in commit a2ba2c8, which went into 0.14.0, you bumped the version of PyYaml to ">4.0.0" which should include the safe secure behaviour for loading yaml. However there's been a fair amount of drama over in pyyaml's repo which you may have missed.

As far as I can tell from this ticket the maintainer got cold feet about the release of 4.x.x as it was a backwards incompatible change to load. They seem to have abandoned the 4.x.x releases up on PyPi.

The new release will be called 5.1 and I'll write up a "PyYAML 5.1 Release Plan" issue when the time is right.

There seems to be some suggesting that people use one of the failed 4.2bx releases to get #74 behavior. This is a bad idea. 3.13 is the current supported release. I could delete the 4.2b-s from PyPI but I haven't. I almost certainly will after 5.1 goes out.

Seems a bit crazy to me, but that's how things stand right now.

My problem and current workaround

This is causing me issues in my project using prance as pipenv is unable to resolve the requirement for a 4.x version of PyYAML as no officially released version of that package exists.

I am able to tell pipenv to consider pre-releases as well, but unfortunately that's a global option meaning that I'll get pre-releases for everything which causes more issues. It'd obviously be better if pipenv was able to more specifically target enable pre-releases (this is tracked in an issue on pipenv).

I am pretty sure I can work around that limitation in pipenv, but even if I do, given the maintainer seems to have abandoned the 4.x release, I'm not sure it's such a good idea.

For the time being I'll probably freeze prance to "<0.14" until the issue is resolved.

Request

That being said, if possible it seems like a good idea to go back to the old PyYaml version for the time being and it'd be awesome to see a new release of prance with that so I can unfreeze again.

Thanks for all the good work!

@jfinkhaeuser

ValidationError: expected string or bytes-like object

I'm creating app with Flask, Connexion and Swagger. I want split swagger yaml file in multilpe files.
I have followed this reply:
spec-first/connexion#254 (comment)

When I execute my app appears the exception. This exception is raised by the function _validate_openapi_spec_validator on class BaseParser, line 235.
validate_v2_spec(self.specification)
specification is a dict.

This is the yaml with the reference:

swagger: "2.0"

info:
  title: "Document Repository Service API"
  version: "1.0"

basePath: /v1.0

paths:
  $ref: ./resolutions.yaml

And this is the yaml referenced:

 /resolutions/{id}:
      get:
        x-swagger-router-controller: controllers.resolution_controller
        operationId: get_resolution
        parameters:
          - name: id
            in: path
            description: Resolution ID
            type: string
            required: true
          - name: format
            in: query
            type: string
            enum: [xml, json]
            required: false
            default: xml
          - name: encoding
            in: query
            type: string
            enum: [utf-8, original]
            required: false
            default: utf-8
        responses:
          200:
            description: OK
          404:
            description: A resolution with the specified ID was not found.

parsing recursive json schema

I am using connexion and prance with openapi and a recursive json schema that looks like:

"information": {
    "type": "object",
    "properties": {
        "object": {
            "type": "array",
            "items": {
                "anyOf": [
                    {
                        "$ref": "#/definitions/object"
                    },
                    {
                        "$ref": "#/definitions/information"
                    }
                ]
            }
        },
        "feature": {
            "type": "array",
            "items": {
                 "$ref": "#/definitions/feature"
            }
        }
    }
}

where the object can be a simple object or a nested information
I have seen issue #55 and tried to set a max recursion limit together with an handle that return None but I get a prance.ValidationError

following is the traceback
traceback (most recent call last):
File "C:\code\test\test_openapi_json_schema_validation\env\lib\site-packages\prance_init_.py", line 224, in _validate_openapi_spec_validator
validate_v3_spec(self.specification)
File "C:\code\test\test_openapi_json_schema_validation\env\lib\site-packages\openapi_spec_validator\shortcuts.py", line 7, in validate
return validator_callable(spec, spec_url=spec_url)
File "C:\code\test\test_openapi_json_schema_validation\env\lib\site-packages\openapi_spec_validator\validators.py", line 48, in validate
raise err
openapi_spec_validator.exceptions.OpenAPIValidationError

and here is the code

def recursion_limit_handler_none(limit, refstring, recursions):
    return None

def get_bundled_specs(main_file: Path) -> Dict[str, Any]:
    parser = prance.ResolvingParser(str(main_file.absolute()),
    lazy=True,
    recursion_limit=1,
    recursion_limit_handler=recursion_limit_handler_none,
    backend='openapi-spec-validator')
    parser.parse()
    return parser.specification

app = connexion.FlaskApp(name)
app.add_api(
    get_bundled_specs(Path("api.yaml")),
    strict_validation=True,
    validate_responses=True,
    resolver=connexion.RestyResolver("cms_rest_api"))

It is not clear to me whether it is a problem due to the recursive json, that might not be supported or I made it wrong, or is due to my poor understanding on issue #55 and how to use it

Any help it is appreciated!!

Error messages do not include context to locate problematic parts in schema

Expected Behaviour

Providing some context which part of my 113k line file failed.

Actual Behaviour

prance validate v1.16.4.json 
Processing "v1.16.4.json"...
 -> Resolving external references.
Traceback (most recent call last):
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/bin/prance", line 8, in <module>
    sys.exit(cli())
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/cli.py", line 108, in command_invoke
    original_invoke(ctx)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/cli.py", line 197, in validate
    __validate(parser, name)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/cli.py", line 59, in __validate
    parser.parse()
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/__init__.py", line 138, in parse
    self._validate()
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/__init__.py", line 294, in _validate
    resolver.resolve_references()
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 96, in resolve_references
    self.specs = self._resolve_partial(self.parsed_url, self.specs, ())
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 207, in _resolve_partial
    changes = dict(tuple(self._dereferencing_iterator(base_url, partial, (),
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 133, in _dereferencing_iterator
    ref_value = self._dereference(ref_url, obj_path, next_recursions)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 190, in _dereference
    value = self._resolve_partial(ref_url, value, recursions)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 207, in _resolve_partial
    changes = dict(tuple(self._dereferencing_iterator(base_url, partial, (),
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 133, in _dereferencing_iterator
    ref_value = self._dereference(ref_url, obj_path, next_recursions)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 190, in _dereference
    value = self._resolve_partial(ref_url, value, recursions)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 207, in _resolve_partial
    changes = dict(tuple(self._dereferencing_iterator(base_url, partial, (),
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 133, in _dereferencing_iterator
    ref_value = self._dereference(ref_url, obj_path, next_recursions)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 190, in _dereference
    value = self._resolve_partial(ref_url, value, recursions)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 207, in _resolve_partial
    changes = dict(tuple(self._dereferencing_iterator(base_url, partial, (),
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 133, in _dereferencing_iterator
    ref_value = self._dereference(ref_url, obj_path, next_recursions)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 190, in _dereference
    value = self._resolve_partial(ref_url, value, recursions)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 207, in _resolve_partial
    changes = dict(tuple(self._dereferencing_iterator(base_url, partial, (),
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/resolver.py", line 112, in _dereferencing_iterator
    ref_url, obj_path = _url.split_url_reference(base_url, refstring)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/url.py", line 120, in split_url_reference
    parsed_url = absurl(reference, base_url)
  File "/home/f.ludwig/.local/share/virtualenvs/pdk8s-oowwi4Ru/lib/python3.8/site-packages/prance/util/url.py", line 56, in absurl
    parsed = parse.urlparse(url)
  File "/usr/lib64/python3.8/urllib/parse.py", line 372, in urlparse
    url, scheme, _coerce_result = _coerce_args(url, scheme)
  File "/usr/lib64/python3.8/urllib/parse.py", line 124, in _coerce_args
    return _decode_args(args) + (_encode_result,)
  File "/usr/lib64/python3.8/urllib/parse.py", line 108, in _decode_args
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
  File "/usr/lib64/python3.8/urllib/parse.py", line 108, in <genexpr>
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'dict' object has no attribute 'decode'

Steps to Reproduce

wget https://raw.githubusercontent.com/kubernetes/kubernetes/v1.16.4/api/openapi-spec/swagger.json
prance validate swagger.json

Environment

  • OS: Linux
  • Python version: 3.8
  • Swagger/OpenAPI version used: 2.0
  • Backend: best

@jfinkhaeuser

Strange 'yaml.reader.ReaderError: unacceptable character #x0082: special caracters are not allowed in "<unicode string>", position 446' in modular openapi definitionh

A have a simple openapi definition prepared to present the problem, with a sinlge Polish letter: ล‚.

openapi.yaml:

openapi: 3.0.2
info:
  title: title
  version: '1.0'
paths:
  /something/:
    post:
      summary: summary
      requestBody:
        content:
          application/json:
            schema:
              type: string
      responses:
        '400':
          description: description
          content:
            application/json:
              schema:
                type: object
                properties:
                  info:
                    type: array
                    items:
                      type: string
                      enum:
                        - ล‚

That snippet compiles as expected:

% prance compile --strict openapi.yaml all-in-one-openapi.yaml
Processing "openapi.yaml"...
 -> Resolving external references.
Validates OK as OpenAPI 3.0.2!
Output written to "all-in-one-openapi.yaml".

But I wanted to spit this definition into parts. So I prepared two another files:

openapi-main.yaml:

openapi: 3.0.2
info:
  title: title
  version: '1.0'
paths:
  /something/:
    $ref: 'openapi-part.yaml'

openapi-part.yaml:

---
post:
  summary: summary
  requestBody:
    content:
      application/json:
        schema:
          type: string
  responses:
    '400':
      description: description
      content:
        application/json:
          schema:
            type: object
            properties:
              info:
                type: array
                items:
                  type: string
                  enum:
                    - ล‚

And now my problem arises:

% prance compile --no-strict openapi-main.yaml all-in-one-openapi.yaml

Processing "openapi-main.yaml"...
 -> Resolving external references.
Traceback (most recent call last):
  File "/home/xxx/.virtualenvs/prance/bin/prance", line 8, in <module>
    sys.exit(cli())
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/cli.py", line 103, in command_invoke
    original_invoke(ctx)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/cli.py", line 220, in compile
    __validate(parser, name)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/cli.py", line 54, in __validate
    parser.parse()
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/__init__.py", line 134, in parse
    self._validate()
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/__init__.py", line 271, in _validate
    resolver.resolve_references()
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/resolver.py", line 79, in resolve_references
    self.specs = self._resolve_partial(self.parsed_url, self.specs, ())
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/resolver.py", line 175, in _resolve_partial
    recursions)))
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/resolver.py", line 117, in _dereferencing_iterator
    ref_value = self._dereference(ref_url, obj_path, next_recursions)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/resolver.py", line 139, in _dereference
    contents = _url.fetch_url(ref_url, self.__reference_cache)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/url.py", line 215, in fetch_url
    result = parse_spec(content, url.path, content_type = content_type)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/formats.py", line 205, in parse_spec
    result, ctype, ext = parse_spec_details(spec_str, filename, **kwargs)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/formats.py", line 180, in parse_spec_details
    result = parser(spec_str)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/prance/util/formats.py", line 67, in __parse_yaml
    return yaml.safe_load(six.text_type(spec_str))
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/yaml/__init__.py", line 112, in load
    loader = Loader(stream)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/yaml/loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/yaml/reader.py", line 74, in __init__
    self.check_printable(stream)
  File "/home/xxx/.virtualenvs/prance/lib/python3.7/site-packages/yaml/reader.py", line 144, in check_printable
    'unicode', "special characters are not allowed")
yaml.reader.ReaderError: unacceptable character #x0082: special characters are not allowed
  in "<unicode string>", position 432

In this case I can do some workarounds. Eg. remove three dashes (---) from the beginning of openapi-part.yaml file, which are not obligatory in such a situation, but it desnโ€™t work in all my real cases. Itโ€™s also really amazing that I can leave those dashes in the file and remove its third line (summary:) and it works fine, as well.

I can't discover any rules in this behavior.

I use prance==0.16.2 with python 3.7.5.

Syntax warning over comparison of literals using is

Steps to Reproduce

find . -iname '*.py' | grep -Ev 'setup|rdf4|tool' | xargs -P4 -I{} python3.8 -Wall -m py_compile {}
./tests/test_util_path.py:56: SyntaxWarning: "is" with a literal. Did you mean "=="?
  assert result is 666

Environment

  • OS: Ubuntu
  • Python version: 3.8

@jfinkhaeuser

Crash when response code is integer

Hi!
First of thank you for this project - it is exactly what I need!

However, I discovered a bug that occurred when the response code of a method in the Swagger file is interpreted as an integer.
The error is expected string or bytes-like object.

Example:

swagger: '2.0'
info:
  title: Test API
  version: "0.1"
produces:
  - application/json
basePath: /api/v1
paths:
  /test:
    post:
      operationId: test_api.api.fromurl.post
      responses:
        200: #<---
          description: Returns something

If I use "200" instead, it works.

I discovered that the bug actually happens in jsonschema._utils, where a regex fails to run over the integer. Strangely, other packages that I previously used to interpret Swagger (like connexion) also depend on the jsonschema package and they work fine.

I tested this on Python 3.6.1 on OS X 10.11.6. The version of jsonschema is 2.6.0 and of prance 0.5.1

I would really appreciate it, if you could investigate this bug!

Try to use parser context before passing on to validators, to enrich error messages.

As stated in #69, it would be great if we could get some more context into error messages. One way way it might work is if we get file/line information from the YAML/JSON parsers and annotate the specs with something that the validators can ignore, then use those annotations in error messages.

With pyyaml, we should be able to get this information out of buffers. With JSON, it might be trickier at first glance - worst case we have to use some dependency.

Suggestions are welcome. This issue is more of a reminder to myself to find what we can do here.

@jfinkhaeuser

Dependancy on click should be optional.

Expected Behaviour

I can use this as a library only with no dependency on click

Actual Behaviour

My CLI (which uses Click 7.0) is unable to use this as a library as there is a hard dependency for Click 6.7. It be nice if this could be used as a library only without any CLI specific dependencies (and not requiring a fork).

@jfinkhaeuser

Configuration for "partial" resolution

Expected Behaviour

I wish a parameter for a partial resolution of items, eg:

  • append external references to components
  • retain inner references to #/components/schemas/...

Minimal Example Spec

...
schemas:
  A:
    properties:
      a: string
      b:
         $ref: https://foo.com/#B         
...

should become

...
schemas:
  A:
    a: string
    b: 
        $ref: '#/components/schemas/B'
  B:
    <all b stuff>
...

Actual Behaviour

Full resolution is done

@jfinkhaeuser

Convert OpenAPI 2.0 to 3.0

Expected Behaviour

A new command should convert OpenAPI 2.0 specs to 3.0, as well as possible. In code, this should be functionality to run on a parser's spec, so we can use it on the BaseParser prior to resolving references.

Tests would require that converted specs validate as 3.0.

Relative file references are not resolved outside of definitions

Within my JSON I have a $ref to a relative file path but when parsing get the error:

Exception caught: (<class 'swagger_spec_validator.common.SwaggerValidationError'>)

Exception message: unknown url type: ./schema/ChangePasswordEntity.json

Extract from swagger.json

{
    "in": "body",
    "name": "ChangePasswordEntity",
    "required": true,
    "schema": {
        "$ref": "./schema/ChangePasswordEntity.json"
    }
}

Code:

from prance import ResolvingParser
from prance.util import fs, formats

...

parser = ResolvingParser(file)
contents = formats.serialize_spec(parser.specification, filename)
fs.write_file(filename, contents)

Internal references dereferenced in external files

This is more of an question on whether is this even possible. It applies to a state when resolve_types does not include RESOLVE_INTERNAL. The motivation for this is that it is not possible to have recursive schema definitions in referenced local files. Like:

{
  "$defs": {
    "NestedObject": {
      "propertyNames": {
        "minLength": 1
      },
      "additionalProperties": {
        "$ref": "#/$defs/NestedObject"
      }
    }
  }
}

Expected Behaviour

Local references in external files are kept local. They are injected to the main specification and converted to the new path.

Minimal Example Spec

  • OpenAPI specification
    {
      "openapi": "3.0.0",
      "info": {
        "title": "Test API",
        "version": "1"
      },
      "paths": {
        "/endpoint": {
          "post": {
            "requestBody": {
              "content": {
                "application/json": {
                  "schema": {"$ref": "_schemas.json#/$defs/Parameter"}
                }
              }
            },
            "responses": {}
          }
        }
      }
    }

External file with schema definitions.

   {
     "$id": "_schemas.json",
     "$schemas": "https://json-schema.org/draft/2019-09/schema#",
     "$defs": {
       "Something": {
         "type": "object"
       },
       "Parameter": {
         "properties": {
           "something": {
             "$ref": "#/$defs/Something"
           }
         }
       }
     }
   }

Actual Behaviour

Local references in external files are dereferenced. But how? Theyโ€™d need to be somewhere into the composed specification. Something like:

{
  ...
  "paths": {
   "/endpoint": {
     "post": {
       "requestBody": {
         "content": {
           "application/json": {
             "schema": {"$ref": "#/x-prance-refs/_schemas.json%23%2F%24defs%2FSomething"}
           }
         }
       },
       ...
     }
   }
  },
  "x-prance-refs": {
      "_schemas.json#/$defs/Something": {
        "type": "object"
      }
   }
}

Steps to Reproduce

Here is a simple script exposing what is dereferenced:

from prance import ResolvingParser
from prance.util.resolver import RESOLVE_FILES


def _main():
    parser = ResolvingParser("_openapi.json", resolve_types=RESOLVE_FILES)
    parser.parse()
    print(parser.specification["paths"]["/endpoint"]["post"]["requestBody"]["content"]["application/json"]["schema"])


if __name__ == "__main__":
    _main()

Environment

  • Swagger/OpenAPI version used: 3.0.0
  • Backend: openapi-spec-validator

@jfinkhaeuser

Support chardet v4

chardet v4 is finally out, and the setup.py chardet~=3.0 blocks co-habitation with it.

Option to unpack "allOf" arrays in swagger

I have a swagger file, from which I would like to extract a JSON schema object for the response to each API endpoint. JSON schema doesn't like the allOf magic we've used, so it would be nice if this can be automatically expanded out with an option to prance.

I've half hacked together a script to do this for our case, but I feel like this feature ought to sit in prance itself. What do you think? Is this difficult to implement? (I haven't read the source yet)

prance is too slow, probably needs caching

I'm trying to enable breaking up yaml swagger specs into smaller files in a set of connexion projects. Currently, connexion does not support the use of relative file references (see issue here). So I investigated prance as a solution.

Technically, it works, in the sense that it enables us to build a fully-resolved spec from a separated spec by resolving the references. However, it is very slow. Ideally, we would like to just load and resolve the swagger files when the application boots, which is normally instantaneous. One project, which has the following swagger:

  • 2 files
  • 1140 lines of swagger (including blanks, comments, redundant defs b/c we haven't got multifile resolve working, etc)
  • 186 total references (a few are probably commented out)
  • 55 unique references (same again)

takes about 20 seconds to resolve the spec on my fancy, expensive laptop. Obviously, this is unacceptable for anything but a way to "compile" a spec which is just loaded each time you boot. Also, it's just plain weirdly slow.

Profiling ResolvingParser, I see 3097572 calls (30,124,069 entries total) to dpath.path.paths, which seems like a lot. Poking around a bit in prance.util.resolver.RefResolver with my debugger, it looks to me like the following is going on:

  1. Every time we come across the reference path/to/file#/definitions/Something we do some processing and then walk a doc tree, even though this is the same value every time.
  2. In the resolve_references changes loop, the duplication mentioned in the comment is substantial; 659 pairs vs 174 unique pairs. In some cases, a particular key is set 100 times.

The ratio of wall time spent before the biggest changes loop to after the biggest changes loop is about 1:10, so (1) may not be the most pressing issue, but it's still a good couple of seconds, and it may be possible to cut that to milliseconds with some caching. The big problem is changes loop, where deduping the changes list is more significant than the comment makes out.

Fortunately, it's pretty easy to improve this substantially by looping over uniq_changes = dict(changes). Tests pass, although I haven't looked at the test suite to determine whether this actually tells us that nothing has gone wrong. It's still kind of slow, but I'd have to think much harder about whether all the dutil.set ops can be speeded up in any way, and if so whether this is feasible. For example, it looks like dutil.set is designed to operate in batches, so maybe all the ops which set various path keys to a particular resolved reference could be batched into a single operation. That way we do one dutil.set for all the references to some/path#/definitions/Foo. I don't know enough about dutil.set to know whether this would mean a nontrivial speedup, a trivial speedup, or a slowdown.

Anyway, questions:

  • Would you accept PRs to improve this stuff, or is high-performance a non-goal for the project?
  • Is there any chance of you addressing any of these things yourself? I doubt I'm going to have time to do them properly any time soon.

Option to support recursion

Swagger spec supports recursion, and schemas that have recursive references are perfectly valid โ€” no python library seems to support this though. RefResolver has a built-in __recursion_protection, but it protects against all recursion.

Would it be possible to allow it to add an option for recursion depth limit instead, and allow recursion but only until a certain depth limit is reached, and fail silently with a warning?

Add functionality for validating data against models

Expected Behaviour

Given data (as YAML or JSON, or in code as nested structures), validate it against schemas in a spec. Well, basically against anything that can live in the components object (3.0)

Environment

  • Swagger/OpenAPI version used: 3.0
  • Backend: any

@jfinkhaeuser

Prance is not reading ref for encoded url

Expected Behaviour

Swagger reads ref file with encoded url but prance through errr file not found error.

Minimal Example Spec

$ref:'#/paths/1api1%7Bname%7D~1data

Actual Behaviour

Error:
Cannot resolve reference "file:///

/paths/1api1%7Bname%7D~1data

Steps to Reproduce

Environment

  • OS:
  • Python version:
  • Swagger/OpenAPI version used:
  • Backend: (flex, swagger-spec-validator or openapi-spec-validator)

@jfinkhaeuser

Minor readme issue

Hey, @jfinkhaeuser, you probably using compile but should use convert in a readme.

Expected Behaviour

# Convert spec
$ prance convert path/to/swagger.yml path/to/openapi.yml

Actual Behaviour

# Convert spec
$ prance compile path/to/swagger.yml path/to/openapi.yml

Wrong validation messages because of version fallback with openapi 3.0

Currently this code:
``

from .util.exceptions import raise_from
try:
  try:
    validate_v3_spec(self.specification)
    self.__set_version(BaseParser.SPEC_VERSION_3_PREFIX, spec_version)
  except TypeError as type_ex:  # pragma: nocover
    raise_from(ValidationError, type_ex)
  except JSEValidationError as v3_ex:
    try:
      validate_v2_spec(self.specification)
      self.__set_version(BaseParser.SPEC_VERSION_2_PREFIX, spec_version)
    except TypeError as type_ex:  # pragma: nocover
      raise_from(ValidationError, type_ex)
except RefResolutionError as ref_ex:
  raise_from(ValidationError, ref_ex)

``

Produces wrong validation messages when there is an error in openapi schema 3.0.0
It returns openapi 2.0.0 messages instead of 3.0.0. After removing fallback correct error messages are shown.
It should use fallback only when openapi version was not specified. I could produce patch, but I don't know what is your preferred way to solve this problem. This fallback is for a reason there?

Environment

  • OS: Linux
  • Python version: 3.6
  • Swagger/OpenAPI version used: 3.0.0
  • Backend: openapi-spec-validator

@jfinkhaeuser

deprecation warning about collections

Expected Behaviour

No deprecation warnings produced when prance is used

Minimal Example Spec

I use Prance in my OpenAPI / Connexion app like so

def get_bundled_specs(main_file):
    parser = prance.ResolvingParser(str(main_file.absolute()), lazy=True, strict=True)
    parser.parse()
    return parser.specification

Actual Behaviour

This deprecation warning is output:

<snip>/.venv/lib/python3.7/site-packages/prance/util/path.py:122: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    if not isinstance(obj, collections.MutableMapping):  # pragma: nocover

Steps to Reproduce

Simply run the app.

Environment

  • OS: macOS Mojave
  • Python version: 3.7
  • Swagger/OpenAPI version used: 3
  • Backend: (flex, swagger-spec-validator or openapi-spec-validator) - no backend specified

@jfinkhaeuser

Resolve given URL to endpoint

Expected Behaviour

I would like to find the endpoint in my spec file according to a given URL.

Given /networks/my-net/rbac, I would like prance to give me the spec for the endpoint listed under the minimal example.

Minimal Example Spec

paths:
  /networks/{network}/rbac:

Actual Behaviour

I am not aware of a feature in prance that would allow this.

Steps to Reproduce

Environment

  • OS: macOS
  • Python version: 3.8.5
  • Swagger/OpenAPI version used: 3.0.2
  • Backend: (flex, swagger-spec-validator or openapi-spec-validator)

@jfinkhaeuser

prance compile fails with `KeyError: 'Cannot set with an empty path!'` when resolving external paths

Expected Behaviour

I expect:

  • Prance to compile referenced objects {$ref: "http://foo.com/bar.yaml#/Object"}

Minimal Example Spec

https://github.com/teamdigitale/api-starter-kit/blob/master/openapi/simple.yaml.src

Actual Behaviour

Processing "https://gist.githubusercontent.com/ioggstream/c0adc0e37f3a35b2fc97c5d5238ebdda/raw/91c353b5432cbba61cbb479f82f7d835c99df3be/simple.yml"...
 -> Resolving external references.
Traceback (most recent call last):
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/bin/prance", line 11, in <module>
    sys.exit(cli())
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/cli.py", line 103, in command_invoke
    original_invoke(ctx)
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/cli.py", line 182, in validate
    __validate(parser, name)
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/cli.py", line 54, in __validate
    parser.parse()
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/__init__.py", line 133, in parse
    self._validate()
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/__init__.py", line 267, in _validate
    resolver.resolve_references()
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/resolver.py", line 72, in resolve_references
    self.specs = self._resolve_partial(self.parsed_url, self.specs, ())
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/resolver.py", line 164, in _resolve_partial
    recursions)))
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/resolver.py", line 106, in _dereferencing_iterator
    ref_value = self._dereference(ref_url, obj_path, next_recursions)
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/resolver.py", line 146, in _dereference
    value = self._resolve_partial(ref_url, value, recursions)
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/resolver.py", line 171, in _resolve_partial
    path_set(partial, list(path), value, create = True)
  File "/home/rpolli/workspace-py/api-starter-kit/.tox/py36/lib/python3.6/site-packages/prance/util/path.py", line 116, in path_set
    raise KeyError('Cannot set with an empty path!')
KeyError: 'Cannot set with an empty path!'

Steps to Reproduce

prance compile --backend=openapi-spec-validator https://gist.githubusercontent.com/ioggstream/c0adc0e37f3a35b2fc97c5d5238ebdda/raw/91c353b5432cbba61cbb479f82f7d835c99df3be/simple.yml

Environment

  • OS: linux fedora 27
  • Python version: 3.6
  • Swagger/OpenAPI version used: 3.0.0
  • Backend: openapi-spec-validator

Note

This code can resolve references (but does not validate it). Maybe there's something you can reuse.
https://github.com/teamdigitale/api-starter-kit/blob/master/scripts/openapi_resolver.py

@jfinkhaeuser

fetch_url always caches resolved URLs

The util.url.fetch_url and util.url.fetch_url_text functions are supposed to have an optional cache ("Mapping cache: An optional cache. If the URL can be found in the cache, return the cache contents.", as per the documentation)

However, in the current implementation there will always be a cache, which I believe was not the intention.

The root cause is using {} as default for the cache argument; rather than re-instantiating a new dictionary every time the function is called, this re-uses the same dictionary between function calls.

See e.g. https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments for details

A simple fix would be:

def fetch_url(url, cache = None, encoding = None):
  cache = cache or {}

Expected Behaviour

URL contents should not be cached unless the cache argument is provided.

Minimal Example Spec

Actual Behaviour

URL contents get cached if no cache argument is provided.

Steps to Reproduce

Given a file test.yaml with the following contents:

openapi: 3.0.2
info:
  title: Test
  version: '0.0.1'

and the following program:

import os
from prance.util.url import absurl
from prance.util.fs import abspath
from prance.util.url import fetch_url

url = absurl("test.yaml", abspath(os.getcwd()))
fetch_url(url)

we get the following output:

{'openapi': '3.0.2', 'info': {'title': 'Test', 'version': '0.0.1'}}

Change the file to e.g.

openapi: 3.0.2
info:
  title: Foobar
  version: '0.0.1'

and then re-run the fetch_url(url) part. I would expect the output to change, but you still get:

{'openapi': '3.0.2', 'info': {'title': 'Test', 'version': '0.0.1'}}

Environment

  • OS: MacOS 10.15.7
  • Python version: 3.8
  • Swagger/OpenAPI version used: 3.02
  • Backend: openapi-spec-validator

@jfinkhaeuser

Are local references not suppose to be resolved?

Expected Behaviour

Before I work on PR wanted to ask...

Local # references expanded. Instead of this 'paginator': {'$ref': '#/definitions/Paginator'} I'd expect the {ref} dict to be replaced with the actual definition. Or am I just not understanding how this is suppose to work?

Minimal Example Spec

Actual Behaviour

ERROR in "spec.yaml" [ResolutionError]: Recursion reached limit of 1 trying to resolve "file:///spect.yaml#/definitions/V1Fooer"!

Steps to Reproduce

Environment

  • OS: Apple/BSD
  • Python version: 3.7.2
  • Swagger/OpenAPI version used: 2.0
  • Backend: (flex, swagger-spec-validator or openapi-spec-validator) flex

@jfinkhaeuser

If no ICU is present, use chardet

Originally we replaced chardet with ICU because chardet had some issues detecting UTF-8. But in some environments, (e.g. google cloud engine), ICU might not be available, so chardet is still a better choice than nothing at all.

Incorrect type of string with example in schema

In YAML you can write strings without quotes. For some specific cases you are forced to use quotes, for example when you put digits, but you want them to be of type string.
But when you write schema and set type of variable to string - when you provide example you expect that it will also be of type string. But it's converted to the python as integer.

Expected Behaviour

Some schema in requestBody for example:

  type: object
      properties:
          some_variable:
             type: string
             example: 1111111111 # This should be string
      required: true

This should be:

{'some_variable':'1111111111'}

Actual Behaviour

  type: object
      properties:
          some_variable:
             type: string
             example: 1111111111
      required: true

Result:

{'some_variable': 1111111111}

@jfinkhaeuser

ResolvingParser dereferencing local references with RESOLVE_FILES option.

Expected Behaviour

ResolvingParser won't dereference local references when resolve_types option is set to prance.util.resolver.RESOLVE_FILES.

Minimal Example Spec

openapi: "3.0.0"
info:
  title: ''
  version: '1.0.0'
paths: {}
components:
    schemas:
        SampleArray:
            type: array
            items:
              $ref: '#/components/schemas/ItemType'

        ItemType:
          type: integer

Actual Behaviour

ResolvingParser resolves internal references.

Steps to Reproduce

Assume that minimal spec file is in D:/test.yaml.

import prance
import prance.util.resolver
parser = prance.ResolvingParser(url='D:\\test.yaml', resolve_types=prance.util.resolver.RESOLVE_FILES)
parser.specification
{'openapi': '3.0.0', 'info': {'title': '', 'version': '1.0.0'}, 'paths': {}, 'components': {'schemas': {'SampleArray': {'type': 'array', 'items': {'type': 'integer'}}, 'ItemType': {'type': 'integer'}}}}

Environment

  • OS: Any
  • Python version: 3.7 / 3.8
  • Swagger/OpenAPI version used: OpenAPI
  • Backend: openapi-spec-validator

What I managed to find

I guess, problem is here. Prance trying to compare str and ParseResult objects.

@jfinkhaeuser

Bug in how RESOLVE_INTERNAL is handled

Expected Behaviour

If RESOLVE_INTERNAL is unset, internal references within an included file should not be resolved. Discovered while investigating #77

Minimal Example Spec

{
  "openapi": "3.0.0",
  "info": {
    "title": "Test API",
    "version": "1"
  },
  "paths": {
    "/endpoint": {
      "post": {
        "requestBody": {
           "$ref": "_schemas.json#/$defs/Body"
        },
        "responses": {}
      }
    }
  }
}

And _schemas.json:

{
  "$id": "_schemas.json",
  "$schemas": "https://json-schema.org/draft/2019-09/schema#",
  "$defs": {
    "Something": {
      "type": "object"
    },
    "Body": {
      "content": {
        "application/json": {
          "schema": {
            "$ref": "#/$defs/Something"
          }
        }
      }
    }
  }
}

Actual Behaviour

Internal reference to $defs/Something is resolved.

Steps to Reproduce, etc.

See #77

@jfinkhaeuser

Dependency on requests is too restrictive

Are there any reasons to require the very specific version of this library?

I just ran into a situation where I cannot use prance together with docker-compose because the later outlaws both the 2.21 and 2.18 (I believe they've encountered bugs in those versions). This is while your package requires specifically those versions (the 2.18 is the one before the present requirement).

I don't really see in your project any usage of requests that would have worked differently with older versions, why did you choose to require this particular version?

Validation doesn't indicate error line in file when response code is not a string

Expected Behaviour

Given a YAML file which paths.{path}.get.responses has a response code expressed as a number,
two things could happen:

  • The validation runs ok (Connexion accepts response codes as integers;
  • The validation fails, pointing the correct line that the error happened.

Minimal Example Spec

Actual Behaviour

I have a message like this:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "server.py", line 21, in <module>
    app.add_api(get_bundled_specs(Path('specifications/openapi.yaml')))
  File "server.py", line 13, in get_bundled_specs
    parser.parse()
  File "/usr/local/lib/python3.5/dist-packages/prance/__init__.py", line 134, in parse
    self._validate()
  File "/usr/local/lib/python3.5/dist-packages/prance/__init__.py", line 275, in _validate
    BaseParser._validate(self)
  File "/usr/local/lib/python3.5/dist-packages/prance/__init__.py", line 172, in _validate
    validator(parsed)
  File "/usr/local/lib/python3.5/dist-packages/prance/__init__.py", line 225, in _validate_openapi_spec_validator
    raise_from(ValidationError, type_ex)
  File "<string>", line 5, in raise_from
prance.ValidationError: expected string or bytes-like object

This message is very misleading. For a long file, it took me a considerable amount of time to find the problem.

Steps to Reproduce

Create a YAML file like this:

openapi: 3.0.0
info:
  description: My API
  version: "1.0.0"
  title: My Swagger API

paths:
  /people:
    get:
      operationId: controllers.people.read_all
      tags:
        - people
      summary: Retrieves people
      description: Retrieves people
      parameters:
        - name: length
          in: query
          schema:
            type: integer
          description: Number of people to get from database
          required: false
        - name: offset
          in: query
          schema:
            type: integer
          description: Offset from beginning of list where to start gathering people
          required: false
      responses:
        200:
          description: Successful

Make sure the response code is integer, not string.

Environment

  • OS: Ubuntu 14.06
  • Python version: 3.7
  • Swagger/OpenAPI version used: 3.0
  • Backend: (flex, swagger-spec-validator or openapi-spec-validator): openapi-spec-validator

@jfinkhaeuser

Consider creating a changelog

Consider creating a changelog so that it would be possible to easily determine roughly which changes have been made between the released versions.

"This value is required" errors in 0.10.0

I had to rollback to 0.9.0 as the new release doesn't parse my swagger. The errors look like:

ipdb> prance.ResolvingParser(filepath)
*** prance.SwaggerValidationError: 'paths':
    - '/person/alias':
        - 'post':
            - 'parameters':
                - 'referenceObject':
                    - 'required':
                        - '$ref':
                            - 'This value is required'
                - 'parameterObject':
                    - 'required':
                        - 'name':
                            - 'This value is required'
                        - 'in':
                            - 'This value is required'

I can attempt to create a small test case for this if necessary, but I won't get to it for a week or so.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.