gazpachoking / jsonref Goto Github PK

View Code? Open in Web Editor NEW

115.0 3.0 26.0 184 KB

Automatically dereferences JSON references within a JSON document.

Home Page: http://jsonref.readthedocs.org

License: MIT License

Python 100.00%

hacktoberfest

jsonref's Introduction

jsonref

jsonref is a library for automatic dereferencing of JSON Reference objects for Python (supporting Python 3.7+).

This library lets you use a data structure with JSON reference objects, as if the references had been replaced with the referent data.

>>> from pprint import pprint
>>> import jsonref

>>> # An example json document
>>> json_str = """{"real": [1, 2, 3, 4], "ref": {"$ref": "#/real"}}"""
>>> data = jsonref.loads(json_str)
>>> pprint(data)  # Reference is not evaluated until here
{'real': [1, 2, 3, 4], 'ref': [1, 2, 3, 4]}

Features

References are (optionally) evaluated lazily. Nothing is dereferenced until it is used.
Recursive references are supported, and create recursive python data structures.

References objects are replaced by lazy lookup proxy objects to support lazy dereferencing. They are almost completely transparent.

>>> data = jsonref.loads('{"real": [1, 2, 3, 4], "ref": {"$ref": "#/real"}}')
>>> # You can tell it is a proxy by using the type function
>>> type(data["real"]), type(data["ref"])
(<class 'list'>, <class 'jsonref.JsonRef'>)
>>> # You have direct access to the referent data with the __subject__
>>> # attribute
>>> type(data["ref"].__subject__)
<class 'list'>
>>> # If you need to get at the reference object
>>> data["ref"].__reference__
{'$ref': '#/real'}
>>> # Other than that you can use the proxy just like the underlying object
>>> ref = data["ref"]
>>> isinstance(ref, list)
True
>>> data["real"] == ref
True
>>> ref.append(5)
>>> del ref[0]
>>> # Actions on the reference affect the real data (if it is mutable)
>>> pprint(data)
{'real': [2, 3, 4, 5], 'ref': [2, 3, 4, 5]}

jsonref's People

Stargazers

Watchers

jsonref's Issues

error handling

Should errors from dereferencing be wrapped with a common error? What should happen to the data structure when there are errors?

urllib dereferencing

Use urllib as a fallback in default dereferencer for http when requests is not available, and to support other schemes.

loop checking

Reference loops (without content) should be detected and aborted early.

Also investigate the possibility of print/repr of a structure containing a loop without failing.

For Large Complex Schema Loads fails to return valid object

Bug.zip

When an attempt is made to run jsonref on a simple schema, there is success.

However, a more complext scheam (>1500 lines) it returns a dict, not a jsonref object on load.

On print an execption is thrown.

The sample has both simple and complex case

Sample output

Loaded complex schema
data1 <class 'dict'>
data2 <class 'jsonref.JsonRef'>
 data1: {'real': [1, 2, 3, 4], 'ref': [1, 2, 3, 4]}
Traceback (most recent call last):
  File "/var/data/smontsaroff/wrk/boatnet-importer/venv/lib64/python3.6/site-packages/proxytypes.py", line 252, in __subject__
    return self.cache
  File "/var/data/smontsaroff/wrk/boatnet-importer/venv/lib64/python3.6/site-packages/proxytypes.py", line 176, in __getattribute__
    return _oga(self, attr)
AttributeError: 'JsonRef' object has no attribute 'cache'

30GB of RAM used then SIGKILL

$ git clone --depth=1 https://github.com/stripe/openapi stripe-openapi
$ python main.py

Where main.py is:

#!/usr/bin/env python

from os import path
from json import load, dump

from jsonref import replace_refs

with open(path.join("stripe-openapi", "openapi", "spec3.sdk.json"), "rt") as f:
    doc = load(f)

derefed = replace_refs(doc)
with open('derefed_spec3.sdk.json', 'wt') as f:
    dump(f, derefed, indent=4)

I've got a new laptop with 32GB of RAM and a AMD Ryzen™ 9 6900HX with Radeon™ Graphics × 16, it is all used by this code and then SIGKILL is sent and game over. Ubuntu 22.10 on an SSD.

Useful error reporting suppressed by less useful errors

I'm trying to get JsonRef to work and getting a traceback which I'll post below this comment. Trying to make sense of it, it appears that half-way through, an error is trying to be thrown with the message "Unresolvable JSON Pointer. . . " from line 197 of jsonref.py, but then some other error is raised and this error never gets reported.

This makes it hard to debug, or at the very least, it is a bit of a tease.

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/proxytypes.py", line 91, in wrapper
    return method(self, *args, **kwargs)
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/jsonref.py", line 212, in __repr__
    return repr(self.__subject__)
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/proxytypes.py", line 131, in __getattribute__
    return _oga(self, attr)
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/proxytypes.py", line 91, in wrapper
    return method(self, *args, **kwargs)
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/proxytypes.py", line 209, in __subject__
    self.cache = super(LazyProxy, self).__subject__
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/proxytypes.py", line 91, in wrapper
    return method(self, *args, **kwargs)
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/proxytypes.py", line 195, in __subject__
    return self.callback()
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/proxytypes.py", line 91, in wrapper
    return method(self, *args, **kwargs)
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/jsonref.py", line 170, in callback
    result = self.resolve_pointer(base_doc, fragment)
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/proxytypes.py", line 91, in wrapper
    return method(self, *args, **kwargs)
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/jsonref.py", line 197, in resolve_pointer
    self._error("Unresolvable JSON pointer: %r" % pointer, cause=e)
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/proxytypes.py", line 91, in wrapper
    return method(self, *args, **kwargs)
  File "/Users/erik/.virtualenvs/cr2/lib/python2.7/site-packages/jsonref.py", line 207, in _error
    cause=cause
JsonRefError

The JsonRefError is followed by a portion of the JSON object containing the refs I'm trying to resolve.

Incorrect poetry-core constraint

The pyproject.toml uses [tool.poetry.group.dev.dependencies] which is only available in poetry-core 1.2.0 or newer, which will cause errors like this one when using poetry 1.1.14 (which is default on NixOS 22.05)

We would get a better error if we constrained poetry-core to 1.2.0 or newer.

Processing /build/jsonref-1.0.1
  Running command Preparing metadata (pyproject.toml)
  Traceback (most recent call last):
    File "/nix/store/x2s1aqzhk6fkxv9a6fq50y81sib2avl4-python3.8-pip-22.0.4/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/nix/store/x2s1aqzhk6fkxv9a6fq50y81sib2avl4-python3.8-pip-22.0.4/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/nix/store/x2s1aqzhk6fkxv9a6fq50y81sib2avl4-python3.8-pip-22.0.4/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "/nix/store/j0fbyy5lgc58xxvwg2miv24ib8jpzha4-python3.8-poetry-core-1.0.8/lib/python3.8/site-packages/poetry/core/masonry/api.py", line 43, in prepare_metadata_for_build_wheel
      poetry = Factory().create_poetry(Path(".").resolve(), with_dev=False)
    File "/nix/store/j0fbyy5lgc58xxvwg2miv24ib8jpzha4-python3.8-poetry-core-1.0.8/lib/python3.8/site-packages/poetry/core/factory.py", line 43, in create_poetry
      raise RuntimeError("The Poetry configuration is invalid:\n" + message)
  RuntimeError: The Poetry configuration is invalid:
    - Additional properties are not allowed ('group' was unexpected)

Use `jsonref/` and directory for the source code

When installing this package, all files are put into site-packages/ instead of site-packages/jsonref:

This results in the site-packages/ directory being cluttered. I don't know how pip would resolve it, if another packages has the same filenames.

I can do a tiny PR for that if you want.

Best,
Moritz

jsonref removes existing key-values if reference is present

Given the schema:

{
  "foo": {
    "$ref": "#/def/url",
    "title": "something additional that should not be removed"
  },
  "def": {
      "url": {
        "pattern": "pattern",
        "type": "string"
      }
  }
}

When loading it with jsonref.loads, the key title is removed:

import jsonref

loaded = jsonref.loads('''
{
  "foo": {
    "$ref": "#/def/url",
    "title": "something additional that should not be removed"
  },
  "def": {
      "url": {
        "pattern": "pattern",
        "type": "string"
      }
  }
}
''')
loaded

Outputs:

{'foo': {'pattern': 'pattern', 'type': 'string'},
 'def': {'url': {'pattern': 'pattern', 'type': 'string'}}}

But I'd expect

{'foo': {'pattern': 'pattern', 'type': 'string', 'title': 'something additional that should not be removed'},
 'def': {'url': {'pattern': 'pattern', 'type': 'string'}}}

Is this expected?

Thanks!

replace_ref() loses additional fields

With a schema similar to this:

"myProps": {
       "description": "Properties unique to my networks",
       "type": "object",
       "properties": {
              "freqInfo": {
                   "$ref": "../types/freqInfo.json",
                   "additionalNote": "frequency"
              }
       }
}```

with `replace_ref()`, the `$ref` is correctly resolved, but the additional field `additionalNote` is
discarded, which should have been remained in the resolved schema.

OAuth2 support

Hi,

it'd be great if your library would support OAuth2 auth flow for specific URLs, since all our $refs are secured by that method. For example, we could provide client credentials (clientid and client secret + token endpoint) via ENV and would then expect that the library requests a bearer token and send it along in the header for the respective requests to our selected referred URLs.

Add CI test for Python3.12

Currently, it only tests until 3.11. I can do a tiny PR if you want.

Best,
Moritz

How to save to regular JSON without ref?

Hey,

I'm trying to convert JSON from a file with ref into a file without ref. Is it any way to do so?

_replace_refs should use mapping and sequence type information from inbound object

I would like to defer some steps, e.g. deserialization of a complex object, until object access time.
This could be accomplished by setting json.load(*, cls=) and returning a collection that defers the object deserialization until a later point.
Unfortunately _replace_refs assumes all mapping types are Dicts and thus the type of the inbound Mapping object is lost.
This could be easily overcome by changing the building of the mapping
obj.class(**{.....})
this would preserve the inbound type that comes from json.load.
The same could be applied to Sequence types.

Obviously this does not apply to the replacement of the referent.

Add links to Pypi

Can I suggest adding links from the Pypi page to the docs and Github?

If it was in setup.py you can do:

project_urls={
    "Documentation": "https://aaaaa.readthedocs.io/en/latest/",
    "Issues": "https://github.com/aaaa/bbbb/issues",
    "Source": "https://github.com/aaa/bbbbb",
},

But I'm not sure how you do it in pyproject.toml

Escaped characters in references are not handled correctly

If I have a key for an item called /content inside parent then the correct way to reference this would be #/parent/~1content

But jsonref says this is an invalid reference. It seems that it cannot handle the escaping characters ~1 for a / character.

Support $ref replacement without the proxies

When using a jsonref'd data structure with third-party libraries, the presence of the proxy objects in the object graph can sometimes pose compatiblity issues.

In cases where it is not easy to replace usages of json.load() and json.dump() with their jsonref counterparts, an option to produce a json data structure without proxy objects is the simplest solution.

Something equivalent to what this utility function does:

def replace_jsonref_proxies(obj):
    """
    Replace jsonref proxies in the given json obj with the proxy target.
    Updates are made in place. This removes compatibility problems with 3rd
    party libraries that can't handle jsonref proxy objects.
    :param obj: json like object
    :type obj: int, bool, string, float, list, dict, etc
    """
    # TODO: consider upstreaming in the jsonref library as a util method
    def descend(fragment):
        if is_dict_like(fragment):
            for k, v in iteritems(fragment):
                if isinstance(v, jsonref.JsonRef):
                    fragment[k] = v.__subject__
                descend(fragment[k])
        elif is_list_like(fragment):
            for element in fragment:
                descend(element)

    descend(obj)

Would a PR with this functionality be considered for merging?

Switch to using wrapt proxy?

Getting the transparent proxies right is a bit complicated, and there have been some testing inconsistencies (especially pypy.) I'm wondering if using the wrapt version (https://wrapt.readthedocs.io/en/latest/wrappers.html) would be more stable/tested/well maintained rather than staying with my custom one. If anyone using this has any feelings one way or the other, let me know.

Regarding "A note on base_uri"

I think the docs and error handling would benefit from explicitly writing that the base_uri parameter must be set to the directory of the file that has references in it. I tried to set base_uri to the directory of the file from which to inject references, and that caused the program to crash. Took me a couple of hours to isolate the issue. I'm not sure if this is expected behavior or not, but in any case it is not apparent from the docs or the error. You can see the error thrown in example 2 below.

Assuming we have 2 files:

Example 1

`/project/folder/file-a.json`:

{
  "bat": {
      "$ref": "file:../file-b.json#/cat"
  }
}

`/project/file-b.json`:

{
  "cat": "hat"
}

`/project/main.py`

from pathlib import Path
import jsonref

with open("./folder/file-a.json", "r") as f:
    config: dict = jsonref.load(f, base_uri=Path("./folder/file-a.json").absolute().as_uri())
    print(config)

output:

{'bat': 'hat'}

Example 2

`/project/folder/file-a.json`:

{
  "bat": {
      "$ref": "file:./file-b.json#/cat"
  }
}

`/project/file-b.json`:

unchanged

`/project/main.py`

from pathlib import Path
import jsonref

with open("./folder/file-a.json", "r") as f:
    config: dict = jsonref.load(f, base_uri=Path("./file-b.json").absolute().as_uri())
    print(config)

expected output:

{'bat': 'hat'}

actual:

Traceback (most recent call last):
  File "/Library/Python/3.9/site-packages/jsonref.py", line 179, in resolve_pointer
    document = document[part]
KeyError: 'cat'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/<username>/pythonProject2/main.py", line 6, in <module>
    print(config)
  File "/Library/Python/3.9/site-packages/proxytypes.py", line 121, in wrapper
    return method(self, *args, **kwargs)
  File "/Library/Python/3.9/site-packages/jsonref.py", line 199, in __repr__
    return repr(self.__subject__)
  File "/Library/Python/3.9/site-packages/proxytypes.py", line 163, in __getattribute__
    return _oga(self, attr)
  File "/Library/Python/3.9/site-packages/proxytypes.py", line 121, in wrapper
    return method(self, *args, **kwargs)
  File "/Library/Python/3.9/site-packages/proxytypes.py", line 243, in __subject__
    self.cache = super(LazyProxy, self).__subject__
  File "/Library/Python/3.9/site-packages/proxytypes.py", line 121, in wrapper
    return method(self, *args, **kwargs)
  File "/Library/Python/3.9/site-packages/proxytypes.py", line 227, in __subject__
    return self.callback()
  File "/Library/Python/3.9/site-packages/proxytypes.py", line 121, in wrapper
    return method(self, *args, **kwargs)
  File "/Library/Python/3.9/site-packages/jsonref.py", line 140, in callback
    result = self.resolve_pointer(base_doc, fragment)
  File "/Library/Python/3.9/site-packages/proxytypes.py", line 121, in wrapper
    return method(self, *args, **kwargs)
  File "/Library/Python/3.9/site-packages/jsonref.py", line 181, in resolve_pointer
    raise self._error(
jsonref.JsonRefError: Error while resolving `file:///Users/<username>/file-b.json#/cat`: Unresolvable JSON pointer: '/cat'

Process finished with exit code 1

JsonRef.replace_refs mocking issue

I have had quite a bit of trouble using a mock on JsonRef.replace_refs for testing my application which uses it.

Here is an example:

@patch.object(JsonRef, "replace_refs")
def test_mock(self, mocked_replace_refs):
        reffed_json = {"foo": {"$ref": "#bar/baz"}, "bar": {"baz": "bar"}}
        dereffed_json = {"foo": "bar", "bar": {"baz": "bar"}}
        mocked_replace_refs.return_value = dereffed_promo

       code_calling_replace_refs(reffed_json)

This the error I get:

    @wraps(method)
    def wrapper(self, *args, **kwargs):
>       notproxied = _oga(self, "__notproxied__")
E       AttributeError: 'dict' object has no attribute '__notproxied__'

I think it has to do with the JsonRef class being a proxy and some interaction with that and @classmethod. The easy fix is to just make replace_refs a free floating function (I'd be happy to make a PR with that change) but I suspect there is a better solution? Lastly, thanks for your work on this library, I have been using it quite a bit and it's 👍.

Different behavior when using external file in $ref

Hi there,

First of all, thank you for creating the software.

I am getting different results when my $ref are located in an external file:

My code:

#!/usr/bin/env python3
import jsonref
import json
import sys

def default_for_jsonref(o):
    if isinstance(o, jsonref.JsonRef):
        return o.__subject__

def main():
    json_in = open(sys.argv[1], 'r', encoding="utf-8")
    print(json.dumps(jsonref.load(json_in), default=default_for_jsonref, indent=4))

if __name__ == "__main__":
    main()

Let's try first with this file as an argument:

{
  "foo": {
    "variation": {
          "$ref": "#/definitions/MolecularVariation"
    }
  },
  "definitions": {
      "MolecularVariation": {
         "description": "A variation on a contiguous molecule."
         }
      }
}

Results:

{
    "foo": {
        "MolecularVariation": {
            "description": "A variation on a contiguous molecule."
        }
    }
}

Note the hierarchy "foo/MolecularVariation/description".
Ok, now let's use an external file as $ref. The name of the file is 'vrs.json'

{
   "definitions": {
      "MolecularVariation": {
         "description": "A variation on a contiguous molecule."
         }
      }
}

And let's change the original (I am omitting my local path to the file):

{
  "foo": {
    "variation": {
          "$ref": "file:MY_PATH/vrs.json#/definitions/MolecularVariation"
    }
  }
}

This is the result:

{
    "foo": {
        "variation": {
            "description": "A variation on a contiguous molecule."
        }
    }
}

Note that we don't get the key "MolecularVariation".

Have you guys bumped into that? Any solutions?

Thanks!

Manu

Jsonref in conda-forge is obsolete

Jsonref in conda-forge is obsolete. It is still at version 0.2. It would be nice if the conda-forge feedstock gets updated as well.

Relative references pointers in multiple directories and files

I have 3 json schemas in 2 directories, for example:

/working/A.json
/ressource/B.json
/ressource/C.json

As shown, B and C schemas are in the same directory.

A uses relative references from B (from another directory):

"$ref": "../ressource/B.json#/definitions/myItemFromB"

and B uses relative references from C (from the same directory):

"$ref": "C.json#/definitions/myItemFromC"

If I load A with a base_uri = /working/ and print it, I get a path error for B referencing C:

jsonref.JsonRefError: URLError: <urlopen error [Errno 2] No such file or directory: /working/C.json

meaning that in jsonref, does not "update" the base_uri when jumping to a different schema file when following references chain.
Indeed, I would expect jsonref to search in /ressource/C.json.

How can I get the behavior I expect there?

pypy Travis job failing

I just now set up a build on Travis, and the pypy job is reporting two errors.

They are the same two errors that appeared on the last official build (#81) at year ago:
https://travis-ci.org/gazpachoking/jsonref/jobs/13109024

It is very unlikely that the commit (0ad88d1) caused it, so something else changed between May 2013 and Jan 2015 . Probably pypy changed something.

Unclosed local files from urlopen

When loading schemas from local files, I get a warning for all referenced schemas:

ResourceWarning: unclosed file <_io.BufferedReader name='/path/to/file.json'>
  result = json.loads(urlopen(uri).read().decode("utf-8"), **kwargs)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

The urlopen method should be used in a context manager to assure the files are closed, e.g.:

with urlopen(uri) as f:
    result = json.loads(f.read().decode("utf-8"), **kwargs)

Disabling proxies doesn't work recursively

When set proxies=False, the $ref fields in the first document are converted to a regular dict, but if there is a $ref in the resolved document, it remains a jsonref proxy and the document won't serialize via vanilla json.dumps.

Cannot dereference top level '$ref'

I'm trying to use jsonref to expand a JSON Schema which has a top level $ref but I'm getting a ValueError.

This is enough to to reproduce the issue:

schema = '{"definition": "helloworld", "$ref": "#/definition"}'
print(jsonref.loads(schema))

And this is the error:

Traceback (most recent call last):
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/proxytypes.py", line 252, in __subject__
    return self.cache
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/proxytypes.py", line 176, in __getattribute__
    return _oga(self, attr)
AttributeError: 'JsonRef' object has no attribute 'cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/jsonref.py", line 178, in callback
    base_doc = self.loader(uri)
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/jsonref.py", line 299, in __call__
    result = self.get_remote_json(uri, **kwargs)
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/jsonref.py", line 316, in get_remote_json
    result = json.loads(urlopen(uri).read().decode("utf-8"), **kwargs)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 509, in open
    req = Request(fullurl, data)
  File "/usr/lib/python3.8/urllib/request.py", line 328, in __init__
    self.full_url = url
  File "/usr/lib/python3.8/urllib/request.py", line 354, in full_url
    self._parse()
  File "/usr/lib/python3.8/urllib/request.py", line 383, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: ''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/shishax/.config/JetBrains/PyCharm2022.1/scratches/jsonref-test.py", line 5, in <module>
    print(jsonref.loads(schema))
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/proxytypes.py", line 202, in proxied
    args.insert(arg_pos, self.__subject__)
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/proxytypes.py", line 176, in __getattribute__
    return _oga(self, attr)
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/proxytypes.py", line 254, in __subject__
    self.cache = super(LazyProxy, self).__subject__
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/proxytypes.py", line 240, in __subject__
    return self.callback()
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/jsonref.py", line 180, in callback
    self._error("%s: %s" % (e.__class__.__name__, unicode(e)), cause=e)
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/shishax/.env/sam-test-jsonref/lib/python3.8/site-packages/jsonref.py", line 223, in _error
    raise JsonRefError(
jsonref.JsonRefError: ValueError: unknown url type: ''

Is it allowed to have a top level $ref?
I didn't found any mention in the docs.

OpenAPI v3.1 causes stack overflow

Hi,

I've been trying to get jsonref to work with the OpenAPI v3.1 spec and always hit an error when the lazy-loaded refs are expanded. OpenAPI v3.0 works fine, so I guess the cause is down to the introduction of a $defs section. Here's an example program:

from pprint import pprint
import jsonref

result = jsonref.load_uri("https://spec.openapis.org/oas/3.1/schema/2022-10-07")
pprint(result)

Running this on my system with Python 3.10 results in this error (with a very long stack trace):

jsonref.JsonRefError: Error while resolving `https://spec.openapis.org/oas/3.1/schema/2022-10-07#/$defs/specification-extensions`: RecursionError: maximum recursion depth exceeded while calling a Python object

Do you have any suggestions for a work-around? Thanks in advance.

Unresolvable JSON pointer with references/paths that contains ~1 and ~0

Hello there
I'm working on a project for resolving the refs in open API documentation.
when I resolve refs of this API: https://api.apis.guru/v2/specs/eos.local/1.0.0/openapi.json
it shows me this exception: jsonref.JsonRefError: Unresolvable JSON pointer: '/paths/~1net~1status/post/responses/200/content/application~1json/schema/properties/last_handshake/properties/token'
when I trace the path, it appeared to me that jsonref doesn't handle/resolve ~1 and ~0 properly.

here's a snippet of the code:

import json

import requests
import jsonref
api = 'https://api.apis.guru/v2/specs/eos.local/1.0.0/openapi.json'
data = requests.get(api).content
jsonref.JsonRef.replace_refs(
    json.loads(
        json.dumps(
            jsonref.loads(data), default=dict
        )
    )
)

and here's the entire exception:

Traceback (most recent call last):
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 252, in __subject__
    return self.cache
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 176, in __getattribute__
    return _oga(self, attr)
AttributeError: 'JsonRef' object has no attribute 'cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/jsonref.py", line 217, in resolve_pointer
    document = document[part]
KeyError: '~1net~1status'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/abdullah/Abdullah/coding/python/json/main_error.py", line 9, in <module>
    json.dumps(
  File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 175, in __getattribute__
    return getattr(self.__subject__, attr)
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 176, in __getattribute__
    return _oga(self, attr)
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 254, in __subject__
    self.cache = super(LazyProxy, self).__subject__
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 240, in __subject__
    return self.callback()
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/jsonref.py", line 174, in callback
    result = self.resolve_pointer(self.store[uri], fragment)
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/jsonref.py", line 219, in resolve_pointer
    self._error("Unresolvable JSON pointer: %r" % pointer, cause=e)
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/abdullah/Abdullah/coding/python/json/venv/lib/python3.8/site-packages/jsonref.py", line 223, in _error
    raise JsonRefError(
jsonref.JsonRefError: Unresolvable JSON pointer: '/paths/~1net~1status/post/responses/200/content/application~1json/schema/properties/last_handshake/properties/token'

but when I use only the built-in JSON library, it works without any errors:

import json

import requests
import jsonref
api = 'https://api.apis.guru/v2/specs/eos.local/1.0.0/openapi.json'
data = requests.get(api).content
jsonref.JsonRef.replace_refs(
    json.loads(
        json.dumps(
            json.loads(data), default=dict
        )
    )
)

any help if anyone has faced this issue and solved it?

Wrong reference resolve due to memory reuse.

For

def _walk_refs(obj, func, replace=False, _processed=None):
    # Keep track of already processed items to prevent recursion
    _processed = _processed or {}
    oid = id(obj)
    if oid in _processed:
        return _processed[oid]
    if type(obj) is JsonRef:
        r = func(obj)
        obj = r if replace else obj
    _processed[oid] = obj
    if isinstance(obj, Mapping):
        for k, v in obj.items():
            r = _walk_refs(v, func, replace=replace, _processed=_processed)
            if replace:
                obj[k] = r
    elif isinstance(obj, Sequence) and not isinstance(obj, str):
        for i, v in enumerate(obj):
            r = _walk_refs(v, func, replace=replace, _processed=_processed)
            if replace:
                obj[i] = r
    return obj

The memory of obj might be released if obj has been updated. Another object might occupied that piece of memory later. Then, id of it will be identical to the original one which causing return the wrong reference.

Can not handle circular references

Sample json

{
  "swagger": "2.0",
  "host": "xxxxxxxx",
  "paths": {
    "/": {
      "get": {
        "parameters": [],
        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "$ref": "#/definitions/ResponseData«List«DcDistrictVO»»"
            }
          }
        },
        "deprecated": false
      }
    }
  },
  "definitions": {
    "DcDistrictVO": {
      "type": "object",
      "properties": {
        "districtList": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/DcProvinceVO"
          }
        },
        "initials": {
          "type": "string"
        }
      },
      "title": "DcDistrictVO"
    },
    "DcProvinceVO": {
      "type": "object",
      "properties": {
        "code": {
          "type": "string"
        },
        "districtList": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/DcDistrictVO"
          }
        },
        "name": {
          "type": "string"
        }
      },
      "title": "DcProvinceVO"
    },
    "ResponseData«List«DcDistrictVO»»": {
      "type": "object",
      "properties": {
        "code": {
          "type": "integer",
          "format": "int32"
        },
        "data": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/DcDistrictVO"
          }
        },
        "message": {
          "type": "string"
        },
        "success": {
          "type": "boolean"
        }
      },
      "title": "ResponseData«List«DcDistrictVO»»"
    }
  }
}

DcProvinceVO and DcDistrictVO are circular referenced, there is not issue when jsonref.loads the raw json, but raise Exception when jsonref.dumps
Traceback:

Traceback (most recent call last):
  File "/Users/megachweng/Documents/Code/Test/zest-engine/zest_engine/cook/swagger/derefer.py", line 7, in <module>
    a = jsonref.dumps(raw_data)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/zest-engine/lib/python3.10/site-packages/jsonref.py", line 556, in dumps
    return json.dumps(obj, **kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/zest-engine/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/zest-engine/lib/python3.10/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/zest-engine/lib/python3.10/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/zest-engine/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/opt/homebrew/Caskroom/miniconda/base/envs/zest-engine/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/opt/homebrew/Caskroom/miniconda/base/envs/zest-engine/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  [Previous line repeated 6 more times]
  File "/opt/homebrew/Caskroom/miniconda/base/envs/zest-engine/lib/python3.10/json/encoder.py", line 340, in _iterencode_dict
    raise ValueError("Circular reference detected")
ValueError: Circular reference detected

Other properties on reference objects

I think there should be a couple more properties on the proxy objects. I think I'll create a JsonRef subclass of LazyProxy with:

subject: same as now, has the referent data
reference: the original json reference object
baseuri: the base uri of the document this reference is in
error: not sure about this one, maybe the exception if there was an error retrieving the data

Feature Request: when serializing, replace references with referents.

This is a feature request, but first of all: Thanks for making jsonref. It's handy.

From the documentation:

dump() and dumps() work just like their json counterparts, except they output the original reference objects when encountering JsonRef instances.

I was a bit unclear whether the "original reference object" was the reference itself, versus the object being referred to. I had hoped it was the latter, but it turned out to be the former.

Confirmed by this experiment

import jsonref
print(jsonref.dumps(jsonref.loads('''
{
  "foo": {
    "$ref": "#/def/obj"
  },
  "def": {
    "obj": {
      "bar": "baz"
    }
  }
}
''')))

Output references (this is what happens):

{"foo": {"$ref": "#/def/obj"}, "def": {"obj": {"bar": "baz"}}}:

Output objects referred to (this is what I hoped for):

{"foo": {"bar" : "baz"}, "def": {"obj": {"bar": "baz"}}}:

I found a way to get what I want here, but maybe this could be optional built-in functionality? Perhaps like this:

jsonref.dumps(obj, references_as='values')

Restoring process of escaped characters in JSON pointers is wrong

The part where escaped characters are getting restored is wrong.
According to the official specification of JSON pointers, the correct way of escaping characters is:

Replace ^ with ^^.
Replace / with ^/.

Fail to resolve the schema which contains `anyOf` field

Schema:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "fruit": {
      "anyOf": [
        {
          "$ref": "#/definitions/apple"
        },
        {
          "$ref": "#/definitions/banana"
        }
      ]
    }
  },
  "definitions": {
    "apple": {
      "type": "object",
      "properties": {
        "type": {
          "const": "apple"
        },
        "variety": {
          "type": "string"
        },
        "color": {
          "type": "string"
        }
      },
      "required": ["type", "variety", "color"],
      "additionalProperties": false
    },
    "banana": {
      "type": "object",
      "properties": {
        "type": {
          "const": "banana"
        },
        "variety": {
          "type": "string"
        },
        "curvature": {
          "type": "string"
        }
      },
      "required": ["type", "variety", "curvature"],
      "additionalProperties": false
    }
  },
  "required": ["fruit"]
}

Statement: replace_ref(schema)
Exception: JsonRefError: Error while resolving #/definitions/apple: Unresolvable JSON pointer: '/definitions/apple'

RFE: Please update to use `pdm-backend`

jsonref/pyproject.toml

Lines 25 to 27 in 82c038b

 [build-system] 

 requires = ["pdm-pep517>=1.0.0"] 

 build-backend = "pdm.pep517.api"

pdm-pep517 gone and should be replaced by pdm-backend.

jsonref vs jsonref-ap on PyPI

I am seeing two packages on pypi that both claim to be jsonref. I don't know which one is authentic. These are:

https://pypi.org/project/jsonref/ (Last released: May 17, 2013)
https://pypi.org/project/jsonref-ap/ (Last released: Mar 30, 2016)

The change in package name throws me off. Is the more recently-released one (2016) authentic?

Can't find reference

Hello,

my goal is to create a JSON file that has all its references expanded. My code for that looks like that:

            prev_cwd = os.getcwd()
            os.chdir(args.infile.parent)
            print(os.getcwd())
            in_dict = jsonref.loads(in_text, base_uri=str(args.infile))
            in_text = json.dumps(in_dict)
            os.chdir(prev_cwd)

The references looks like:

        "components": {
            "$ref": "file:common.json#/launch"
        },

The infile and the common.json both reside in the same directory.

However, this gives me multiple exceptions:

/home/floli/workspace/myproject/evaluation_launcher/config/schemas
Traceback (most recent call last):
  File "/home/floli/.local/lib/python3.8/site-packages/proxytypes.py", line 252, in __subject__
    return self.cache
  File "/home/floli/.local/lib/python3.8/site-packages/proxytypes.py", line 176, in __getattribute__
    return _oga(self, attr)
AttributeError: 'JsonRef' object has no attribute 'cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/urllib/request.py", line 1511, in open_local_file
    stats = os.stat(localfile)
FileNotFoundError: [Errno 2] No such file or directory: '/common.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/floli/.local/lib/python3.8/site-packages/jsonref.py", line 178, in callback
    base_doc = self.loader(uri)
  File "/home/floli/.local/lib/python3.8/site-packages/jsonref.py", line 299, in __call__
    result = self.get_remote_json(uri, **kwargs)
  File "/home/floli/.local/lib/python3.8/site-packages/jsonref.py", line 316, in get_remote_json
    result = json.loads(urlopen(uri).read().decode("utf-8"), **kwargs)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1489, in file_open
    return self.open_local_file(req)
  File "/usr/lib/python3.8/urllib/request.py", line 1528, in open_local_file
    raise URLError(exp)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory: '/common.json'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "visualize_json.py", line 33, in <module>
    in_text = json.dumps(in_dict)
  File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
  File "/home/floli/.local/lib/python3.8/site-packages/proxytypes.py", line 175, in __getattribute__
    return getattr(self.__subject__, attr)
  File "/home/floli/.local/lib/python3.8/site-packages/proxytypes.py", line 176, in __getattribute__
    return _oga(self, attr)
  File "/home/floli/.local/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/floli/.local/lib/python3.8/site-packages/proxytypes.py", line 254, in __subject__
    self.cache = super(LazyProxy, self).__subject__
  File "/home/floli/.local/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/floli/.local/lib/python3.8/site-packages/proxytypes.py", line 240, in __subject__
    return self.callback()
  File "/home/floli/.local/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/floli/.local/lib/python3.8/site-packages/jsonref.py", line 180, in callback
    self._error("%s: %s" % (e.__class__.__name__, unicode(e)), cause=e)
  File "/home/floli/.local/lib/python3.8/site-packages/proxytypes.py", line 134, in wrapper
    return method(self, *args, **kwargs)
  File "/home/floli/.local/lib/python3.8/site-packages/jsonref.py", line 223, in _error
    raise JsonRefError(
jsonref.JsonRefError: URLError: <urlopen error [Errno 2] No such file or directory: '/common.json'>

I have verified that the current working directory is set to where both files are, input and common.json. I am not really sure about the meaning of the base_uri argument and how it should look like?

Any help appreciated! Thanks!

Python 3.2 not supported by coverage

As coverage now breaks on Python 3.2, something needs to change in .travis.yml.
Now is a good time to consider de-supporting Python 3.2, by removing it from both .travis.yml and setup.py.
Or, with a tiny bit of complexity, .travis.yml could avoid using coverage on Python 3.2, so it can be supported a little longer.

pprint.pprint(value)

{'components': {'schemas': {'filter': {'properties': {'and': {'items': <Recursion on dict with id=4621593792>,
                                                              'type': 'array'}}},
                            'filterList': {'items': {'properties': {'and': <Recursion on dict with id=4621593216>}},
                                           'type': 'array'}}},

json.dumps(value)

    print(json.dumps(dumps))
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected

The json-schema-ref-parser library supports the circular references, the support for the option bundle in jsonref would be great.

tests

Need tests for more functionality.

DeprecationWarning in Python 3.7: collections imports should try abc submodule first

I am getting the following warning when I run a manubot command which calls jsonref internally:

/home/dhimmel/anaconda3/envs/manubot-dev/lib/python3.7/site-packages/jsonref.py:8: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping, MutableMapping, Sequence

Here is the code:

jsonref/jsonref.py

Lines 8 to 11 in 0235f49

 try: 

 from collections import Mapping, MutableMapping, Sequence 

 except ImportError: 

 from collections.abc import Mapping, MutableMapping, Sequence

Switching the order of the try except statements should prevent the warning. In other words, we want to try the new method first and fallback to the deprecated method. I'll submit a PR.

	[build-system]
	requires = ["pdm-pep517>=1.0.0"]
	build-backend = "pdm.pep517.api"

	try:
	from collections import Mapping, MutableMapping, Sequence
	except ImportError:
	from collections.abc import Mapping, MutableMapping, Sequence

gazpachoking / jsonref Goto Github PK

jsonref's Introduction

jsonref

Features

jsonref's People

Stargazers

Watchers

Forkers

jsonref's Issues

Example 1

/project/folder/file-a.json:

/project/file-b.json:

/project/main.py

output:

Example 2

/project/folder/file-a.json:

/project/file-b.json:

/project/main.py

expected output:

actual:

Recommend Projects

Recommend Topics

Recommend Org

`/project/folder/file-a.json`:

`/project/file-b.json`:

`/project/main.py`

`/project/folder/file-a.json`:

`/project/file-b.json`:

`/project/main.py`