davishmcclurg / json_schemer Goto Github PK

View Code? Open in Web Editor NEW

383.0 10.0 63.0 1000 KB

JSON Schema validator. Supports drafts 4, 6, 7, 2019-09, 2020-12, OpenAPI 3.0, and OpenAPI 3.1.

License: MIT License

Ruby 95.36% Shell 0.03% Python 4.62%

ruby json-schema json-validation

json_schemer's People

Contributors

Stargazers

Watchers

json_schemer's Issues

Support for loading referenced schemas?

Is loading fragments with $ref supported? Inside my schema, I'm trying to reference (load) a separate file with:

"$ref": "base_schema.json"

But it seems this is ignored.

Type conversion

It would be very handy if json_schemer could also convert values. I was thinking about a new option like conversion: true, which could be false by default. If set to true and the data is valid, json_schemer could either modify the original data or return a new object.

There are not many things one would have to convert:

{ type: string, format: date } -> Date
{ type: string, format: date-time } -> DateTime
{ type: string, format: time } -> Time
{ type: number } -> Float

What do you think?

"anyOf" and "oneOf" expected behavior

I encountered a bug with "anyOf" and "oneOf" when playing with nullable fields.
For the following test:
schemer.validate({'a' => nil}).to_a

If I use "oneOf" in my schema like the following, it will fail
{ "type": "object", "properties": { "a": { "oneOf": [ {"type": "null"}, {"type": "nubmer"} ] } } }

If I use "anyOf" in my schema like the following, it will succeed,
{ "type": "object", "properties": { "a": { "anyOf": [ {"type": "null"}, {"type": "nubmer"} ] } } }

JRuby/Windows regex test failure

One test isn't passing on JRuby/Windows: 5dc1a94

It looks like something related to ASCII and \u escaping.

{
    "description": "ECMA 262 \\D matches everything but ascii digits",
    "schema": {
        "type": "string",
        "pattern": "^\\D$"
    },
    "tests": [
        ...,
        {
            "description": "NKO DIGIT ZERO (as \\u escape) matches",
            "data": "\u07c0",
            "valid": true
        }
    ]
}

regexp_parser dependency is old

The dependency on regexp_parser is old. For example, when running with capybara version 3 or later:

Bundler could not find compatible versions for gem "regexp_parser":
  In snapshot (Gemfile.lock):
    regexp_parser (= 1.5.1)

  In Gemfile:
    capybara (~> 3) was resolved to 3.25.0, which depends on
      regexp_parser (~> 1.5)

    json_schemer (~> 0.2.0) was resolved to 0.2.0, which depends on
      regexp_parser (~> 1.2.0)

is it possible to update the dependency, or make it not restricted to a minor version?

JSONSchemerTest#test_json_schema_test_suite fails in master

Current master fails one test:

% bundle exec rake
Run options: --seed 64684

# Running:

...............F.............

Finished in 0.015892s, 1824.8175 runs/s, 3901.3340 assertions/s.

  1) Failure:
JSONSchemerTest#test_json_schema_test_suite [/Users/clivec/Repositories/git/GitHub/json_schemer/test/json_schemer_test.rb:621]:
Expected false to be truthy.

29 runs, 62 assertions, 1 failures, 0 errors, 0 skips```

support JSON Schema 2019-09 (formerly draft-08)

More Precise Error Validation for allOf, anyOf, oneOf

Given the following schema:

{
  "type": "object",
  "required": [
    "numberOfModules"
  ],
  "properties": {
    "numberOfModules": {
      "allOf": [
        {
          "not": {
            "type": "integer",
            "minimum": 38
          }
        },
        {
          "not": {
            "type": "integer",
            "maximum": 37,
            "minimum": 25
          }
        },
        {
          "not": {
            "type": "integer",
            "maximum": 24,
            "minimum": 12
          }
        }
      ]
    }
  }
}

When I validate, I get a high level error description saying allOf failed:

2.3.3 :009 > schemer.validate({ 'numberOfModules' => 32 }).to_a
 => [{"data"=>21, "schema"=>{"x-notification"=>"Reduce system size - Deselect modules to reduce the cost of a main service panel upgrade.", "allOf"=>[{"not"=>{"x-cost"=>7500, "type"=>"integer", "minimum"=>38}}, {"not"=>{"x-cost"=>5000, "type"=>"integer", "maximum"=>37, "minimum"=>25}}, {"not"=>{"x-cost"=>2500, "type"=>"integer", "maximum"=>24, "minimum"=>12}}, {"x-error"=>"Number of modules cannot be negative.", "type"=>"integer", "minimum"=>0}]}, "pointer"=>"/numberOfModules", "type"=>"allOf"}]

https://ajv.js.org/ gives a more specific error path:

> ajv.validate(s.design, { numberOfModules: 32 })
false
> ajv.errors
[ { keyword: 'maximum',
    dataPath: '.numberOfModules',
    schemaPath: '#/properties/numberOfModules/then/then/else/maximum',
    params: { comparison: '<=', limit: 24, exclusive: false },
    message: 'should be <= 24' } ]

It's nice to have the schema path for the lowest level of failure. I'm potentially open to working on a PR, but first wanted to know your thoughts on whether:

You are open to moving in this direction.
Moving in this direction would open up a whole can of worms that would require a large effort. For instance, if we made error paths more precise of allOf - would we then need to work on that for anyOf, oneOf, if, then, etc - or could we add these piecemeal?

processing validation errors raises if a key happens to be an integer

Hi,

Basically, we're using OpenAPI v2 with the official JSON schema file.
Response objects are specified using the http status code as a key, like for instance in yaml:

'200':
  description: a pet to be returned
  schema:
    $ref: '#/definitions/Pet'

The problem is that if the author of the OpenAPI spec file forgets the quotes around the status code it is then parsed as an integer and when such a schema is fed to json_schemer for validation it raises when trying to process the errors with TypeError: no implicit conversion of Fixnum into String.

Calling validate itself doesn't raise, it's when trying to process the errors (calling to_a for instance) that things explode.

The culprit is this line.

Thoughts on how to best handle this?

Remove Ruby version constraint from the gemspec

Hello,

we would love to use this gem to validate JSON payloads on-failure in our gem, jsonapi_parameters. This gem provides a simple way to consume JSON:API compliant inputs in Ruby, with Rails support included. You can read more about the standard here: https://jsonapi.org

Sadly, we are facing an issue where choosing json_schemer as the validation engine would mean that we would have to give up support for Ruby 2.4. While we are aware that this Ruby version has reached its EOL, we would rather not limit our users to specific Ruby versions unless it is required by the gem's logic.

Because of that we are currently seeing three scenarios:

either exporting the validation as a plug-in (so the validation part would be excluded out of JsonApi::Parameters gem and instead be provided as a complementary gem),
leave it as it is in the current state (PR link: visualitypl/jsonapi_parameters#45), thus give up for Ruby versions below 2.5.0,

and, the third scenario is: reaching out and asking whether its possible (and if not - what is the reason for this decision) to remove the constraint in json_schemer that enforces Ruby versions above 2.4.0. This is the path that we decided to choose first.

Commit link: 8cbd5cc#diff-6f69d521c8f547072ea493be84d7a52eL23-R23

allOf and referenced schemas always returning true for the referenced type

allOf with refs to remote schemas (or resolved local schemas) do not validate referenced types properly
i.e

schemaWithRefs  = {
   "$id"=>"NAME_HERE",
   "$schema"=>"http://json-schema.org/draft-04/schema#",
   "type"=>"object",
   "description"=>"A Host entity has been created or changed",
   "required"=>["host"],
   "additionalProperties"=>false,
   "properties"=> {
      "host"=> {
        "description"=>"Host", 
       "allOf"=>[{"$ref"=>"https://my-url-here.com/EDM.json#Host"}], 
       "required"=>["attribute1"]
      }
   } 
}

And the referenced schema fragment being

{
 "Host" => {
   "$id" => "#Host",
   "properties" => { "attribute1" => { "type" => "string"  } }
 }
}

schema = JSONSchemer.schema(schemaWithRefs, ref_resolver: 'net/http')
schema.valid?({
  'host' => {
   "attribute1" => nil
  }
})
// true

How to validate data against schema fragments?

Hi,

I am currently trying to replace json-schema gem with this gem. json-schema allowed validating data against a sub-schema specified by a fragment path, like this:

schema = {
  "type"=>"object",
  "required" => ["a"],
  "properties" => {
    "a" => {
      "type" => "integer",
      "default" => 42
    },
    "b" => {
      "type" => "object",
      "properties" => {
        "x" => {
          "type" => "integer"
        }
      }
    }
  }
}

JSON::Validator.validate(schema, { "x" => 1 }, :fragment => "#/properties/b")

Is there anything similar possible here? Of course I could manually select the relevant part of schema and validate against that, however this would break references to any other parts of the schema. Especially in OpenApi3 schemas these references to other parts of the same json schema are very common.

I tried to manually create instances of JSONSchemer::Schema::Base::Instance with data_pointer and schema_pointer set but no real success.

Feature: Support for the OpenAPI variation of JSON Schema

The OpenAPI specification contains a slightly modified subset of JSON Schema. It would be handy for my use case if I could validate documents against it.

I'm interested in implementing this, would that be acceptable to integrate into this project? Any advice on how to approach it?

Docs for local "$refs"

I'd love to use the library, but I can't unless I can easily get local "$refs" working.

If this is supported, it could use some documentation / examples.

See https://json-schema.org/understanding-json-schema/structuring.html#reuse for what I'm talking about.

Documentation

Improve the readme or introduce some other documentation.

Date validation misses some invalid values

The DateTime.rfc3339 used here to check for valid dates doesn't raise an error when the year has more than four digits or starts with a -.

Technically -0001-12-31 is a real date but it breaks our BigQuery implication.

Support readOnly, writeOnly (>= Draft 7)

The readOnly and writeOnly keywords (see specification (Draft 7)) make validation behave differently in the context of a write or read operation.

Unfortunately this does not seem to be covered by JSON-Schema-Test-Suite.

I found one (Java) library that supports this.

Add support for symbolized keys

Currently, parsed JSON with symbolized key fails validation. Please add support for validating JSON with symbolized keys.

Validate Only Works When Errors Exist

It seems wasteful to run:

valid=schemer.valid?(data)
errors = schemer.validate(data).to_a unless valid

Why evaluate the data twice to display the errors? So I try simply doing:

errors = schemer.validate(data).to_a
return true if errors.empty?

// process validation errors here.

However, if there are no errors, validate fails and raise exception:

KeyError: key not found: ""

It would be useful if the enumerable was able to work when validate is called on a valid json data.

Markdown generator of spec

I was wondering if we could extend this to a generator aswell, i.e generate markdown from the schema? Is that something the maintainers would accept?

URI::RFC3986_PARSER.parse is too restrictive for the JSON pointers

URI::RFC3986_PARSER.parse is too restrictive for the JSON pointers. Here is the fragment from RFC 6901:

 For example, given the JSON document

   {
      "foo": ["bar", "baz"],
      "": 0,
      "a/b": 1,
      "c%d": 2,
      "e^f": 3,
      "g|h": 4,
      "i\\j": 5,
      "k\"l": 6,
      " ": 7,
      "m~n": 8
   }

   The following JSON strings evaluate to the accompanying values:

    ""           // the whole document
    "/foo"       ["bar", "baz"]
    "/foo/0"     "bar"
    "/"          0
    "/a~1b"      1
    "/c%d"       2
    "/e^f"       3
    "/g|h"       4
    "/i\\j"      5
    "/k\"l"      6
    "/ "         7
    "/m~0n"      8

InvalidFileURI "cannot have a host" on Windows

This code works on Mac:

path = File.join(File.dirname(__dir__),'rsmp.json')
schema = JSONSchemer.schema( Pathname.new(path) )

But fails on Windows with:

JSONSchemer::InvalidFileURI:
  cannot have a host (use `file:///`)
# ./vendor/bundle/ruby/2.6.0/gems/json_schemer-0.2.17/lib/json_schemer.rb:42:in `block in <module:JSONSchemer>'
# ./vendor/bundle/ruby/2.6.0/gems/json_schemer-0.2.17/lib/json_schemer/cached_ref_resolver.rb:10:in `call'
# ./vendor/bundle/ruby/2.6.0/gems/json_schemer-0.2.17/lib/json_schemer.rb:57:in `schema'
# ./vendor/bundle/ruby/2.6.0/bundler/gems/rsmp-9383b3296fb7/lib/rsmp/message.rb:16:in `load_schemas'
# ./vendor/bundle/ruby/2.6.0/bundler/gems/rsmp-9383b3296fb7/lib/rsmp/message.rb:30:in `<class:Message>'
# ./vendor/bundle/ruby/2.6.0/bundler/gems/rsmp-9383b3296fb7/lib/rsmp/message.rb:3:in `<module:RSMP>'
# ./vendor/bundle/ruby/2.6.0/bundler/gems/rsmp-9383b3296fb7/lib/rsmp/message.rb:2:in `<top (required)>'
# ./vendor/bundle/ruby/2.6.0/bundler/gems/rsmp-9383b3296fb7/lib/rsmp.rb:30:in `require'
# ./vendor/bundle/ruby/2.6.0/bundler/gems/rsmp-9383b3296fb7/lib/rsmp.rb:30:in `<top (required)>'
# ./spec/support/spec_helper.rb:2:in `require'
# ./spec/support/spec_helper.rb:2:in `<top (required)>'

Dereferencing schemas does not work as expected in RFC section 8.3.2

it seems like, dereferencing is not working as expected, consider this example

test_schemer.rb

require 'json_schemer'

def evaluate(instance)
  schema = Pathname.new('schema1.json')
  schemer = JSONSchemer.schema(schema)

  result = {
    is_valid: schemer.valid?(instance),
    errors: schemer.validate(instance).to_a
  }
  puts result
end

evaluate({'a' => 1})
evaluate({'a' => 'b'})

schema1.json

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "schema1.json",
  "type": "object",
  "properties": {
    "a": {
      "$ref": "schema2.json"
    }
  }
}

schema2.json

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "schema2.json",
  "type": "integer"
}

running ruby test_schemer.rb results in the following exception:

Traceback (most recent call last):
	17: from test_schema.rb:14:in `<main>'
	16: from test_schema.rb:8:in `evaluate'
	15: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:55:in `valid?'
	14: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:65:in `valid_instance?'
	13: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:65:in `none?'
	12: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:65:in `each'
	11: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:138:in `validate_instance'
	10: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:223:in `validate_type'
	 9: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:459:in `validate_object'
	 8: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:459:in `each'
	 7: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:478:in `block in validate_object'
	 6: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:97:in `validate_instance'
	 5: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:228:in `validate_ref'
	 4: from /Users/joe.cabezas/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.9/lib/json_schemer/schema/base.rb:534:in `join_uri'
	 3: from /Users/joe.cabezas/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/uri/common.rb:275:in `join'
	 2: from /Users/joe.cabezas/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/uri/rfc3986_parser.rb:89:in `join'
	 1: from /Users/joe.cabezas/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/uri/rfc3986_parser.rb:89:in `inject'
/Users/joe.cabezas/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/uri/generic.rb:1110:in `merge': both URI are relative (URI::BadURIError)

which is expected to work as per 8.2.3[1]

anyways, it does work if I remove the $id field from schema1.json, why is this?, all schemas are suposed to have an id, please advice

also, thanks for the library!

[1] https://json-schema.org/latest/json-schema-core.html#rfc.section.8.3

support for local ref schemas

My schema is complex with lots of reusable components, and they are split into multiple schema files. I took a quick look at the source, but it looks like this doesn't support loading multiple schemas? For example, AJV is a popular JSON schema validator in JavaScript and it supports loading multiple schema files.

I'm trying to get the below code to work:

# json_schema_validator.rb
schemer = JSONSchemer.schema(Pathname.new(Rails.root.join(..., 'config', 'config_mapper.json')))
raw = YAML.load_file(..., 'input', 'input.yaml'))
schemer.validate(raw).to_a

validator/json_schema_validator.rb
validator/config/config_mapper.json
validator/config/mappers.json
validator/input/input.yaml

JSON schema files at: https://gist.github.com/rhuang/2e4b21bb1fb7b52a14a4a18b02a1dd7c

Running into: JSONSchemer::InvalidFileURI (must use file scheme), can someone please advise?

It also looks like this library doesn't support:

$ref: mappers.json/Mapper

# mappers.json
SomeMapper:
  type: object
OtherMapper:
  type: object
Mapper:
  oneOf: [SomeMapper, OtherMapper]

Is there a way to get this format to work?

undefined values in the json that have a default in the schema are replaced with the default before validation

How I can replace undefined values in the json that have a default in the schema are replaced with the default before validation?

Wrong behavior of insert_property_defaults: true with oneOf

insert_property_defaults: true causes validation failure when used with 'oneOf'. In the following example JSONSchemer inserts 'a' as a default for the 'bar' key:

require 'json_schemer'

schema = {
  'oneOf' => [
    { 
      'type' => 'object',
      'properties' => {
        'foo' => { 'enum' => ['a'] },
        'bar' => { 'enum' => ['a'], 'default' => 'a' }
      } 
    },
    {
      'type' => 'object',
      'properties' => {
        'foo' => { 'enum' => ['b'] },
        'bar' => { 'enum' => ['b'], 'default' => 'b' }
      } 
    },
  ]
}

schemer = JSONSchemer.schema(schema)

schemer.valid?('foo' => 'b')
# => true

schemer = JSONSchemer.schema(schema, insert_property_defaults: true)

schemer.valid?('foo' => 'b')
# => false

Convert error hash to object with helpful error rendering messages?

Hello, and thanks for working on this project. I'm working to get some helpful error messages produced by schema validation and having to jump through some hoops:

  def steps_must_conform_to_schema
    steps.each.with_index do |step, index|
      schema_errors = SCHEMA.validate step

      if schema_errors.any?
        errors.add(
          "step #{index}",
          decode_schema_error(schema_errors.first)
        )
      end
    end
  end

  def decode_schema_error(error)
    messages = []

    error["details"].each do |error_type, details|
      case error_type
      when "missing_keys"
        messages << "is missing required keys #{details.join(", ")}"
      when "schema"
      end
    end

    messages.join ", "
  end

Would it be a useful enhancement to convert the error hash into an object which has rendering methods for building the plain text version of the error?

Invalid additionalProperties implementation

schema = {
  "type"=>"object",
  "required"=>["foo", "bar"],
  "additionalProperties"=>false,
  "properties"=>{
    "foo"=>{"type"=>"number", "minimum"=>0},
    "bar"=>{"type"=>"string"}
  }
}

=> {"type"=>"object", "required"=>["foo", "bar"], "additionalProperties"=>false, "properties"=>{"foo"=>{"type"=>"number", "minimum"=>0}, "bar"=>{"type"=>"string"}}}

ap JSONSchemer.schema(schema).validate( {foo: 12, bar: 'qw'} ).to_a

[
    [0] {
                  "data" => {
            :foo => 12,
            :bar => "qw"
        },
          "data_pointer" => "",
                "schema" => {
                            "type" => "object",
                        "required" => [
                [0] "foo",
                [1] "bar"
            ],
            "additionalProperties" => false,
                      "properties" => {
                "foo" => {
                       "type" => "number",
                    "minimum" => 0
                },
                "bar" => {
                    "type" => "string"
                }
            }
        },
        "schema_pointer" => "",
           "root_schema" => {
                            "type" => "object",
                        "required" => [
                [0] "foo",
                [1] "bar"
            ],
            "additionalProperties" => false,
                      "properties" => {
                "foo" => {
                       "type" => "number",
                    "minimum" => 0
                },
                "bar" => {
                    "type" => "string"
                }
            }
        },
                  "type" => "required"
    },
    [1] {
                  "data" => 12,
          "data_pointer" => "/foo",
                "schema" => false,
        "schema_pointer" => "/additionalProperties",
           "root_schema" => {
                            "type" => "object",
                        "required" => [
                [0] "foo",
                [1] "bar"
            ],
            "additionalProperties" => false,
                      "properties" => {
                "foo" => {
                       "type" => "number",
                    "minimum" => 0
                },
                "bar" => {
                    "type" => "string"
                }
            }
        },
                  "type" => "schema"
    },
    [2] {
                  "data" => "qw",
          "data_pointer" => "/bar",
                "schema" => false,
        "schema_pointer" => "/additionalProperties",
           "root_schema" => {
                            "type" => "object",
                        "required" => [
                [0] "foo",
                [1] "bar"
            ],
            "additionalProperties" => false,
                      "properties" => {
                "foo" => {
                       "type" => "number",
                    "minimum" => 0
                },
                "bar" => {
                    "type" => "string"
                }
            }
        },
                  "type" => "schema"
    }
]

cat Gemfile.lock | grep json_schemer
    json_schemer (0.1.10)

Output formatting

Implement the JSON Schema spec's "basic" output format: json-schema-org/json-schema-spec#679

Empty pointers returned from .validate with valid? returning false

I'm interested in adopting this gem but I'm having trouble getting started. Validation is clearly failing (valid? returns false) when I use this gem but passes when I use another. When I explore the result of validate I have empty pointers and data and schemas that look like they should match. Any suggestions?

NoMethodError for schemas resolved to nil

I stubled across this chasing a bug in my code.
If you pass a proc/lambda with ref_resolver to the schemer, and this block resolves the passed uri to nil, it will cause an exception:

NoMethodError: undefined method `empty?' for nil:NilClass
/home/sr/.rvm/gems/ruby-2.6.0/gems/json_schemer-0.1.10/lib/json_schemer/schema/base.rb:79:in `validate_instance'

You'll probably want to raise an UnknownRef exception when the resolver returns something unuseable.

Invalid additionalProperties implementation

Gem for 0.2.9 has a broken link

When trying to use 0.2.9, the Gem unpacks with a broken symlink:

lrwxr-xr-x wheel/wheel       0 2020-02-01 19:06 JSON-Schema-Test-Suite/tests/latest -> JSON-Schema-Test-Suite/tests/draft2019-09

This is causing my rspec to fail in weird ways:

$ bundle exec rspec

An error occurred in a `before(:suite)` hook.
Failure/Error:
  File.open(path()) do |s|
    File.open(dest, 'wb', s.stat.mode) do |f|
      IO.copy_stream(s, f)
    end
  end

Errno::ENOENT:
  No such file or directory @ rb_sysopen - /home/ben/gitlab.com/gitlab-cookbooks/gitlab-exporters/.bundle/gems/json_schemer-0.2.9/JSON-Schema-Test-Suite/tests/latest
# ./.bundle/gems/berkshelf-7.0.9/lib/berkshelf/berksfile.rb:667:in `block (3 levels) in vendor'

Using "$ref" and "oneOf" occurs JSONSchemer::InvalidFileURI

Somehow I got an issue on both latest release and master branch for those cases.

root.json

{
  "type": "object",
  "properties": {
    "foo": {
      "$ref": "foo.json"
    }
  }
}

foo.json

{
  "definitions": {
    "foo1": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "bar": { "$ref": "bar.json" }
      }
    },
    "foo2": {
      "type": "object",
      "properties": {
        "description": { "type": "string" }
      }
    }
  },
  "allOf": [
    { "$ref": "#/definitions/foo1" },
    { "$ref": "#/definitions/foo2" }
  ]
}

bar.json

{
  "type": "object",
  "properties": {
    "id": { "type": "number" }
  }
}

and this causes this example into JSONSchemer::InvalidFileURI error which doesn't make sense.

require "json_schemer"
schema = JSONSchemer.schema(Pathname.new("path/to/root.json"))
schema.valid?(
  "foo" => {
    "name" => "Greeter",
    "bar" => { "id" => 123 },
    "description" => "Hello World!"
  }
) # => must use `file` scheme (JSONSchemer::InvalidFileURI)

while this example works fine.

require "json_schemer"
schema = JSONSchemer.schema(Pathname.new("path/to/foo.json"))
schema.valid?(
  "name" => "name",
  "bar" => { "id" => 123 },
  "description" => "Hello World!"
) # => true

Looking through the stack trace URI for bar.json became #<URI::Generic bar.json> in the $ref-resolving steps.

insert_property_defaults does not work for nested schemas (eg. oneOf)

Hi, first of all, thank you for this wonderful gem!

I tried to set up the smallest possible reproduction script. It seems that the workaround to this issue is to use a top-level if-then-else clause, but it makes it harder to use nested schemas.

# frozen_string_literal: true

require 'json_schemer'
require 'json'

schema_def = {
  "$schema" => "http://json-schema.org/draft-07/schema#",
  "type" => "object",
  "oneOf" => [
    "$ref" => "#/definitions/a"
  ],
  "required" => ["field", "default_field"],
  "definitions" => {
    "a" => {
      "properties" => {
        "field" => { "type" => "string", "const" => "a" },
        "default_field" => { "enum" => ["f1", "f2"], "default" => "f1" },
      }
    }
  }
}.to_json

schema_def2 = {
  "$schema" => "http://json-schema.org/draft-07/schema#",
  "type" => "object",
  "required" => ["field", "default_field"],
  "properties" => {
    "field" => { "type" => "string", "const" => "a" },
    "default_field" => { "enum" => ["f1", "f2"], "default" => "f1" },
  }
}.to_json

schema = JSONSchemer.schema(schema_def, insert_property_defaults: true)
schema2 = JSONSchemer.schema(schema_def2, insert_property_defaults: true)

params = { "field" => "a" }
params2 = params.dup

schema.valid?(params) 
schema2.valid?(params2)
pp params # => {"field"=>"a"}
pp params2 # => {"field"=>"a", "default_field"=>"f1"}

Support version-less $schema keyword

Understanding JSON Schema:

It is recommended that all JSON Schemas have a $schema entry, which must be at the root. Therefore most of the time, you’ll want this at the root of your schema:
"$schema": "http://json-schema.org/schema#"

What do you think? We had to add a specific draft to our schemas.

URI validation can take a very long time

This could potentially lead to a DOS attack, if you are validating user-supplied data.

Example timing of validating similar urls. The key pattern that I have found seems to be long parameters in a URL that ends with a space. There may be other ways to achieve the same effect.

checking 5 -> http://example.com/?zzzzz=uhoh 
	checked in 0 seconds
checking 10 -> http://example.com/?zzzzzzzzzz=uhoh 
	checked in 0 seconds
checking 15 -> http://example.com/?zzzzzzzzzzzzzzz=uhoh 
	checked in 1 seconds
checking 20 -> http://example.com/?zzzzzzzzzzzzzzzzzzzz=uhoh 
	checked in 23 seconds
checking 25 -> http://example.com/?zzzzzzzzzzzzzzzzzzzzzzzzz=uhoh

the version with 25 z's takes a very long time, I haven't let it finish yet.

The above output was made with the following script:

schema = JSONSchemer.schema({
                              "type" =>  "object",
                              "properties" =>  {
                                "url" =>  {
                                  "type" =>  "string",
                                  "format" =>  "uri"
                                }
                              },
                              "required" =>  %w(url)
                            })

(1..10).each do |i|
  url = "http://example.com/?#{'z' * i * 5}=uhoh "

  puts "checking #{i * 5} -> #{url}"

  start = DateTime.now
  schema.validate({"url" => url}).to_a
  done = DateTime.now

  puts "\tchecked in #{done.to_i - start.to_i} seconds"
end

I think perhaps switching to using URI::Parser could be a way to fix this problem. I'm not sure how the performance of the library would be affected by such a change, but it can parse all of the URIs in the above example very quickly.

Does not work with the following example

My schema definition is defined as the following:

  SCHEMA = {
    "type": "object",
    "required": [
      "type_of_formative_quiz",
      "email_notification",
      "email_notification_tutorial_taken",
      "email_notification_quiz_completed",
      "email_notification_question_missed",
    ],
    "properties": {
      "type_of_formative_quiz": {
        "type": "string",
        "enum": ["practice", "graded"],
      },
      'formative_quiz_passing_score': {
        "type": "number",
        "minimum": 0,
        "maximum": 100,
      },
      'will_have_summative_quiz': {
        "type": "boolean",
      },
      'completion_required_for_summative_quiz': {
        "type": "boolean",
      },
      'summative_quiz_passing_score': {
        "type": "number",
        "minimum": 0,
        "maximum": 100,
      },
      'email_notification': {
        "type": "boolean",
      },
      'email_notification_tutorial_taken': {
        "type": "boolean",
      },
      'email_notification_quiz_completed': {
        "type": "boolean",
      },
      'email_notification_question_missed': {
        "type": "boolean",
      },
      'missed_question_threshold_type': {
        "type": "string",
        "enum": ["percentage", "people"],
      },
      'missed_question_threshold': {
        "type": "number",
        "minimum": 0,
        "maximum": 100,
      },
    },
    "dependencies": {
      "will_have_summative_quiz": {
        "required": [
          "completion_required_for_summative_quiz",
          "summative_quiz_passing_score"
        ],
      },
    },
  }

I am doing the following:

schemer = JSONSchemer.schema(SCHEMA)
pp schemer.validate({'abc': 'def'}).to_a

The output is blank array.
=> []

It should thorw the errors as required keys are missing. It works fine as described in the example on README. What am I doing wrong?

Allow for regexp_parser 2.x

Hi there,

The regexp_parser project has released the 2.0 version, which RuboCop now depends on. The conflicts with the 1.x release supported in this gem and results in nasty conflicts for anyone that tries to link their Ruby code. It would be wonderful if a release could be cut supporting the 2.x regexp_parser release.

Is there any way to get the resolved schema?

Hi @davishmcclurg ! I was trying to use the library but not to validate data, i need to get the schema's refs resolved. is there any way to get it?

URI.unescape is obsolete

Lots of such warnings are printed in the console:

json_schemer-0.1.7/lib/json_schemer/schema/base.rb:231:in `validate_ref': warning: URI.unescape is obsolete

This issue has plagued any ruby library that more or less deals with URIs for the last few years and while the replacement is a case-by-case thing, I believe a common way to handle this is to just rely on Addressable.
Just an idea but how about replacing uri_template, which has not been updated in 4 years, by Addressable and kill two birds with one stone?

For reference, here's the original deprecation "bug" https://bugs.ruby-lang.org/issues/4167

Does not validate pattern properties correctly

Problem

Per the JSON Schema validation spec, pattern properties should be evaluated per the "ECMA 262 regular expression dialect."

Of course, exact compliance is impossible, but Ruby's Regexp behavior for ^ and $ makes JSONSchemer very noncompliant.

Failure 1

An externally published schema uses ^foo$. JSONSchemer will fail to reject multiline instances.

JSONSchemer erroneously accepts:

foo
foo
foo

Failure 2

As a Ruby developer, I publish a schema that uses \Afoo\z. Compliant JSON Schema implementations (say, running on an actual ECMA 262 engine) fail to accept and fail to reject correctly.

ECMA 262 erroneously accepts:
Afooz

ECMA 262 erroneously rejects:
foo

Solutions

Obviously, this is a tricky situation, so I am opening the issue mostly to start a discussion.

Ruby's quirky behavior is very well documented by now, and I have not seen anything that says you can turn off its multiline mode.

If I had to propose a solution, I would suggest either requiring a foreign regex library like https://github.com/mudge/re2, or gracefully upgrading to it when available.

To me this issue is fairly serious because faulty regex checking can and will produce security vulnerabilities.

Invalid schema is silently accepted

I'm using json_schemer (0.2.11) in the project and couldn't help but notice some unexpected behaviour.

Is it expected that no errors are being raised when an invalid schema is provided to the .schema method?

I noticed it when I was debugging a schema that looked correct at a glance, but proved invalid upon closer inspection:

{
      "type" => "object",
      "properties" => {
        "type" => "array",
        "items" => {
          "type" => "object",
          "properties" => {...some properties...}
        }
      }
    }

The above is invalid, because the nested array value has no property name that would act as a key for the object type.

I digged a little deeper, and it seems like JSONSchemer doesn't care about schema validity at all.
For instance:

[5] pry> JSONSchemer.schema("potato cheese pierogi" => "Margaret Thatcher").valid?({ "some" => "json data" })
=> true

For contrast, an online json schema validator I found offers a following error in such a case:

[ {
  "level" : "fatal",
  "message" : "invalid JSON Schema, cannot continue\nSyntax errors:\n[ {\n  \"level\" : \"error\",\n  \"schema\" : {\n    \"loadingURI\" : \"#\",\n    \"pointer\" : \"/properties/type\"\n  },\n  \"domain\" : \"syntax\",\n  \"message\" : \"JSON value is of type string, not a JSON Schema (expected an object)\",\n  \"found\" : \"string\"\n} ]",
  "info" : "other messages follow (if any)"
}, {
  "level" : "error",
  "schema" : {
    "loadingURI" : "#",
    "pointer" : "/properties/type"
  },
  "domain" : "syntax",
  "message" : "JSON value is of type string, not a JSON Schema (expected an object)",
  "found" : "string"
} ]

Would it make more sense to raise a similar error in the described case? Either on .schema call or on #valid? call?

Add meta schema tests

#40 (comment)

What specifically is invalid?

      print JSONSchemer.schema(schema).validate(data) # see below
      JSONSchemer.schema(schema).valid?(data) # false

Returns:

{"name"=>"P.S. 1 - K", "org_id"=>"K001", "bldg_id"=>"K001", "level"=>"ps", "district"=>15, "subdistrict"=>1, "source"=>"bluebook", "capacity"=>nil, "capacityFuture"=>nil, "enroll"=>nil, "address"=>"309 47 STREET", "bldg_name"=>"P.S. 1 - K", "borocode"=>3, "excluded"=>false, "geojson"=>{"type"=>"Feature", "geometry"=>{"type"=>"Point", "coordinates"=>[-74.012289, 40.649042]}, "properties"=>{}}}[{"data"=>nil, "data_pointer"=>"/capacity", "schema"=>{"$id"=>"#/properties/capacity", "type"=>"integer", "default"=>0}, "schema_pointer"=>"/properties/capacity", "root_schema"=>{"definitions"=>{}, "$schema"=>"http://json-schema.org/draft-07/schema#", "$id"=>"http://example.com/root.json", "type"=>"object", "properties"=>{"name"=>{"$id"=>"#/properties/name", "type"=>"string", "default"=>"", "examples"=>["P.S. 7 - Q"]}, "level"=>{"$id"=>"#/properties/level", "type"=>"string", "default"=>"", "examples"=>["ps"]}, "enroll"=>{"$id"=>"#/properties/enroll", "type"=>"integer", "default"=>0}, "org_id"=>{"$id"=>"#/properties/org_id", "type"=>"string", "default"=>"", "examples"=>["Q007"]}, "source"=>{"$id"=>"#/properties/source", "type"=>"string", "title"=>"The Source Schema", "default"=>"", "examples"=>["bluebook"], "pattern"=>"^(.*)$"}, "address"=>{"$id"=>"#/properties/address", "type"=>"string", "examples"=>["80-55 CORNISH AVENUE"]}, "bldg_id"=>{"$id"=>"#/properties/bldg_id", "type"=>"string", "examples"=>["Q007"]}, "geojson"=>{"$id"=>"#/properties/geojson", "type"=>"object", "required"=>["type", "geometry", "properties"]}, "borocode"=>{"$id"=>"#/properties/borocode", "type"=>"integer"}, "capacity"=>{"$id"=>"#/properties/capacity", "type"=>"integer", "default"=>0}, "district"=>{"$id"=>"#/properties/district", "type"=>"integer"}, "excluded"=>{"$id"=>"#/properties/excluded", "type"=>"boolean"}, "bldg_name"=>{"$id"=>"#/properties/bldg_name", "type"=>"string"}, "subdistrict"=>{"$id"=>"#/properties/subdistrict", "type"=>"integer"}, "capacityFuture"=>{"$id"=>"#/properties/capacityFuture", "type"=>"integer", "default"=>0}}}, "type"=>"integer"}, {"data"=>nil, "data_pointer"=>"/capacityFuture", "schema"=>{"$id"=>"#/properties/capacityFuture", "type"=>"integer", "default"=>0}, "schema_pointer"=>"/properties/capacityFuture", "root_schema"=>{"definitions"=>{}, "$schema"=>"http://json-schema.org/draft-07/schema#", "$id"=>"http://example.com/root.json", "type"=>"object", "properties"=>{"name"=>{"$id"=>"#/properties/name", "type"=>"string", "default"=>"", "examples"=>["P.S. 7 - Q"]}, "level"=>{"$id"=>"#/properties/level", "type"=>"string", "default"=>"", "examples"=>["ps"]}, "enroll"=>{"$id"=>"#/properties/enroll", "type"=>"integer", "default"=>0}, "org_id"=>{"$id"=>"#/properties/org_id", "type"=>"string", "default"=>"", "examples"=>["Q007"]}, "source"=>{"$id"=>"#/properties/source", "type"=>"string", "title"=>"The Source Schema", "default"=>"", "examples"=>["bluebook"], "pattern"=>"^(.*)$"}, "address"=>{"$id"=>"#/properties/address", "type"=>"string", "examples"=>["80-55 CORNISH AVENUE"]}, "bldg_id"=>{"$id"=>"#/properties/bldg_id", "type"=>"string", "examples"=>["Q007"]}, "geojson"=>{"$id"=>"#/properties/geojson", "type"=>"object", "required"=>["type", "geometry", "properties"]}, "borocode"=>{"$id"=>"#/properties/borocode", "type"=>"integer"}, "capacity"=>{"$id"=>"#/properties/capacity", "type"=>"integer", "default"=>0}, "district"=>{"$id"=>"#/properties/district", "type"=>"integer"}, "excluded"=>{"$id"=>"#/properties/excluded", "type"=>"boolean"}, "bldg_name"=>{"$id"=>"#/properties/bldg_name", "type"=>"string"}, "subdistrict"=>{"$id"=>"#/properties/subdistrict", "type"=>"integer"}, "capacityFuture"=>{"$id"=>"#/properties/capacityFuture", "type"=>"integer", "default"=>0}}}, "type"=>"integer"}, {"data"=>nil, "data_pointer"=>"/enroll", "schema"=>{"$id"=>"#/properties/enroll", "type"=>"integer", "default"=>0}, "schema_pointer"=>"/properties/enroll", "root_schema"=>{"definitions"=>{}, "$schema"=>"http://json-schema.org/draft-07/schema#", "$id"=>"http://example.com/root.json", "type"=>"object", "properties"=>{"name"=>{"$id"=>"#/properties/name", "type"=>"string", "default"=>"", "examples"=>["P.S. 7 - Q"]}, "level"=>{"$id"=>"#/properties/level", "type"=>"string", "default"=>"", "examples"=>["ps"]}, "enroll"=>{"$id"=>"#/properties/enroll", "type"=>"integer", "default"=>0}, "org_id"=>{"$id"=>"#/properties/org_id", "type"=>"string", "default"=>"", "examples"=>["Q007"]}, "source"=>{"$id"=>"#/properties/source", "type"=>"string", "title"=>"The Source Schema", "default"=>"", "examples"=>["bluebook"], "pattern"=>"^(.*)$"}, "address"=>{"$id"=>"#/properties/address", "type"=>"string", "examples"=>["80-55 CORNISH AVENUE"]}, "bldg_id"=>{"$id"=>"#/properties/bldg_id", "type"=>"string", "examples"=>["Q007"]}, "geojson"=>{"$id"=>"#/properties/geojson", "type"=>"object", "required"=>["type", "geometry", "properties"]}, "borocode"=>{"$id"=>"#/properties/borocode", "type"=>"integer"}, "capacity"=>{"$id"=>"#/properties/capacity", "type"=>"integer", "default"=>0}, "district"=>{"$id"=>"#/properties/district", "type"=>"integer"}, "excluded"=>{"$id"=>"#/properties/excluded", "type"=>"boolean"}, "bldg_name"=>{"$id"=>"#/properties/bldg_name", "type"=>"string"}, "subdistrict"=>{"$id"=>"#/properties/subdistrict", "type"=>"integer"}, "capacityFuture"=>{"$id"=>"#/properties/capacityFuture", "type"=>"integer", "default"=>0}}}, "type"=>"integer"}]

What is invalid about the input object? I can't interpret this to fix the problem.

If, then, else keywords

Hi there,
Thanks for the amazing work you guys have accomplished.

I was wondering, are If, else, and then keywords supported in this gem https://json-schema.org/understanding-json-schema/reference/conditionals.html ?

Thanks

Add "pathname" gem requirement to the docs

Otherwise...

$ ruby bin/validate.rb 
Traceback (most recent call last):
	1: from bin/validate.rb:51:in `<main>'
/Users/dan/.rvm/gems/ruby-2.5.1/gems/json_schemer-0.1.7/lib/json_schemer.rb:36:in `schema': uninitialized constant #<Class:JSONSchemer>::Pathname (NameError)

Tanks!

Is the ReadMe upto date ?

I am using https://github.com/ruby-json-schema/json-schema as of now and as far as I can tell it not being maintained anymore. I was looking for similar feature but with better support and came across this one.

From what I can see in issue's it seems to be this gem can do a lot more than the Readme provides . Can the ReadMe be updated please ?

Things I want to know.
1)Is there a way to see errors on which there was a failure.examples in readMe would be great
2)Is there a way to customize the error message returned from json schema (ajv-errors)
3)Is there a way to fail on first validation issue vs make a list of all validation issue's (validate and fully_valdate in json shema)
4)Is there a way to access a portion of json schema (i.e my json data only consists on one property in json schema) eg: $ref = /path/to/json#properties/columns

I am new to this whole ruby gem community thing so apologies if I looked rude or something

Strict validation

Issue/Question

Trying to migrate from ruby-json-schema, I realized json_schemer doesn't seem to have a :strict validation option.

Given

{
  "type": "object", 
  "properties": {
    "name": {
      "type": "string"
    }
  }
}

Using ruby-json-schema

::JSON::Validator.validate(schema, {"name" => "test", "extra_col" => 1}, :strict => true )
=> false

::JSON::Validator.fully_validate(schema, {"name" => "test", "extra_col" => 1}, :strict => true )
=> ["The property '#/' contained undefined properties: 'extra_col' in schema ca942ccb-0676-5741-b04d-40ca0787af04"]

The strict validation fails because of the extra column.

Is there anyway currently to achieve this with json_schemer?

`format.rb` throws an error if anything other than a string is passed to it

Hello,

I was experimenting with your gem when I noticed an issue. It appears that any when a schema specifying a format is passed a non-string value, then it will throw an exception. Code to repeat:

require 'json_schemer'
schema = {"format"=>"email" }
data = []
JSONSchemer.schema(schema).validate(data)

I will create a PR to address it. Let me know if you can merge the fix and push it out?

Thank you!

P. S. it looks like the test suite has not been pushed up. Getting:

Unable to checkout '2146b02555b163da40ae98e60bf36b2c2f8d4bd0' in submodule path 'JSON-Schema-Test-Suite'