jonasschmidt / ex_json_schema Goto Github PK
View Code? Open in Web Editor NEWAn Elixir JSON Schema validator
License: MIT License
An Elixir JSON Schema validator
License: MIT License
The json schema validation is really great!
I had some trouble writing a schema for validation. The current documentation didn't guide me well. After a while.. I finally got it down but I think it can be better.
I would like to write a few examples and stick them in the current README, if it's okay. I think it would make it easier for other people that come across it.
What do you think? Or are there docs I'm missing somewhere?
The repo isn't seeing a lot of traffic but some PRs take a bit for turnaround time (the 1.9.x fix for example), would it make sense to add a contributor with merge/publish rights to speed up turn around on small fixes?
It would be helpful if the error messages for these combining properties included the failure messages for their component schemas when they fail. Thus for oneOf show the messages from all failing items if any when it fails to match, or list the number of passing matches if more than one. anyOf should show failures if none match, and all of should show all failures.
How can I parse a schema with a local file reference (common.json#/example
)?
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Example Schema",
"type": "object",
"required": [ "example" ],
"additionalProperties": false,
"properties": {
"example": { "$ref": "common.json#/example" }
}
}
If I just do:
File.read!("lib/myapp/schemas/myschema.json") |> Poison.decode! |> ExJsonSchema.Schema.resolve
then we get a (confusing) error:
** (BadFunctionError) expected a function, got: nil
.
Understandably the resolver doesn't know where to look for the file, is there a way I can pass in a base directory or similar?
Hello,
I am trying to validate basic JSON example and have troubles. Can't understand, do I do something wrong with the ex_json_schema library or something wrong with my JSON schema.
The JSON schema is:
{
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "http://example.com/example.json",
"type": "object",
"properties": {
"auth": {
"description": "",
"type": "object",
"$ref": "#/$defs/proxy_server"
}
},
"$defs": {
"proxy_server": {
"type": "object",
"oneOf": [
{
"type": "object",
"additionalProperties": false
},
{
"patternProperties": {
".*": { "type": "object", "$ref": "#/$defs/server" }
}
}
]
},
"server": {
"type": "object",
"properties": {
"listen": {
"type": "string",
"format": "ip-address"
}
},
"required": ["listen"]
}
}
}
I want to have an object that will be or {"auth": {}}
or {"auth": {"server": {"listen": ....}}}
. But even if we'll omit nested examples, the:
ExJsonSchema.Validator.validate(schema, %{"auth": 1})
returns :ok
while it clearly should fail because auth
has to be an object.
The schema
after resolving is:
schema #{'__struct__' => 'Elixir.ExJsonSchema.Schema.Root',
custom_format_validator => nil,definitions => #{},location => root,
refs => #{},
schema =>
#{<<"$defs">> =>
#{<<"proxy_server">> =>
#{<<"oneOf">> =>
[#{<<"additionalProperties">> => false,
<<"properties">> => #{},
<<"type">> => <<"object">>}],
<<"type">> => <<"object">>},
<<"server">> =>
#{<<"properties">> =>
#{<<"listen">> =>
#{<<"format">> => <<"ip-address">>,
<<"type">> => <<"string">>}},
<<"required">> => [<<"listen">>],
<<"type">> => <<"object">>}},
<<"$id">> => <<"http://example.com/example.json">>,
<<"$schema">> => <<"http://json-schema.org/draft-07/schema">>,
<<"properties">> =>
#{<<"auth">> =>
#{<<"$ref">> => [root,<<"$defs">>,<<"proxy_server">>],
<<"description">> => <<>>,
<<"type">> => <<"object">>}},
<<"type">> => <<"object">>},
version => 7}
Could you please give me a hint what could be wrong?
Draft 06 of the JSON schema has been released, here's a summary of the changes.
I might work on a pull request (time permitting), but wanted to get an issue up.
defmodule Main do
def main do
schema = %{
"type" => "object",
"properties" => %{
"foo" => %{
"type" => "number"
},
"bar" => %{
"type" => ["number", "null"]
},
"baz" => %{
"type" => "string"
},
}
} |> ExJsonSchema.Schema.resolve
foo = Poison.decode!(~s({"whut": null, "tha": false, "hak": 455}))
case ExJsonSchema.Validator.validate(schema, foo) do
:ok -> foo |> inspect |> IO.puts
# I can't get this thing to error out when I give it JSON that does not match the schema.
{:err, messages} -> messages |> inspect |> IO.puts
end
end
end
Main.main
defmodule ElixirFoo.MixProject do
use Mix.Project
def project do
[
app: :elixirFoo,
version: "0.1.0",
elixir: "~> 1.6-dev",
start_permanent: Mix.env == :prod,
deps: deps()
]
end
def deps do
[
{:poison, "~> 3.1"},
{:ex_json_schema, "~> 0.5.4"}
]
end
end
Currently, array
(in JSON) has 1-to-1 correspondence to list
. It is probably right decision for pure JSON validator, although dependent on JSON parser.
I'm considering to use the validator to verify general data structure. In order to do that, I would need to specify that certain piece is tuple. OTOH, proper representation for JSON array
in some cases is tuple too.
What would be your opinion about making tuple support conditional, based on configuration? For example, config ..., tuple_type: true
to allow to specify tuple type as similar to array
, but mapped to tuple, and config ..., array_maps_to: <:list | :tuple | :list_or_tuple >
?
I could try to prepare PR if you are interested.
Thank you.
I'm trying to import a schema from a decoded object but I'm hitting errors.
Here is the json file
[
{
"$id": "user",
"type": "object",
"properties": {
"id": {
"minimum": 0,
"description": "Unique Identifier for this user",
"examples": [
1234
],
"type": "integer"
},
"name": {
"minLength": 2,
"maxLength": 20,
"pattern": "^[A-Za-z0-9_]+$",
"type": "string"
}
},
"required": [
"id",
"name"
]
},
{
"$id": "privateUser",
"allOf": [
{
"$ref": "user"
},
{
"type": "object",
"properties": {
"permissions": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"permissions"
]
}
]
}
]
I started by trying this:
File.read!("schemas.json") |> Jason.decode! |> ExJsonSchema.Schema.resolve
Unfortunately this errors, presumably because it's a list of schemas. My next try was to break them into a list and handle each of them in turn:
"schemas.json"
|> File.read!
|> Jason.decode!
|> Enum.map(fn json_def ->
ExJsonSchema.Schema.resolve(json_def)
end)
This chokes on the privateUser
entry but only if the "$ref": "user"
part was there.
Obviously the ref isn't known to the library at the time so I'm trying to work out what I should be doing.
Cheers for any assistance you can give.
JSON Schema draft 2019-09 supports extending vocabularies.
Is it something you plan to incorporate in ex_json_schema
with some guide/example?
Hi, would you be open to accepting a PR that allows using custom format for non-string types? To the best of my knowledge, this would be a valid non-standard extension. Please let me know if this is a bad idea.
We're specifically looking for enabling custom format support for numbers/integers. We're working with APIs that return timestamps in different formats, including a float with 3 decimal points of precision. We're considering using the format
field to describe that the value has a very specific interpretation, but it would be great to be able to validate it with ExJsonSchema.Validator.validate
. While a lot can be expressed with built-in options, running custom validation on any data gives the comfort of 100% flexibility.
An example schema would look like this:
{
"type": "object",
"properties": {
"foo": {
"type": "integer",
"format": "integer-even"
}
}
}
The format
option is ignored for non-strings (test).
I'd be happy to contribute a PR:
I think the only thing to decide would be the behaviour of built-in formats on non-strings. My take is that they should still be ignored and only custom formats should be allowed.
I'm interested to hear the background on validation error presentation because I'm currently working to improve the way my company's application handles invalid data matched against schemas. ex_json_schema
currently returns validations misses in an error tuple with the following spec:
{:error, [{String.t, String.t | [String.t | integer]}]}
The problem is that the list of errors returned is difficult to act upon since it uses binaries to present the reason. That is, the reason, which is at position 0 in the tuple, isn't guaranteed to have a stable reference. A minimum value violation may have a reason of "Expected the value to be > 4", "Expected the value to be > 5", etc. That is to say, the value will not be the same and therefore can't be matched upon and acted upon intuitively.
In my mind, it would be more useful to have the validation errors be presented using atoms. For example:
{:error,
[
{:minimum_violation, [minimum: 4], "#/foo", "Expected the value to be > 4"},
{:exclusive_minimum_violation, [exclusive_minimum: 5], "#/bar", "Expected the value to be >= 5"},
{:maximum_length_violation, [maximum_length: 25], "#/person/firstName", "Expected value to have a maximum length of 25 but was 38"}
]
}
By presenting validation errors in this fashion, it's easy to have my code match on the atom and then take appropriate action based on that. This includes building custom error strings and creating appropriate exceptions.
(Also, for the same reason, I could see making the reference to the path as something like [:root, "person", "firstName"]
), but that's another matter.
Happy to help out with this, but I wanted to see what others opinions of this was.
When trying to resolve a schema (available here), I get the following error:
** (ExJsonSchema.Schema.InvalidSchemaError) invalid reference #verb#languagemap
(ex_json_schema) lib/ex_json_schema/schema.ex:119: ExJsonSchema.Schema.get_fragment/2
(ex_json_schema) lib/ex_json_schema/schema.ex:110: ExJsonSchema.Schema.resolve_ref/2
(ex_json_schema) lib/ex_json_schema/schema.ex:98: ExJsonSchema.Schema.resolve_property/3
(ex_json_schema) lib/ex_json_schema/schema.ex:73: anonymous fn/3 in ExJsonSchema.Schema.do_resolve/3
(stdlib) lists.erl:1263: :lists.foldl/3
...
Replacing the $ref values w/ #/properties/xxx
instead of #xxx
seems to get me past this issue but others arise.
I'm unsure whether this is a bug, a limitation or something wrong I'm [not] doing?
We have a need to present error messages from ex_json_schema in various languages. Its work I am willing to do, and you should see a PR from me eventually. However, I thought I would get it on your radar to see if you had any comments.
I got an ExJsonSchema.Schema.InvalidReferenceError
when trying to resolve (Draft 04 compliant) Signal-K json schema.
To me it looks like the resolver incorrectly amends the $ref": "aircraft.json#"
to the root "id": "https://signalk.org/specification/1.0.0/schemas/signalk.json#
element instead of replacing the signalk.json#
part in the uri with aircraft.json#
(as described in section 8.3.2 of the json schema spec).
Attached please find my local stackrace:
** (ExJsonSchema.Schema.InvalidReferenceError) invalid reference https://signalk.org/specification/1.0.0/schemas/signalk.json#aircraft.json#
stacktrace:
(ex_json_schema) lib/ex_json_schema/schema.ex:287: ExJsonSchema.Schema.raise_invalid_reference_error/1
(ex_json_schema) lib/ex_json_schema/schema.ex:150: ExJsonSchema.Schema.resolve_ref/2
(ex_json_schema) lib/ex_json_schema/schema.ex:160: ExJsonSchema.Schema.resolve_ref!/2
(ex_json_schema) lib/ex_json_schema/schema.ex:138: ExJsonSchema.Schema.resolve_property/3
(ex_json_schema) lib/ex_json_schema/schema.ex:108: anonymous fn/3 in ExJsonSchema.Schema.do_resolve/3
(stdlib) maps.erl:257: :maps.fold_1/3
(ex_json_schema) lib/ex_json_schema/schema.ex:107: ExJsonSchema.Schema.do_resolve/3
(ex_json_schema) lib/ex_json_schema/schema.ex:116: ExJsonSchema.Schema.resolve_property/3
(ex_json_schema) lib/ex_json_schema/schema.ex:108: anonymous fn/3 in ExJsonSchema.Schema.do_resolve/3
(stdlib) maps.erl:257: :maps.fold_1/3
(ex_json_schema) lib/ex_json_schema/schema.ex:107: ExJsonSchema.Schema.do_resolve/3
(ex_json_schema) lib/ex_json_schema/schema.ex:116: ExJsonSchema.Schema.resolve_property/3
(ex_json_schema) lib/ex_json_schema/schema.ex:108: anonymous fn/3 in ExJsonSchema.Schema.do_resolve/3
(stdlib) maps.erl:257: :maps.fold_1/3
(ex_json_schema) lib/ex_json_schema/schema.ex:107: ExJsonSchema.Schema.do_resolve/3
(ex_json_schema) lib/ex_json_schema/schema.ex:116: ExJsonSchema.Schema.resolve_property/3
(ex_json_schema) lib/ex_json_schema/schema.ex:108: anonymous fn/3 in ExJsonSchema.Schema.do_resolve/3
(stdlib) maps.erl:257: :maps.fold_1/3
(ex_json_schema) lib/ex_json_schema/schema.ex:107: ExJsonSchema.Schema.do_resolve/3
(ex_json_schema) lib/ex_json_schema/schema.ex:116: ExJsonSchema.Schema.resolve_property/3
If an object is provided to be validated that has unexpected type (atom in this case), the validator throws an exception inType.data_type. There should be an else on the cold returning something that indicates an unexpected type.
The following causes an infinite loop on ex_json_schema 0.5.0 but not on the json-schema ruby gem.
{
"$schema": "http://json-schema.org/draft-04/schema#",
"oneOf": [
{ "$ref": "#definitions/invalid" },
{ "$ref": "#definitions/valid" }
],
"type": "object",
"definitions": {
"invalid": {
"additionalProperties": false,
"properties": {
"errors": {
"type": "array",
"items": {
}
}
},
"required": [
"errors"
]
},
"valid": {
"additionalProperties": false,
"properties": {
"results": {
"type": "array",
"items": {
"type": "object",
"required": ["data"],
"properties": {
"data": {
"type": "object",
"required": ["value"]
}
}
}
}
}
}
}
}
Adding /
s in the oneOf
definition fixes this:
"oneOf": [
{ "$ref": "#/definitions/invalid" },
{ "$ref": "#/definitions/valid" }
],
I'm not sure if this lib or the ruby version is incorrect but I thought I'd better report it.
Is it possible to add the line number at which the error occurred? Is there a setting perhaps?
Something similar to https://www.jsonschemavalidator.net/. It highlights the line at which the error has occurred.
I tried to use this library but ran into a little bummer:
Using Poison to decode json directly into the structs i need, like this:
data = Poison.decode!("{foo: {bar: 42}}", as: %{
"foo" => %Foo{},
})
When trying to validate the following schema on that Foo
struct does not fail, but will return true
without checking anything:
schema = ExJsonSchema.Schema.resolve(%{
"type" => "object",
"properties" => %{
"bar" => %{
"type" => "string",
},
},
})
true = ExJsonSchema.Validator.validate(schema, data["foo"])
Is it possible to integrate checking Structs directly? At least letting the validate()
fail for Structs that should be validated?
My workaround is this line to convert my Struct back to a stringed map:
map = for {k, v} <- Map.from_struct(data), into: %{}, do: {Atom.to_string(k), v}
Hello,
First, thanks a lot for providing this library! We @ https://transport.data.gouv.fr are happy to use it daily in production.
Opening an issue to note that dependentRequired
(conditionally requires that certain properties must be present if a given property is present in an object) is not checked by the current version of this library and we got surprised by this.
It may be because you need to support newer versions of the spec, as discussed in #73.
Hello @jonasschmidt,
I'm using ex_json_schema
for validation of an API by the given swagger specification and I'm getting error that:
{:invalid, {"Required property Feild2 was not present.", "#"}}
My schema looks:
"post": {
"parameters": [
{
"name": "FooObject",
"in": "body",
"required": true,
"schema": {
"$ref": "#/definitions/Foo"
}
}],
The Foo
is object with fields where some of them are required and others not:
"Foo": {
"type": "object",
"required": [
"Field1"
],
"properties": {
"Field1": {
"$ref": "#/definitions/Field1"
},
"Field2": {
"$ref": "#/definitions/Field2"
}
}
How can I fix this?
Thank you
Hi!
I guess there's a problem with validator because as it was described in the documention, an instance validates successfully against const
keyword if its value is equal to the value of the keyword, but it seems like it's not working here.
to say more precisely, the validator doesn't care about const
existance and it returns true for every value as input.
for example, isn't it a case that it should return false here?
> ExJsonSchema.Schema.resolve(%{"type" => "string", "const" => "foo"})
|> ExJsonSchema.Validator.valid?("bar")
true
Thank you so much
We get an error when passing a string, when the schema expects a map. It throws the map error seen below rather than the expected validation error.
iex(13)> validate("test1.json", "88888888")
** (BadMapError) expected a map, got: "88888888"
(stdlib) :maps.is_key("id", "88888888")
(ex_json_schema) lib/ex_json_schema/validator.ex:116: anonymous fn/2 in ExJsonSchema.Validator.validate_aspect/4
(elixir) lib/enum.ex:873: anonymous fn/3 in Enum.flat_map/2
(elixir) lib/enum.ex:1473: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:872: Enum.flat_map/2
(elixir) lib/enum.ex:873: anonymous fn/3 in Enum.flat_map/2
(stdlib) lists.erl:1262: :lists.foldl/3
(elixir) lib/enum.ex:872: Enum.flat_map/2
iex(13)> validate("test1.json", %{})
{:error, [{"Required property id was not present.", "#"}]}
test1.json is:
{
"type": "object"
, "$schema": "http://json-schema.org/draft-04/schema"
, "id": "<obsfuscated>"
, "properties": {
"id": {
"$ref": "/dir1/test2.json"
}
, "name": {
"type": "string"
}
}
, "required": ["id"]
}
test2.json is:
{
"$schema": "http://json-schema.org/draft-04/schema"
, "id": "https://api.sunlightpayments.com"
, "type": "string"
, "pattern": "^[0-9]{8}$"
}
I'm no expert at JSON schema, but I think this is valid:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Comment describing your JSON Schema",
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"minLength": 3,
"maxLength": 64
},
"required": {
"type": "boolean"
},
"type": {
"type": "string",
"oneOf": [
{
"$ref": "lib/schemas/types/varchar.schema.json"
},
{
"$ref": "lib/schemas/types/integer.schema.json"
}
]
}
},
"required": [
"name",
"type"
]
}
And then Im using oneOf as means to specify one of a type (database type) with a pattern.
{
"pattern": "^smallint$"
}
And my schema seems to resolve correctly
"$schema" => "http://json-schema.org/draft-07/schema#",
"additionalProperties" => false,
"properties" => %{"function" => %{"type" => "string"}},
"type" => "object"
},
"lib/schemas/object.schema.json" => %{
"$schema" => "http://json-schema.org/draft-07/schema#",
"additionalProperties" => false,
"description" => "Comment describing your JSON Schema",
"properties" => %{
"id" => %{
"description" => "The id of the object on which you would like to perform the action",
"type" => "string"
},
"schema" => %{
"additionalItems" => true,
"items" => [
%{
"oneOf" => [%{"$ref" => ["lib/schemas/values.schema.json"]}],
"type" => "object"
}
],
"type" => "array"
}
},
"required" => ["schema", "id"],
"type" => "object"
},
"lib/schemas/pairs.schema.json" => %{
"$schema" => "http://json-schema.org/draft-07/schema#",
"additionalProperties" => false,
"description" => "Comment describing your JSON Schema",
"properties" => %{
"name" => %{"maxLength" => 64, "minLength" => 3, "type" => "string"},
"required" => %{"type" => "boolean"},
"type" => %{
"oneOf" => [
%{"$ref" => ["lib/schemas/types/varchar.schema.json"]},
%{"$ref" => ["lib/schemas/types/integer.schema.json"]}
],
"type" => "string"
}
},
"required" => ["name", "type"],
"type" => "object"
},
"lib/schemas/table.schema.json" => %{
"$schema" => "http://json-schema.org/draft-07/schema#",
"additionalProperties" => false,
"description" => "Comment describing your JSON Schema",
"properties" => %{
"name" => %{"maxLength" => 64, "minLength" => 5, "type" => "string"},
"parent" => %{"type" => "string"},
"schema" => %{
"additionalItems" => true,
"items" => [
%{
"oneOf" => [%{"$ref" => ["lib/schemas/pairs.schema.json"]}],
"type" => "object"
}
],
"type" => "array"
}
},
"required" => ["name", "parent", "schema"],
"type" => "object"
},
"lib/schemas/types/integer.schema.json" => %{"pattern" => "^integer$"},
"lib/schemas/types/varchar.schema.json" => %{"pattern" => "^varchar$"},
"lib/schemas/values.schema.json" => %{
"$schema" => "http://json-schema.org/draft-07/schema#",
"additionalProperties" => false,
"description" => "Comment describing your JSON Schema",
"properties" => %{
"key" => %{"maxLength" => 64, "minLength" => 3, "type" => "string"},
"value" => %{"maxLength" => 1024, "minLength" => 3, "type" => "string"}
},
"required" => ["key", "value"],
"type" => "object"
}
},
schema: %{
"$schema" => "http://json-schema.org/draft-07/schema#",
"additionalProperties" => false,
"description" => "Comment describing your JSON Schema",
"properties" => %{
"action" => %{
"description" => "The action you wish to perform",
"enum" => ["index", "create", "read", "update", "delete"],
"type" => "string"
},
"body" => %{
"oneOf" => [
%{"$ref" => ["lib/schemas/table.schema.json"]},
%{"$ref" => ["lib/schemas/object.schema.json"]},
%{"$ref" => ["lib/schemas/function.schema.json"]}
],
"type" => "object"
}
},
"required" => ["body"],
"type" => "object"
},
version: 7
}
It seems to validate the first 'type' param that is type string - and then all other values are not validated. I can add "integerrrr" as a type despite pattern.
I'm not sure if this is a bug or not but If you see that Im doing something wrong, please let me know. Otherwise, I hope it's helpful.
The following code passes without problem:
ExJsonSchema.Schema.resolve(
Jason.decode!("""
{
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"a": {
"type": "string"
}
},
"required": "not an array"
}
""")
)
But I guess it should not.
I was trying to debug this, but saw that you have this code :
with false <- meta04?(schema),
false <- meta06?(schema),
false <- meta07?(schema) do
#...
|> ExJsonSchema.Validator.validate(schema, error_formatter: false)
else
_ -> :ok
end
It specifically prevent validation of schemas with a known meta-schema.
But as far as I understand you cannot have a false
in the with
because the following code will throw:
schema_version =
root_schema
|> Map.get("$schema", @current_draft_schema_url <> "#")
|> schema_version!()
(I know I could use the "latest" URL but that does not work for me. I want a defined spec).
What do you think?
I get the following error when validating a string against a schema with type "object":
** (BadMapError) expected a map, got: "boom"
(stdlib) :maps.is_key("name", "boom")
lib/ex_json_schema/validator.ex:116: anonymous fn/2 in ExJsonSchema.Validator.validate_aspect/4
(elixir) lib/enum.ex:873: anonymous fn/3 in Enum.flat_map/2
(elixir) lib/enum.ex:1473: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:872: Enum.flat_map/2
(elixir) lib/enum.ex:873: anonymous fn/3 in Enum.flat_map/2
(stdlib) lists.erl:1262: :lists.foldl/3
(elixir) lib/enum.ex:872: Enum.flat_map/2
This happens when it tries to validate the required keys of the object when it is in fact a string. I expect that the type should be checked first and if it is invalid then no further validation should be done. Does that sound reasonable?
I'm happy to have a go at fixing this in a PR, I just want to understand what the correct behaviour is first.
What if the provided params matches the schema, but contains extra attributes? It could be nice to inform the user that the extra ones will get ignored. Is this something ex_json_schema
could tackle?
Ran into an issue with a coworker using this library. The problem is the library expects the remote schema resolver to be defined as an anonymous function in the application configuration; however, when compiling distribution releases using Erlang release tooling, this type of configuration value is unsupported. Instead, the schema resolver setting should accept accept a value of the form {mod, fun_name}
which then calls, apply(mod, fun_name, [url])
, using a function already present in the codebase. Open to other ideas.
Is there already a way to see what's changed between versions, besides comparing commit ranges manually? I couldn't find a changelog or release notes on the GitHub Releases page.
We've recently encountered some breaking changes in production when upgrading. For example, email address validation is now considering legit domains as invalid.
Hi!
I'm using ExJsonSchema (version 0.8.0-rc1) to validate json configuration files, and in some cases (enum, additionalProperties) I would very much like to display the actually allowed values in the error message.
However, the custom error does not contain the information needed. It does contain the path but it's a bit complicated to get the actual schema fragment from the path, since it may go through many layers of references. Or is there a simple way to get the schema?
So I'm thinking either we can make the Enum, PropertyNames, and AdditionalProperties include the allowed enums and properties/property names/pattern properties, or we can more generally make the schema fragment for which validation failed available.
Just extending the specific errors seems easier. It only involved a small change in the validation modules to include the part of the schema in the error. If you want I can provide a pull request (I did it in my source tree).
However, maybe it's better to extend the generic error to have a :fragment field in addition to the :error and :path fields. That way we can choose for ourselves how to format errors based on the actual schema fragment (even including using custom properties in our json schema). It will also make it easier to make good custom errors for oneOf etc.
Any one of these changes should be pretty backwards compatible since they are just adding fields to the error structs, and those are only read, not written, by user code.
We are hosting the schema itself so it would be nice if the schema domain can be configurable. Now we get the following error:
Unsupported schema version, only draft 4, 6, and 7 are supported.
If we host it on for example: http://json-schema.test.docker/draft-07/schema
Suggested code at lib/ex_json_schema/schema.ex#L37
@current_draft_schema_url Application.get_env(:ex_json_schema, :schema_host, "http://json-schema.org/schema")
@draft4_schema_url @current_draft_schema_url <> "/draft-04/schema"
@draft6_schema_url @current_draft_schema_url <> "/draft-06/schema"
@draft7_schema_url @current_draft_schema_url <> "/draft-07/schema"
Thanks in advance
Running validation emits the following warning message:
warning: Map.take/2 with an Enumerable of keys that is not a list is deprecated. Use a list of keys instead.
(elixir) lib/map.ex:395: Map.take/2
(ex_json_schema) lib/ex_json_schema/validator/properties.ex:12: ExJsonSchema.Validator.Properties.validate/3
This seems to be the offending line:
We found there is a IO.inspect
in the code pushed to hex somewhere in the lib. We can't know where since it's not in Github.
In case it helps, we run it through: https://diff.jola.dev/diff/ex_json_schema/0.7.1/0.7.2 and we found it was in the formatter.
Thanks
I came accross the following issue while testing a response from a server we are developing,
{
"$schema": "http://json-schema.org/schema#",
"type": "object",
"properties": {
"getSampleResponse": {
"required": [
"samples"
],
"type": "object",
"properties": {
"samples": {
"items": {
"type": "object",
"properties": {
"dateField": {
"type": "string",
"format": "date"
},
"stringField": {
"type": "string"
}
}
},
"type": "array"
}
}
}
}
}
{
"getSampleResponse": {
"samples": [],
"samples": {
"dateField": "2018-03-19",
"stringField": "USD"
}
}
}
Notice how the output has both a samples array (empty) and a sample object (which really should be an instance of the items json nested within the array), validating the above is succesful.
case ExJsonSchema.Validator.validate(schema, data) do
:ok -> {:ok, options, []}
{:error, list} -> pretty_print(list)
end
When specifying a schema:
schema = %{
"type" => "object",
"properties" => %{
"foo" => %{
"type" => "string"
}
}
} |> ExJsonSchema.Schema.resolve
Validating it against data that contains field not declared in the schema returns :ok
> ExJsonSchema.Validator.validate(schema, %{"foo" => "bar", "baz" => "bing"})
:ok
Is there a way to validate "strictly" ? I tried adding the "additionalProperties" field to the schema, but it does not change anything (maybe it's in a later version of the spec than the one supported ?)
I ran several validations against my schemata and found some problems related to for errors of anyOf
keyword. For example I have the following rule inside my schema:
"included": {
"type": "array",
"items": {
"anyOf": [
{ "$ref": "data/image-data.json" },
{ "$ref": "data/partner-data.json" }
]
}
}
refs can contain a lot of additional validation rules. But when I perform json validation against invalid data it says only:
{"Expected any of the schemata to match but none did.", "#/included/0"}
It may be better to list errors against each of validation object (in my case 2 lists).
I don't use oneOf
or allOf
keywords but suppose that they have the same behaviour.
I can make a PR that adds more info about these errors
The JSON schema spec states that the format
property "allows values to be constrained beyond what the other tools in JSON Schema can do". To support this, you can configure a callback validator function which gets called when a format
property is encountered that is not one of the builtin formats:
config :ex_json_schema,
:custom_format_validator,
{MyModule, :my_validator}
The configured function will be called with the arguments (format, data)
and is expected to return a list of %ExJsonSchema.Validator.Error{}
structs, or an empty list in case of a valid format.
ExJsonSchema.Schema.resolve/1 appears to support referencing separate schema files via $ref, e.g.
Load & resolve schema.json, which includes:
"stuff": { "$ref": "sub-schema.json#/stuff" }
But doesn't seem to support multi-level references, e.g.
Where sub-schema.json referenced in schema.json includes:
"morestuff": { "$ref": "sub-sub-schema.json#/morestuff" }
Any plans to include support for that?
warning: Dict.values/1 is deprecated, use the Map module for working with maps or the Keyword module for working with keyword lists
lib/ex_json_schema/validator/properties.ex:55
warning: Set.difference/2 is deprecated, use the MapSet module for working with sets
lib/ex_json_schema/validator/properties.ex:64
warning: Dict.keys/1 is deprecated, use the Map module for working with maps or the Keyword module for working with keyword lists
lib/ex_json_schema/validator/properties.ex:69
warning: HashSet.new/0 is deprecated, use the MapSet module instead
lib/ex_json_schema/validator/properties.ex:69
warning: Dict.values/1 is deprecated, use the Map module for working with maps or the Keyword module for working with keyword lists
lib/ex_json_schema/validator.ex:217
warning: variable "remote_schema_resolver" does not exist and is being expanded to "remote_schema_resolver()", please use parentheses to remove the ambiguity or change the variable name
lib/ex_json_schema/schema.ex:169
warning: Map.take/2 with an Enumerable of keys that is not a list is deprecated. Use a list of keys instead.
(elixir 1.11.4) lib/map.ex:413: Map.take/2
(ex_json_schema 0.5.8) lib/ex_json_schema/validator/properties.ex:12: ExJsonSchema.Validator.Properties.validate/3
(elixir 1.11.4) lib/enum.ex:1075: anonymous fn/3 in Enum.flat_map/2
(stdlib 3.14.1) maps.erl:233: :maps.fold_1/3
(elixir 1.11.4) lib/enum.ex:2209: Enum.flat_map/2
(ex_json_schema 0.5.8) lib/ex_json_schema/validator.ex:28: ExJsonSchema.Validator.validate/4
(elixir 1.11.4) lib/enum.ex:1075: anonymous fn/3 in Enum.flat_map/2
(stdlib 3.14.1) maps.erl:233: :maps.fold_1/3
(elixir 1.11.4) lib/enum.ex:2209: Enum.flat_map/2
(ex_json_schema 0.5.8) lib/ex_json_schema/validator.ex:28: ExJsonSchema.Validator.validate/4
(elixir 1.11.4) lib/enum.ex:1075: anonymous fn/3 in Enum.flat_map/2
(stdlib 3.14.1) maps.erl:233: :maps.fold_1/3
(elixir 1.11.4) lib/enum.ex:2209: Enum.flat_map/2
(ex_json_schema 0.5.8) lib/ex_json_schema/validator/properties.ex:9: ExJsonSchema.Validator.Properties.validate/3
(elixir 1.11.4) lib/enum.ex:1075: anonymous fn/3 in Enum.flat_map/2
(stdlib 3.14.1) maps.erl:233: :maps.fold_1/3
(elixir 1.11.4) lib/enum.ex:2209: Enum.flat_map/2
(ex_json_schema 0.5.8) lib/ex_json_schema/validator.ex:28: ExJsonSchema.Validator.validate/4
The following test succeed due to typo.
assert valid?(%{"typee" => "integer" }, "a") == true
What is the correct way to prevent typo ?
ex_json_schema
?I have the following JSON schema
%ExJsonSchema.Schema.Root{
custom_format_validator: nil,
definitions: %{},
location: :root,
refs: %{
"lib/schemas/column.schema.json" => %{
"$schema" => "http://json-schema.org/draft-07/schema#",
"additionalProperties" => false,
"anyOf" => [
%{
"properties" => %{
"maxLength" => %{
"maximum" => 64,
"minimum" => 3,
"type" => "number"
},
"type" => %{"pattern" => "^varchar$", "type" => "string"}
},
"type" => "object"
},
%{
"properties" => %{
"type" => %{"pattern" => "^smallint$", "type" => "string"}
},
"type" => "object"
},
%{
"properties" => %{
"type" => %{"pattern" => "^integer$", "type" => "string"}
},
"type" => "object"
},
%{
"properties" => %{
"type" => %{"pattern" => "^bigint$", "type" => "string"}
},
"type" => "object"
},
%{
"properties" => %{
"type" => %{"pattern" => "^decimal$", "type" => "string"}
},
"type" => "object"
},
%{
"properties" => %{
"type" => %{"pattern" => "^numeric$", "type" => "string"}
},
"type" => "object"
},
%{
"properties" => %{
"type" => %{"pattern" => "^real$", "type" => "string"}
},
"type" => "object"
},
%{
"properties" => %{
"type" => %{"pattern" => "^double precision$", "type" => "string"}
},
"type" => "object"
},
%{
"properties" => %{
"type" => %{"pattern" => "^serial$", "type" => "string"}
},
"type" => "object"
},
%{
"properties" => %{
"type" => %{"pattern" => "^bigserial$", "type" => "string"}
},
"type" => "object"
}
],
"description" => "Comment describing your JSON Schema",
"properties" => %{
"name" => %{"maxLength" => 64, "minLength" => 3, "type" => "string"},
"required" => %{"type" => "boolean"}
},
"required" => ["name"],
"type" => "object"
},
"lib/schemas/incident.schema.json" => %{
"$schema" => "http://json-schema.org/draft-07/schema#",
"additionalProperties" => false,
"allOf" => [
%{
"additionalProperties" => false,
"properties" => %{
"name" => %{"type" => "string"},
"stuff" => %{"type" => "number"}
},
"type" => "object"
}
],
"items" => false,
"properties" => %{},
"type" => "object"
},
"lib/schemas/table.schema.json" => %{
"$schema" => "http://json-schema.org/draft-07/schema#",
"additionalProperties" => false,
"description" => "Comment describing your JSON Schema",
"properties" => %{
"name" => %{"maxLength" => 64, "minLength" => 5, "type" => "string"},
"parent" => %{"type" => "string"},
"schema" => %{
"items" => %{"$ref" => ["lib/schemas/column.schema.json"]},
"type" => "array"
}
},
"required" => ["name"],
"type" => "object"
}
},
schema: %{
"$schema" => "http://json-schema.org/draft-07/schema#",
"description" => "Comment describing your JSON Schema",
"properties" => %{
"action" => %{
"description" => "The action you wish to perform",
"enum" => ["create", "read", "update", "delete"],
"type" => "string"
},
"body" => %{
"oneOf" => [
%{"$ref" => ["lib/schemas/table.schema.json"]},
%{"$ref" => ["lib/schemas/incident.schema.json"]}
],
"type" => "object"
},
"id" => %{
"description" => "The id of the object on which you woud like to perform the action",
"type" => "string"
}
},
"required" => ["body"],
"type" => "object"
},
version: 7
}```
And I have the following payload
```iex(86)> payload
%{
"action" => "create",
"body" => %{
"name" => "incident",
"parent" => "e5cb0b35-7b30-4613-83f9-31f5d7007b12",
"schema" => [
%{
"maxLength" => 64,
"name" => "name",
"required" => true,
"type" => "varchar"
},
%{"name" => "stuff", "type" => "smallint"}
]
}
}
iex(87)>```
Im getting this error when I try validate.
```{:error,
[
{"Expected exactly one of the schemata to match, but none of them did.",
"#/body"}
]}
iex(89)>```
Hi, well done with this module, what about #38 PR?
It would be nice to have this feature. Also actually the error format don't let easly (from the code viewpoint) know what is going on with the validation errors, for example:
{:error, [{"Type mismatch. Expected String but got Integer.", "#/id"}]}
Why there's always a "#/" prefix?
Was wondering if you had any plans to support the 2020 schema ?
Of the following tests, the first passes, and the remaining 2 do not. They look valid to me, but appear to indicate the validator does not handle properties that are arrays of objects properly.
test "array ref test 2" do
schema = %{
"type" => "array",
"items" => %{"$ref" => "#/definitions/item"},
"definitions" => %{
"item" => %{
"type" => "object",
"properties" => %{
"uid" => %{"type" => "string"}
},
"required" => ["uid"],
"additionalProperties" => false
}
}
} |> ExJsonSchema.Schema.resolve
json = [%{uid: "test", junk: 1}]
{:error, [{msg, var}]} = ExJsonSchema.Validator.validate(schema, string_keys(json))
assert Regex.run(~r/junk/,var) != nil
assert Regex.run(~r/not allow additional/,msg) != nil
end
test "array ref test 3" do
schema = %{
"type" => "object",
"properties" => %{
"tests" => %{
"type" => "array",
"items" => %{"$ref" => "#/definitions/item"}
},
},
"definitions" => %{
"item" => %{
"type" => "object",
"properties" => %{
"uid" => %{
"type" => "string"
}
},
"required" => ["uid"],
"additionalProperties" => false
}
}
} |> ExJsonSchema.Schema.resolve
json = %{tests: [%{uid: "test"}]}
:ok = ExJsonSchema.Validator.validate(schema, string_keys(json))
end
test "array nesting test 4" do
schema = %{
"type" => "object",
"properties" => %{
"tests" => %{
"type" => "array",
"items" => %{
"type" => "object",
"properties" => %{
"uid" => %{
"type" => "string"
}
},
"required" => ["uid"],
"additionalProperties" => false
}
},
},
} |> ExJsonSchema.Schema.resolve
json = %{tests: [%{uid: "test"}]}
:ok = ExJsonSchema.Validator.validate(schema, string_keys(json))
end
Test output:
Hi,
Erlang/OTP 22 [erts-10.4.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]
Interactive Elixir (1.9.0) - press Ctrl+C to exit (type h() ENTER for help)
With elixir 1.9 comes lot of deprecations.
One of them is : warning: Map.take/2 with an Enumerable of keys that is not a list is deprecated. Use a list of keys instead.
The consequence is the follow :
in ex_json_schema/lib/ex_json_schema/validator/properties.ex
line 73: Map.take(properties, unvalidated)
unvalidated
is a MapSet that is now deprecated
one solution could be to do : Map.take(properties, MapSet.to_list(unvalidated))
Other thing,
in ex_json_schema/lib/ex_json_schema/validator.ex
line 182, 189, 197, 204: Map.size
is deprecated
It should to be replaced by map_size
(Kernel.map_size)
The consequence of this deprecation. My log is filled with bunch of warning
I tried do clone the project, fix the issue, but when I try to test it, I have :
Finished in 1.0 seconds
484 tests, 6 failures
Sorry, i don't have more time do spend on this issue.
git diff :
diff --git a/lib/ex_json_schema/validator.ex b/lib/ex_json_schema/validator.ex
index 870d91b..017b310 100644
--- a/lib/ex_json_schema/validator.ex
+++ b/lib/ex_json_schema/validator.ex
@@ -179,14 +179,14 @@ defmodule ExJsonSchema.Validator do
end
defp validate_aspect(_, _, {"minProperties", min_properties}, data) when is_map(data) do
- case Map.size(data) >= min_properties do
+ case map_size(data) >= min_properties do
true ->
[]
false ->
[
%Error{
- error: %Error.MinProperties{expected: min_properties, actual: Map.size(data)},
+ error: %Error.MinProperties{expected: min_properties, actual: map_size(data)},
path: ""
}
]
@@ -194,14 +194,14 @@ defmodule ExJsonSchema.Validator do
end
defp validate_aspect(_, _, {"maxProperties", max_properties}, data) when is_map(data) do
- case Map.size(data) <= max_properties do
+ case map_size(data) <= max_properties do
true ->
[]
false ->
[
%Error{
- error: %Error.MaxProperties{expected: max_properties, actual: Map.size(data)},
+ error: %Error.MaxProperties{expected: max_properties, actual: map_size(data)},
path: ""
}
]
diff --git a/lib/ex_json_schema/validator/properties.ex b/lib/ex_json_schema/validator/properties.ex
index b5882de..033ca2d 100644
--- a/lib/ex_json_schema/validator/properties.ex
+++ b/lib/ex_json_schema/validator/properties.ex
@@ -70,7 +70,7 @@ defmodule ExJsonSchema.Validator.Properties do
defp unvalidated_properties(properties, validated_properties) do
unvalidated = MapSet.difference(keys_as_set(properties), keys_as_set(validated_properties))
- Map.take(properties, unvalidated)
+ Map.take(properties, MapSet.to_list(unvalidated))
end
defp keys_as_set(properties) do
diff --git a/test/JSON-Schema-Test-Suite b/test/JSON-Schema-Test-Suite
index 1bd999a..9cbad89 160000
--- a/test/JSON-Schema-Test-Suite
+++ b/test/JSON-Schema-Test-Suite
@@ -1 +1 @@
-Subproject commit 1bd999ac16bd8d3fdb5c44ef13a0759aefb4ab73
+Subproject commit 9cbad896454d42fb591ffb4ed3b452a5c12ec63a
My schema expects the team_name
field to be a string
field.
schema = %{
"type" => "object",
"properties" => %{
"team_name" => %{"type" => "string"},
"matches" => %{
"type" => "array",
"properties" => %{
"start_at" => %{"type" => "string"},
"home" => %{"type" => "string"},
"guest" => %{"type" => "string"}
}
}
}
}
But the given json string has no team_name
field.
json_b = "{
\"matches\": [
{\"start_at\": \"Sonntag, 13.03.2016 - 12:00 Uhr\",
\"home\": \"Opponent Team 1\",
\"guest\": \"My Team\"
}, {
\"start_at\": \"Sonntag, 03.04.2016 - 14:00 Uhr\",
\"home\": \"My Team\",
\"guest\": \"Opponent Team 2\"
}
]
}"
I decode the string and get a map.
{:ok, map_b} = JSON.decode(json_b)
# {:ok, %{
# "matches" => [
# %{
# "guest" => "My Team",
# "home" => "Opponent Team 1",
# "start_at" => "Sonntag, 13.03.2016 - 12:00 Uhr"
# },
# %{
# "guest" => "Opponent Team 2",
# "home" => "My Team",
# "start_at" => "Sonntag, 03.04.2016 - 14:00 Uhr"
# }
# ]
# }
I validate the map and get :ok
.
schema |> ExJsonSchema.Schema.resolve |> ExJsonSchema.Validator.validate(map_b)
# :ok
I expect to get an :error
. There's something wrong.
Elixir 1.2.2
ex_json_schema 0.3.1
So I have a schema which uses the date-time
format setting.
Given an input 2019-02-28T10:08:30.000Z
ExJsonSchema.Validator.validate
returns :ok
.
Given an input 2019-02-28t10:08:30.000z
ExJsonSchema.Validator.validate
returns {:error, [{"Expected \"2019-02-28t10:08:30.000z\" to be a valid ISO 8601 date-time.", "#/requestTimestamp"}]}
.
RFC 3339 states that this is valid syntax https://tools.ietf.org/html/rfc3339#section-5.6.
And JSON Schema Test Suite has the following test for this:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.