api7 / jsonschema Goto Github PK
View Code? Open in Web Editor NEWPure Lua JSON schema validator for Lua/LuaJIT
Home Page: https://www.apiseven.com/
License: Apache License 2.0
Pure Lua JSON schema validator for Lua/LuaJIT
Home Page: https://www.apiseven.com/
License: Apache License 2.0
Hello
While running make test
on a OSX system, the command failed with messages below:
LUA_PATH="./lib/?.lua;./deps/lib/lua/5.1/?.lua;./deps/share/lua/5.1/?.lua;;" LUA_CPATH="./deps/lib/lua/5.1/?.so;;" resty t/draft4.lua
2020/07/29 09:34:23 [warn] 99289#11162360: *2 [lua] _G write guard:12: writing a global lua variable ('posix') which may lead to race conditions between concurrent requests, so prefer the use of 'local' variables
stack traceback:
[C]: at 0x0368d9e0
[C]: in function 'require'
t/draft4.lua:4: in function 'file_gen'
init_worker_by_lua:45: in function <init_worker_by_lua:43>
[C]: in function 'xpcall'
init_worker_by_lua:52: in function <init_worker_by_lua:50>, context: ngx.timer
ERROR: t/draft4.lua:10: attempt to call upvalue 'clock_gettime' (a nil value)
stack traceback:
t/draft4.lua:104: in function 'file_gen'
init_worker_by_lua:45: in function <init_worker_by_lua:43>
[C]: in function 'xpcall'
init_worker_by_lua:52: in function <init_worker_by_lua:50>
make: *** [test] Error 1
According to https://github.com/luaposix/luaposix/releases/tag/v34.0.4 it seems that clock_gettime was not supported on macOS. Should we replace clock_gettime
with ngx.now
to record the execution time?
wrong case:
local schema = {
type = "object",
oneOf = {
{
title = "work with consumer object",
properties = {
access_key = {type = "string", minLength = 1, maxLength = 256},
secret_key = {type = "string", minLength = 1, maxLength = 256},
algorithm = {
type = "string",
enum = {"hmac-sha1", "hmac-sha256", "hmac-sha512"},
default = "hmac-sha256"
},
clock_skew = {
type = "integer",
default = 300
},
signed_headers = {
type = "array",
items = {
type = "string",
minLength = 1,
maxLength = 20,
}
},
},
required = {"access_key", "secret_key"},
additionalProperties = false,
},
},
{
title = "work with route or service object",
properties = {},
additionalProperties = false,
}
}
local validator = jsonschema.generate_validator(schema)
local conf = {}
local ok = validator(conf)
if not ok then
ngx.say("value checking failure")
return
end
If I change the order of items of the schema object , it will be ok:
local schema = {
type = "object",
oneOf = {
{
title = "work with route or service object",
properties = {},
additionalProperties = false,
},
{
title = "work with consumer object",
properties = {
access_key = {type = "string", minLength = 1, maxLength = 256},
secret_key = {type = "string", minLength = 1, maxLength = 256},
algorithm = {
type = "string",
enum = {"hmac-sha1", "hmac-sha256", "hmac-sha512"},
default = "hmac-sha256"
},
clock_skew = {
type = "integer",
default = 300
},
signed_headers = {
type = "array",
items = {
type = "string",
minLength = 1,
maxLength = 20,
}
},
},
required = {"access_key", "secret_key"},
additionalProperties = false,
},
},
}
local validator = jsonschema.generate_validator(schema)
local conf = {}
local ok = validator(conf)
if not ok then
ngx.say("value checking failure")
return
end
In the wrong case, conf
had been change to {"clock_skew": 300}.
I think this is the reason.
I found this code in jsonschema.lua
if schema.required and #schema.required == 0 then
-- return false if the input data is not empty
ctx:stmt(sformat('if %s ~= 1 then', datakind))
ctx:stmt( ' return false, "the input data should be an empty table"')
ctx:stmt( 'end')
end
if this, when I have a schema
{
"type": "object",
"properties": {
"name": { "type": "string" }
},
"required": []
}
and a json object
{
"name": "Tom"
}
such will return an err: the input data should be an empty table
This way is clear and simple.
we can take a look at this commit:
I am reading a schema from open api specification file which contains internal schema references like "$ref": "#/components/schemas/decimal" but when I try to generate validator with jsonschema.generate_validator, it gives me this error "reference not found: root:#/components/schemas/AddFundsTransferRq (at "components"). I also tried using external_resolver but it didn't work.
Will this be supported?
In the following example, we combine "$ref" with another key, here "properties".
Expected result: Validation returns "false" as modelVersion.major is > 2.
Observed result: Validation returns "true"
local schema = [[
{
"type": "object",
"$defs": {
"version": {
"type": "object",
"properties": {
"major": {
"type": "integer",
"minimum": 0
},
"minor": {
"type": "integer",
"minimum": 0
},
"patch": {
"type": "integer",
"minimum": 0
}
},
"required": [
"major",
"minor",
"patch"
]
}
},
"properties": {
"modelVersion": {
"$ref": "#/$defs/version",
"properties": {
"major": {
"type": "integer",
"minimum": 0,
"maximum": 1
},
"minor": {
"type": "integer",
"minimum": 0
},
"patch": {
"type": "integer",
"minimum": 0
}
}
}
}
}
]]
local v = JsonSchema.generate_validator(Json.decode(schema), {match_pattern = function () end})
print(v(Json.decode([[
{
"modelVersion": {"major": 2, "minor": 2, "patch": 0}
}
]])))
content阶段执行的脚本
local jsonschema = require 'resty.jsonschema'
local myvalidator = jsonschema.generate_validator(
{
type = "object",
required = {
"user",
"pwd",
"rule",
},
properties = {
user = { type = "string" },
pwd = { type = "string" },
rule = {
type = "array",
items = {
type = "object",
required = { "key", "upstream" },
properties = {
key = { type = "string" },
upstreamid = { type = "integer" },
}
}
}
}
}
)
ngx.log(ngx.ERR, 'CHECK:', myvalidator({
user = "xuzz",
pwd = "123456",
rule = {
{ key = "x_"},
{ key = "x_2", upstream = "up_2" },
}
}))
执行访问
curl -v http://172.16.1.232:41000/test
* About to connect() to 172.16.1.232 port 41000 (#0)
* Trying 172.16.1.232...
* Connected to 172.16.1.232 (172.16.1.232) port 41000 (#0)
> GET /test HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.16.1.232:41000
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Server: openresty/1.15.8.1
< Date: Thu, 31 Oct 2019 08:02:34 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 183
< Connection: close
<
<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>
nginx错误日志:
2019/10/31 16:09:01 [error] 30599#30599: *38 lua entry thread aborted: runtime error: [string "jsonschema:anonymous"]:2: attempt to index global 'lib' (a nil value)
stack traceback:
coroutine 0:
[string "jsonschema:anonymous"]: in function 'loader'
lualib/resty/jsonschema.lua:193: in function 'generate_validator'
Hi,
I'm using lua 5.3 and I was following the instruction in the readme file.
luarocks install lrexlib-pcre
luarocks install net_url
make dev
make test
Makefile:39: warning: overriding recipe for target 'help'
Makefile:11: warning: ignoring old recipe for target 'help'
LUA_PATH="./lib/?.lua;/usr/local/share/lua/5.3/?.lua;/usr/local/share/lua/5.3/?/init.lua;/usr/local/lib/lua/5.3/?.lua;/usr/local/lib/lua/5.3/?/init.lua;./?.lua;./?/init.lua" LUA_CPATH="/usr/local/lib/lua/5.3/?.so;/usr/local/lib/lua/5.3/loadall.so;./?.so" resty t/draft4.lua
ERROR: t/draft4.lua:4: module 'posix.time' not found:
no field package.preload['posix.time']
no file '/usr/local/openresty/site/lualib/posix/time.ljbc'
hello
after upgrade to v0.9 release, make test
failed with uniqueItems validation, messages below:
LUA_PATH="./lib/?.lua;./deps/lib/lua/5.1/?.lua;./deps/share/lua/5.1/?.lua;;" LUA_CPATH="./deps/lib/lua/5.1/?.so;;" resty t/draft4.lua
skip suite case: [object type matches objects] -> [an array is not an object]
skip suite case: [array type matches arrays] -> [an object is not an array]
skip suite case: [required validation] -> [ignores arrays]
skip suite case: [regexes are not anchored by default and are case sensitive] -> [recognized members are accounted for]
skip suite case: [minProperties validation] -> [ignores arrays]
skip suite case: [exclusiveMinimum validation] -> [above the minimum is still valid]
skip suite case: [exclusiveMinimum validation] -> [boundary point is invalid]
skip suite case: [exclusiveMaximum validation] -> [below the maximum is still valid]
skip suite case: [exclusiveMaximum validation] -> [boundary point is invalid]
testcases loaded: 0.01800012588501
validate res: true err:
case: {"valid":false,"data":[{"foo":"bar"},{"foo":"bar"}],"description":"non-unique array of objects is invalid"} suite: uniqueItems validation
validate res: true err:
case: {"valid":false,"data":[{"foo":{"bar":{"baz":true}}},{"foo":{"bar":{"baz":true}}}],"description":"non-unique array of nested objects is invalid"} suite: uniqueItems validation
validate res: true err:
case: {"valid":false,"data":[["foo"],["foo"]],"description":"non-unique array of arrays is invalid"} suite: uniqueItems validation
validate res: true err:
case: {"valid":false,"data":[{},[1],true,null,{},1],"description":"non-unique heterogeneous types are invalid"} suite: uniqueItems validation
validations: 0.002000093460083
LUA_PATH="./lib/?.lua;./deps/lib/lua/5.1/?.lua;./deps/share/lua/5.1/?.lua;;" LUA_CPATH="./deps/lib/lua/5.1/?.so;;" resty t/draft6.lua
skip suite case: [object type matches objects] -> [an array is not an object]
skip suite case: [array type matches arrays] -> [an object is not an array]
skip suite case: [required validation] -> [ignores arrays]
skip suite case: [regexes are not anchored by default and are case sensitive] -> [recognized members are accounted for]
skip suite case: [minProperties validation] -> [ignores arrays]
skip suite case: [propertyNames validation] -> [some property names invalid]
skip suite case: [contains keyword validation] -> [not array is valid]
testcases loaded: 0.023000001907349
validate res: true err:
case: {"valid":false,"data":[{"foo":"bar"},{"foo":"bar"}],"description":"non-unique array of objects is invalid"} suite: uniqueItems validation
validate res: true err:
case: {"valid":false,"data":[{"foo":{"bar":{"baz":true}}},{"foo":{"bar":{"baz":true}}}],"description":"non-unique array of nested objects is invalid"} suite: uniqueItems validation
validate res: true err:
case: {"valid":false,"data":[["foo"],["foo"]],"description":"non-unique array of arrays is invalid"} suite: uniqueItems validation
validate res: true err:
case: {"valid":false,"data":[{},[1],true,null,{},1],"description":"non-unique heterogeneous types are invalid"} suite: uniqueItems validation
validations: 0.003000020980835
LUA_PATH="./lib/?.lua;./deps/lib/lua/5.1/?.lua;./deps/share/lua/5.1/?.lua;;" LUA_CPATH="./deps/lib/lua/5.1/?.so;;" resty t/draft7.lua
skip suite case: [object type matches objects] -> [an array is not an object]
skip suite case: [array type matches arrays] -> [an object is not an array]
skip suite case: [required validation] -> [ignores arrays]
skip suite case: [regexes are not anchored by default and are case sensitive] -> [recognized members are accounted for]
skip suite case: [minProperties validation] -> [ignores arrays]
skip suite case: [propertyNames validation] -> [some property names invalid]
skip suite case: [contains keyword validation] -> [not array is valid]
testcases loaded: 0.024999856948853
validate res: true err:
case: {"valid":false,"data":[{"foo":"bar"},{"foo":"bar"}],"description":"non-unique array of objects is invalid"} suite: uniqueItems validation
validate res: true err:
case: {"valid":false,"data":[{"foo":{"bar":{"baz":true}}},{"foo":{"bar":{"baz":true}}}],"description":"non-unique array of nested objects is invalid"} suite: uniqueItems validation
validate res: true err:
case: {"valid":false,"data":[["foo"],["foo"]],"description":"non-unique array of arrays is invalid"} suite: uniqueItems validation
validate res: true err:
case: {"valid":false,"data":[{},[1],true,null,{},1],"description":"non-unique heterogeneous types are invalid"} suite: uniqueItems validation
validations: 0.002000093460083
LUA_PATH="./lib/?.lua;./deps/lib/lua/5.1/?.lua;./deps/share/lua/5.1/?.lua;;" LUA_CPATH="./deps/lib/lua/5.1/?.so;;" resty t/default.lua
passed: table value as default value
passed: check uniqueItems array
After reviewing this PR, unique_item_in_array
uses hash to check unique items and that's very fast. But this not works for table item
cause tables have different memory address. Should we downgrade to the origin deepeq
method for the table item
parts in array?
Hi,
jsonschema uses the function loadstring(). This function has been deprecated in Lua 5.2 and is not available in Lua >= 5.3.
The load() function can be used as it accepts the same arguments as loadstring().
Can you adapt the code so that it works with the different Lua 5.X versions ?
Regards
While looking at docs I noticed that I can pass a custom match_pattern
implementation, but I also see that there are a few validations (eg. email, IPv4) that depend on it being PCRE-compatible.
Also, docs describe this parameter as:
function called to match patterns, defaults to string.find.
while the code tries hard to use PCRE:
string.find
is used only as a last resort, but then validators that depend on PCRE (eg. email, IPv4) will break.
It's not a problem for me, just reporting what I found.
-- data
local uris = {}
for i = 1, 1000 * 2.5 * 1.5 * 20 do
uris[i] = "/api/o2/open/pre_auqz/mmmm_auqz" .. i
end
-- schema
{
type = "array",
items = {
description = "HTTP uri",
type = "string",
},
uniqueItems = true,
}
jsonschema/lib/jsonschema/store.lua
Line 100 in 32e8c1f
should be
if type(new) == "nil" then
or just
if new == nil then
function validatorlib.utf8_len(s)
local c, j=0, 1
while j <= #s do
local cb = str_byte(s, j)
if cb >= 0 and cb <= 127 then j = j + 1
elseif cb >= 192 and cb <= 223 then j = j + 2
elseif cb >= 224 and cb <= 239 then j = j + 3
elseif cb >= 240 and cb <= 247 then j = j + 4
end
c = c + 1
end
return c
end
input "%ff", after urldecode ,str_byte(s,j), when cb > 247,#s = 1, j = 1 , will cause endless loop
Hello, @spacewander ,
The problem of 200 local variables limitation often occur to us when the json schema file geting bigger.
error info like:
main function has more than 200 local variables
I found 2 relative record from other repo:
can we do this workaround too? thanks.
In jsonschema we have not
keyword, but how can I use it in Lua?
For example, We have jsonschema:
"dependencies": {
"field_2": { "not": { "required": ["field_3"] } },
"field_3": { "not": { "required": ["field_2"] } }
}
How do we represent it in Lua, I try like below but it doesn't work:
dependencies = {
field_2 = {
["not"] = {
required = {
"field_3"
}
}
},
field_3 = {
["not"] = {
required = {
"field_2"
}
}
},
}
Can someone help me?
This jsonschema is depends upon one of my module.
While compiling that I am getting following error.
The unauthenticated git protocol on port 9418 is no longer supported.
No more unauthenticated git https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git
So Need to update the rockspec in https://github.com/api7/jsonschema/blob/master/rockspec/jsonschema-0.9.5-0.rockspec#L4 to https://github.com/api7/jsonschema/archive/refs/tags/v0.9.5.tar.gz
Updating older release rockspec files also recommented.
I'm using a validator to check my schemas against JSON Schema Draft-07 schema (http://json-schema.org/draft-07/schema), and found that validation method incorrectly applies default
to validated table.
Reproduction:
local cjson = require "cjson"
local jsonschema = require "jsonschema"
-- simplified metaschema, stripped so that it contains only elements that generate invalid valdiator code
local metaschema = [[
{
"definitions": {
"schemaArray": {
"type": "array",
"minItems": 1,
"items": { "$ref": "#" }
}
},
"properties": {
"items": {
"anyOf": [
{ "$ref": "#" },
{ "$ref": "#/definitions/schemaArray" }
],
"default": true
}
}
}
]]
local metaschema_validator = jsonschema.generate_validator(cjson.decode(metaschema))
local string_array_schema = {
type = "array",
items = { type = "string", minLength = 1 }
}
print("BEFORE> ", cjson.encode(string_array_schema))
metaschema_validator(string_array_schema)
print("AFTER> ", cjson.encode(string_array_schema))
local string_array_validator = jsonschema.generate_validator(string_array_schema)
print(string_array_validator({ "" }))
Result:
BEFORE> {"items":{"minLength":1,"type":"string"},"type":"array"}
AFTER> {"items":{"items":true,"minLength":1,"type":"string"},"type":"array"}
true
Adding "items":true
is invalid here and breaks source schema - it makes it accept invalid inputs (the last output line should say falsefailed to validate item 1: string too short, expected at least 1, got 0
).
This error is caused by handing of default values added in 66242c3. Generated code looks like this:
-- we're validating "items" object, i.e. {"minLength":1,"type":"string"}
local propvalue = p_1["items"]
if propvalue ~= nil then
-- nested validation is skipped because there is no nested "items" key
-- ...
end
if propvalue == nil then
-- and here validator breaks valdiated object
p_1["items"] = true
end
https://github.com/api7/jsonschema/search?q=sctrictly
sctrictly -> strictly
welcome PR to fix this issue
In the following example, "default" fields have wrong type. Yet, validator gives a positive result.
local schema = [[
{
"$schema": "https://json-schema.org/draft/2019-09/schema#",
"type": "object",
"$defs": {
"parameter": {
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": [
"string",
"bool",
"int",
"real"
]
}
},
"required": [
"type"
],
"allOf": [
{
"if": {
"properties": {
"type": {
"const": "string"
}
}
},
"then": {
"$ref": "#/$defs/parameter-string"
}
},
{
"if": {
"properties": {
"type": {
"const": "bool"
}
}
},
"then": {
"$ref": "#/$defs/parameter-bool"
}
},
{
"if": {
"properties": {
"type": {
"const": "int"
}
}
},
"then": {
"$ref": "#/$defs/parameter-int"
}
},
{
"if": {
"properties": {
"type": {
"const": "real"
}
}
},
"then": {
"$ref": "#/$defs/parameter-real"
}
}
]
},
"parameter-string": {
"properties": {
"default": {
"type": "string",
"default": ""
}
}
},
"parameter-bool": {
"properties": {
"default": {
"type": "boolean",
"default": false
}
}
},
"parameter-int": {
"properties": {
"default": {
"type": "integer",
"default": 0
}
}
},
"parameter-real": {
"properties": {
"default": {
"type": "number",
"default": 0
}
}
}
},
"properties": {
"test1": {"$ref": "#/$defs/parameter"}
}
}
]]
local json = [[
{
"test1" : {"type":"string", "default": false},
"test2" : {"type":"bool", "default": 43},
"test3" : {"type":"int", "default": 22.2},
"test4" : {"type":"real", "default": "foo"}
}
]]
local options = {match_pattern=function () return true end, name="test"}
local validator = JsonSchema.generate_validator(Json.decode(schema), options)
local isValid, errorStr = validator(Json.decode(json)) -- returns true, nil
Currently it always returns true.
I cannot help out with this right now, this issue is to document the shortcoming so other users are aware.
Hey there.
The Bug is quite easy:
jsonschema.lua line 261:
local loader, err = loadstring(tab_concat(self._code_table, ""), 'jsonschema:' .. (name or 'anonymous'))
On my system with Lua 5.4 that line gives this error:
attempt to call a nil value (upvalue 'loadstring')
it works when I call it like this:
local loader, err = load(tab_concat(self._code_table, ""), 'jsonschema:' .. (name or 'anonymous'))
some solution would probably be:
local loadfkt = loadstring or load
local loader, err = loadfkt(tab_concat(self._code_table, ""), 'jsonschema:' .. (name or 'anonymous'))
Example, worked in 0.8:
jsonschema.generate_validator({
type = "object",
properties = {
["$id"] = { type = "string" }
}
})
Now (0.9.5) fails with:
.../lib/openresty/luajit/share/lua/5.1/jsonschema/store.lua:199: attempt to concatenate field '$id' (a table value)
stack traceback:
.../lib/openresty/luajit/share/lua/5.1/jsonschema/store.lua:199: in function 'walk'
.../lib/openresty/luajit/share/lua/5.1/jsonschema/store.lua:235: in function 'walk'
.../lib/openresty/luajit/share/lua/5.1/jsonschema/store.lua:240: in function 'new'
/usr/lib/openresty/luajit/share/lua/5.1/jsonschema.lua:323: in function 'codectx'
/usr/lib/openresty/luajit/share/lua/5.1/jsonschema.lua:1139: in function 'generate_main_validator_ctx'
/usr/lib/openresty/luajit/share/lua/5.1/jsonschema.lua:1163: in function 'generate_validator'
I found this when checking validation against draft 7 metaschema from http://json-schema.org/draft-07/schema
https://rjsf-team.github.io/react-jsonschema-form/
Please take a look at the oneOf
example:
{
"type": "object",
"oneOf": [
{
"properties": {
"lorem": {
"type": "string"
}
},
"required": [
"lorem"
]
},
{
"properties": {
"ipsum": {
"type": "string"
}
},
"required": [
"ipsum"
]
}
]
}
Hello!
I hope you are doing well!
We are a security research team. Our tool automatically detected a vulnerability in this repository. We want to disclose it responsibly. GitHub has a feature called Private vulnerability reporting, which enables security research to privately disclose a vulnerability. Unfortunately, it is not enabled for this repository.
Can you enable it, so that we can report it?
Thanks in advance!
PS: you can read about how to enable private vulnerability reporting here: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.