wanglingsong / jsonsurfer Goto Github PK

View Code? Open in Web Editor NEW

288.0 17.0 55.0 1.08 MB

A streaming JsonPath processor in Java

License: MIT License

Java 99.53% ANTLR 0.47%

jsonpath java streaming non-blocking

jsonsurfer's Issues

Issue on jsonPath filter

Hello,

Given this json :

{
    "store": {
        "book": [
            {
                "category": "reference",
                "author": "Nigel Rees",
                "title": "Sayings of the Century",
                "price": 8.95,
                "description":{
                       "year":"2000"
                 }
            },
            {
                "category": "fiction",
                "author": "Evelyn Waugh",
                "title": "Sword of Honour",
                "price": 12.99,
                "description":{
                       "year":"2010"
                 }
            },
            {
                "category": "fiction",
                "author": "Herman Melville",
                "title": "Moby Dick",
                "isbn": "0-553-21311-3",
                "price": 8.99,
                "description":{
                       "year":"2000"
                 }
            },
            {
                "category": "fiction",
                "author": "J. R. R. Tolkien",
                "title": "The Lord of the Rings",
                "isbn": "0-395-19395-8",
                "price": 22.99
            }
        ],
        "bicycle": {
            "color": "red",
            "price": 19.95
        }
    },
    "expensive": 10
}

The path $.store.book[?(@.description.year=='2010')]

should return :

{
    "category": "fiction",
    "author": "Evelyn Waugh",
    "title": "Sword of Honour",
    "price": 12.99,
    "description":{
            "year":"2010"
    }
}

But when I execute this code :

JsonSurfer surfer = JsonSurferJackson.INSTANCE;
Object actualValue = surfer.collectOne(json, "$.store.book[?(@.description.year=='2010')]");

I get this exception :

org.antlr.v4.runtime.misc.ParseCancellationException
	at org.antlr.v4.runtime.BailErrorStrategy.recoverInline(BailErrorStrategy.java:66)
	at org.antlr.v4.runtime.Parser.match(Parser.java:206)
	at org.jsfr.json.compiler.JsonPathParser.filter(JsonPathParser.java:726)
	at org.jsfr.json.compiler.JsonPathParser.path(JsonPathParser.java:260)
	at org.jsfr.json.compiler.JsonPathCompiler.compile(JsonPathCompiler.java:220)
	at org.jsfr.json.SurfingConfiguration$Builder.bind(SurfingConfiguration.java:131)
	at org.jsfr.json.JsonSurferTest.testJsonPathFilterDeep(JsonSurferTest.java:117)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: org.antlr.v4.runtime.InputMismatchException
	at org.antlr.v4.runtime.BailErrorStrategy.recoverInline(BailErrorStrategy.java:61)
	... 29 more

SurfingContext class should be public

The SurfingContext class should be made public, otherwise its not possible to implement a custom JsonParserAdapter in any package other than org.jsfr.json

Gson parser should support lenient parsing

By default Gson's JsonReader class uses strict parsing which seems logical, however once you start trying parse real world json data you frequently run into the following:

MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line ...

It would be nice if the JsonReader used by GsonParser was lenient by default, or at least had a way to configure it for lenient parsing.

JsonSurfer cannot parse boolean attributes

Given the following JSON

[{"a": true, "b": 1}, {"a": false, "b": 2}, {"a": false, "b": 3}]

and code

JsonSurferJackson.INSTANCE.collectAll(inputStream, "$[?(@.a==false)]")

JsonSurfer fails with

line 1:9 no viable alternative at input '@.a==false'

org.antlr.v4.runtime.misc.ParseCancellationException
    at org.antlr.v4.runtime.BailErrorStrategy.recover(BailErrorStrategy.java:51)
    ...
Caused by: org.antlr.v4.runtime.NoViableAltException

java annotation binding

For example:

@JsonPath("$.foo.bar")
private String bar;

Negation in JSON filter

Following JSON path fails- $..book[?(!(@.price<10 && @.category=='fiction'))] while it works fine on http://jsonpath.com/.

Looking at the code seems like the JSON Path grammar (JsonPath.g4) doesn't have negation support. Are there plans for supporting it?

Integrate with Java native Json API - JSR 353

Minimum required Java version will be 7

Not working if node key is a number

It looks like the path could not be parsed if the node key is a number, such as "$.tree.taxonomy.100177". Is there a dedicated expression that used for this case to work?

Add Support For Knowing Parsing Completed

Right now on trying to convert this Api to Reactive Streams we don't have any way to
know that parsing completed, hence can con signal onComplete event in Reactive Streams. So please add support for same.
If I am missing something then correct me.

Transform POJO by JsonPath

Allow regular expression matching in filter

Some other JsonPath libraries allow this, like Jayway JsonPath using the [?(@.name =~ /foo.*?/i)] syntax.

GsonParser converting longs to doubles

The GsonParser is converting longs to doubles within the numberHolder implementation. Instead of calling return jsonProvider.primitive(jsonReader.nextString()); perhaps something like this would work better:

final String value = jsonReader.nextString();
try {
    return jsonProvider.primitive(Long.parseLong(value));
}
catch (final NumberFormatException e) {
    return jsonProvider.primitive(Double.parseDouble(value));
}

Reusable JsonConfiguration

It is reported by user that the surf(reader, surfingContext) API is not efficient enough because the user have to initiate SurfingContext once and once again for every json processing. Current implementation is too heavy for building SurfingContext from scratch. A reusable configuration object should be introduced to eliminate such overhead.

Support binary format

Streaming support

JsonSurf is currently event-driven, let's consider use case of processing big json with JsonPath expression:

it will not load entire json in memory
it will load all results in memory (in a collection) in event listener, in order to return it to caller.

The proposal is to provide an iterator, so that client code can interact with parser in pull manner, getting events one by one.

Values nested in array not supported

My use case is rather simple as I do not do any filtering but flattening of remote service response. My use case is supported by jq, but fails to get evaluated with JsonSurfer (and json path too).

My jq expression is
.entry[] | {url: .fullUrl, values: [ .resource.component[]? | .valueQuantity.value ] }

Which could be translated to json surfer:
$.entry['fullUrl', resource.component[*].valueQuantity.value]

However there are two major issues. From my evaluation it seems that ['property1', 'property2'] syntax always returns first property value and ommits second. Second issue is array wildcard path operator after which no more additional emenets may occur (althrought it passes compilation). Depending on expression variant it will cause class cast exception (usage of ['property'] syntax in context of array) or operation not supported exceptions ([*].property).

Are there any chances for support of such scenarios?

For reference I attach simplified resource model I have.

{
  "entry": [{
    "fullUrl": "http://localhost:8080/test/314",
    "resource": {
      "id": "56",
      "meta": {
        "lastUpdated": "2017-12-07T22:44:51.744+01:00"
      },
      "effectiveDateTime": "2009-09-09T00:00:00",
      "component": [{
        "valueQuantity": {
          "value": 109
        }
      },{
        "valueQuantity": {
          "value": 119
        }
      }]
    }
  },{
    "fullUrl": "http://localhost:8080/test/318",
    "resource": {
      "id": "57",
      "valueQuantity": {
        "value": 100
      }
    }
  }]
}

Output given by jq:

{
  "url": "http://localhost:8080/test/314",
  "values": [
    109,
    119
  ]
}
{
  "url": "http://localhost:8080/test/318",
  "values": []
}

(second resource does not have "component" embedded but valueQuantity directly at resource level)

Get attribute value with filtering not working

Hello

I have a json structure like this
{ "policy" : { "policy-details" : { "parties" : { "party" : [{"role" : "role1", "name" : "name1"}, {"role" : "role2", "name" : "name2"}]}} } }
With this jsonPath $.policy.policy-details.parties.party[?(@.role == 'role1')].name I got
org.antlr.v4.runtime.misc.ParseCancellationException
at org.antlr.v4.runtime.BailErrorStrategy.recoverInline(BailErrorStrategy.java:66)
at org.antlr.v4.runtime.Parser.match(Parser.java:206)
at org.jsfr.json.compiler.JsonPathParser.path(JsonPathParser.java:154)
at org.jsfr.json.compiler.JsonPathCompiler.compile(JsonPathCompiler.java:238)
at org.jsfr.json.SurfingConfiguration$Builder.bind(SurfingConfiguration.java:148)

But the path is working while testing on http://jsonpath.com/.

Is it a bug or not supported ?

Thanks

Regression in 1.5.0

Hello,

In 1.4.3, I have a code working well like that :
surfer.configBuilder()
.bind("$.header", (JsonPathListener) (final Object value, final ParsingContext context) -> {
}).bind("$.data[?(@.device-type=='vmanage')]",
(JsonPathListener) (final Object value, final ParsingContext context) -> {
})
.bind("$.data[?(@.device-type=='vsmart')]",
(JsonPathListener) (final Object value, final ParsingContext context) -> {
})
.buildAndSurf(entity.getContent());
Now, in 1.5.0, only the first bind found data. No data for the second and third bind (we have data with version 1.4.3)
If you removed the third bind, the second bind has data.
From my point of view, you have an issue when you have several binding with a filter on same attribute.

Wildward does not work on Pojo

When using jsonpath on objects having arrays or list, wildcard exception does not work. The following exception is thrown :
java.lang.UnsupportedOperationException: Not supported
at org.jsfr.json.path.PathOperator.resolve(PathOperator.java:48)
at org.jsfr.json.path.JsonPath.resolve(JsonPath.java:177)

package test;

import java.util.Arrays;
import java.util.List;

import org.jsfr.json.compiler.JsonPathCompiler;
import org.jsfr.json.path.JsonPath;
import org.jsfr.json.resolver.PoJoResolver;
import org.junit.jupiter.api.Test;

public class JsonSurferTest {

    class B {
        private boolean c = true;

    }

    class A {
        private List<B> b = Arrays.asList(new B[] { new B(), new B()});
        private B[] b2 = new B[] { new B(), new B()};

    }
    
    @Test
    public void testList() {
        A a = new A();
        JsonPath compiledPath = JsonPathCompiler.compile("$.b[*].c");
        compiledPath.resolve(a, new PoJoResolver());
        
    }
    
    @Test
    public void testArray() {
        A a = new A();
        JsonPath compiledPath = JsonPathCompiler.compile("$.b2[*].c");
        compiledPath.resolve(a, new PoJoResolver());
        
    }
}

Add support for Java 8 Stream

Thanks for your great work. Is it possible to add in support for Java 8 Stream? e.g. return a stream of objects so that the stream can emit new elements as soon as they arrive from the Json input stream.

Expose location from parsing context

I have a use case, where I need to capture the current offset (i.e., line number, column number) in the stream from the ParsingContext. In jackson speak, this is the com.fasterxml.jackson.core.JsonLocation. I could probably capture this manually, by providing a custom JsonFactory that captures the returned parser from the createParser method (I haven't tried) so that I can query the native method.

It would be nice if this is natively supported in the ParsingContext.

Support Non-blocking parsing for Jackson parser

Jackson provides non-blocking parsing since 2.9. FasterXML/jackson-core#57
With that, JsonSurfer may become a 100% non-blocking json processing solution.

streaming json writer

JacksonSurfer

relative path after filter

I am trying JSON path $.store.book[?(@.price < 10 || @.category =='fiction')].author but getting an error Caused by: org.antlr.v4.runtime.InputMismatchException

is this due to the assumption that after a filter, there may not be a relative path? the antlr grammar file JsonPath.g4 currently has- path: '$' relativePath* filter? EOF;

Supply an OutputStream to a SurfingConfiguration

First: Good job with JsonSurfer! It's a helpful tool, for sure.

I am looking for a way to read from a large JSON file in a streaming fashion and write an excerpt from that file to a new file in a streaming fashion. The onValue method of the JsonPathListener receives an Object which was obtained in a streaming manner. This is great, but at that point the full Object is loaded in memory. Would it be possible to specify an OutputStream to a SurfingConfiguration, such that the matching portions of the incoming file could be written to the stream instead of memory?

Actually, I really need a different OutputStream for every excerpt that is matched by the specified JSON Path. Perhaps a new SurfingConfiguration.setStreamHandler method would receive a new OutputStreamHandler interface that two methods: OutputStream onNewOutput(ParsingContext context) and void end(OutputStream out)... or something. There are a few ways to do it, I'm sure.

Maybe there is already support for something like this. I looked but couldn't find any.

License

Looks cool. Can you add a license like MIT or whatever you want?

Property binding not working when it's coupled with a filtered binding

Hi,

I found a strange behaviour, not sure if it's a bug or it's just me using the API in the wrong manner.
(At least) With JsonSurfer and FastJsonParser, if I try this code

public void testStrangeBehavihour() throws Exception {
        surfer.configBuilder()
                .bind("$.store.book[?(@.category == 'reference')]", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .bind("$.store.bicycle.color", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value); //This never happen
                    }
                })
                .bind("$.store.bicycle", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                }).buildAndSurf(read("sample.json"));
    }

the second listener will never be called.

If I remove the first binding everything works well:

    public void testStrangBehavihour() throws Exception {
        surfer.configBuilder()
                .bind("$.store.bicycle.color", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                })
                .bind("$.store.bicycle", new JsonPathListener() {
                    @Override
                    public void onValue(Object value, ParsingContext context) {
                        System.out.println(value);
                    }
                }).buildAndSurf(read("sample.json"));
    }

I tried to find an explanation in the code but found nothing.

Empty results for root array

JsonSurfer returns empty results for the following test, whereas Jayway implementation returns 2 maps correctly:

JsonSurferJackson.INSTANCE.collectAll(
	"[\n" +
	"    {\n" +
	"      \"type\"  : \"iPhone\",\n" +
	"      \"number\": \"0123-4567-8888\"\n" +
	"    },\n" +
	"    {\n" +
	"      \"type\"  : \"home\",\n" +
	"      \"number\": \"0123-4567-8910\"\n" +
	"    }\n" +
	"  ]", JsonPathCompiler.compile("$.*"))

JsonPath Filter

Android compatibility issue

JsonSurfer works in my Note 4 running Android 6 but in my Nexus 4 device which is
running on Android 4.2 it crashes on the second line:

JsonSurfer surfer = JsonSurferGson.INSTANCE;

Object statementIDObj = surfer.collectOne(myResponse, "$.results[0].statement_id");

producing the error: java.lang.NoClassDefFoundError: java.nio.charset.StandardCharsets .

Now I know that Standard Charsets were introduced in KitKat but I wanted to ask if there
is a workaround for this. Thank you

Results do not match other implementations

The following queries provide results that do not match those of other implementations of JSONPath
(compare https://cburgmer.github.io/json-path-comparison/):

$.2
Input:

{"a": "first", "2": "second", "b": "third"}

Expected output:

["second"]

Error:

Exception in thread "main" org.antlr.v4.runtime.misc.ParseCancellationException
	at org.antlr.v4.runtime.BailErrorStrategy.recoverInline(BailErrorStrategy.java:66)
	at org.antlr.v4.runtime.Parser.match(Parser.java:206)
	at org.jsfr.json.compiler.JsonPathParser.childNode(JsonPathParser.java:672)
	at org.jsfr.json.compiler.JsonPathParser.relativePath(JsonPathParser.java:272)
	at org.jsfr.json.compiler.JsonPathParser.path(JsonPathParser.java:159)
	at org.jsfr.json.compiler.JsonPathCompiler.compile(JsonPathCompiler.java:283)
	at org.jsfr.json.compiler.JsonPathCompiler.compile(JsonPathCompiler.java:273)
	at org.jsfr.json.JsonSurfer.collectAll(JsonSurfer.java:262)
	at query.App.main(App.java:27)
Caused by: org.antlr.v4.runtime.InputMismatchException
	at org.antlr.v4.runtime.BailErrorStrategy.recoverInline(BailErrorStrategy.java:61)
	... 8 more

$[-1]
Input:

["first", "second", "third"]

Expected output:

["third"]

Actual output:

[]

$[-1:]
Input:

["first", "second", "third"]

Expected output:

["third"]

Actual output:

["first", "second", "third"]

$[?(@.key=="some.value")]
Input:

[{"key": "some"}, {"key": "value"}, {"key": "some.value"}]

Expected output:

[{"key": "some.value"}]

Error:

line 1:11 token recognition error at: '"'
line 1:22 token recognition error at: '"'
line 1:12 no viable alternative at input '@.key==some'
Exception in thread "main" org.antlr.v4.runtime.misc.ParseCancellationException
	at org.antlr.v4.runtime.BailErrorStrategy.recover(BailErrorStrategy.java:51)
	at org.jsfr.json.compiler.JsonPathParser.filterExpr(JsonPathParser.java:947)
	at org.jsfr.json.compiler.JsonPathParser.filter(JsonPathParser.java:768)
	at org.jsfr.json.compiler.JsonPathParser.relativePath(JsonPathParser.java:307)
	at org.jsfr.json.compiler.JsonPathParser.path(JsonPathParser.java:159)
	at org.jsfr.json.compiler.JsonPathCompiler.compile(JsonPathCompiler.java:283)
	at org.jsfr.json.compiler.JsonPathCompiler.compile(JsonPathCompiler.java:273)
	at org.jsfr.json.JsonSurfer.collectAll(JsonSurfer.java:262)
	at query.App.main(App.java:27)
Caused by: org.antlr.v4.runtime.NoViableAltException
	at org.antlr.v4.runtime.atn.ParserATNSimulator.noViableAlt(ParserATNSimulator.java:2026)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.execATN(ParserATNSimulator.java:467)
	at org.antlr.v4.runtime.atn.ParserATNSimulator.adaptivePredict(ParserATNSimulator.java:393)
	at org.jsfr.json.compiler.JsonPathParser.filterExpr(JsonPathParser.java:843)
	... 7 more

For reference, the output was generated by the program in https://github.com/cburgmer/json-path-comparison/tree/master/implementations/Java_com.github.jsurfer.

Documentation

1.API:
collectOne
collectAll
2.ErrorHandlingStrategy
3. performance tuning tips

definite path vs indefinite
skip overlapped path

Parsing maps/jackson

In this document, metricValues is a Map<string,double>.

"metricValues": {
"metric1": 10.977,
"metric2": 0.062111801242236024
}

Variations of this expression:

Collection<Object> metricValues = surfer.collectAll(json, "$.body.metricValues.*");

Are pulling up the numbers (10.977) but not the keys ("metric1").

I've tried to cast to maps, classes with name-value, etc.

Suggestions?

Support script expressions

The current JsonPath spec allows script expressions aka functions, are there any plans to support it?

$.books.size()

$.somestring.split(".")[0]

Collector interface enhancement

Currently, the "collect" interfaces are not handy, because It surfs the whole json and apply only one JsonPath for each collection which is obviously not efficient at all. So I plan to add following interface:

    JsonSurfer surfer = JsonSurferJackson.INSTANCE;
    Builder builder = surfer.configBuilder();
    Holder holder1 = builder.collectOne("$.store");
    Holder holder2 = builder.collectAll("$.store.book[*]");
    builder.buildAndSurfer(json);
    holder1.get();
    holder2.get();

The enhanced interface will allow user to add any desired JsonPath during config building phase and collect everything matched into holders within a single surfing.

Error handling strategy

Can JsonSurfer be used over multiple input files with same structure ?

I wanted explorer if JsonSurfer can work on scanning over similar json structure distributed over multiple files ?

JsonPath Resolver for POJO

Support definite JsonPath only.
e.g. $.foo.bar
No for $..foo..bar

Redesign SAX like interface

Keys containin a space cannot be parsed

I have a json with some keys that contain spaces ("Image Player". "Phone Number"). There's a ParseCancellationException if I try to parse it:

MyApp.getJsonSurfer().collectOne(jsonAsString, "$.fields.Phone Number")

How to handle when keys have spaces?

Here's the sample json:

{ "id": "some_id", "fields": { "Email": "[email protected]", "Status": "Amateur", "Image player": [ { "id": "some_other_id", "url": "https://www.google.es", "filename": "image.png", "size": 413642, "type": "image/png", "thumbnails": { "small": { "width": 36, "height": 36 }, "large": { "width": 500, "height": 500 }, "full": { "width": 500, "height": 500 } } } ], "Phone Number": "666666666", "Name": "Player one", "Wins": [ 0 ], "Standing": [ "Eliminated" ] }, "createdTime": "2018-06-28T22:32:47.000Z" }

wanglingsong / jsonsurfer Goto Github PK

jsonsurfer's Issues

Recommend Projects

Recommend Topics

Recommend Org