Giter Club home page Giter Club logo

embulk-parser-json's People

Contributors

bizenn avatar myoan avatar takumakanari avatar

Stargazers

avatar avatar avatar avatar avatar avatar

Watchers

avatar

Forkers

mitomemel miseyu bizenn ykst myoan

embulk-parser-json's Issues

"jsonl" parser plugin has bundled in official package as "json"

https://github.com/embulk/embulk/blob/master/embulk-docs/src/release/release-0.8.5.rst

I can not use embulk-parser-json.

in:
  type: s3
  bucket: <bucket name>
  path_prefix: <path prefix>
  auth_method: profile
  profile_name: <profile>
  decoders:
  - {type: gzip}
  parser: 
    type: json
    root: $.Records
    schema:
      - {name: eventName, type: string}
      - {name: eventSource, type: string}
      - {name: awsRegion, type: string}
      - {name: sourceIPAddress, type: string}
      - {name: eventTime, type: string}
      - {name: requestID, type: string}
      - {name: eventID, type: string}
      - {name: userType, path: userIdentity.type, type: string}
      - {name: userArn, path: userIdentity.arn, type: string}
      - {name: userName, path: userIdentity.userName, type: string}
      - {name: accesskeyid, path: userIdentity.accessKeyId, type: string}
      - {name: userAgent, type: string}
      - {name: errorCode, type: string}
      - {name: errorMessage, type: string}
out: 
  type: file
  path_prefix: /tmp/cloudtrail/
  file_ext: json
$ embulk preview cloudtrail.yml
org.embulk.spi.DataException: org.embulk.spi.json.JsonParseException: Failed to parse JSON: in
        at org.embulk.standards.JsonParserPlugin.run(org/embulk/standards/JsonParserPlugin.java:89)
        at org.embulk.spi.FileInputRunner.run(org/embulk/spi/FileInputRunner.java:153)
        at org.embulk.exec.PreviewExecutor$2$1.run(org/embulk/exec/PreviewExecutor.java:122)
        at org.embulk.spi.util.Filters$RecursiveControl.transaction(org/embulk/spi/util/Filters.java:96)
        at org.embulk.spi.util.Filters.transaction(org/embulk/spi/util/Filters.java:49)
        at org.embulk.exec.PreviewExecutor$2.run(org/embulk/exec/PreviewExecutor.java:111)
        at org.embulk.spi.FileInputRunner$RunnerControl$1$1.run(org/embulk/spi/FileInputRunner.java:123)
        at org.embulk.standards.JsonParserPlugin.transaction(org/embulk/standards/JsonParserPlugin.java:48)
        at org.embulk.spi.FileInputRunner$RunnerControl$1.run(org/embulk/spi/FileInputRunner.java:117)
        at org.embulk.spi.util.Decoders$RecursiveControl.transaction(org/embulk/spi/util/Decoders.java:77)
        at org.embulk.spi.util.Decoders$RecursiveControl$1.run(org/embulk/spi/util/Decoders.java:73)
        at org.embulk.standards.GzipFileDecoderPlugin.transaction(org/embulk/standards/GzipFileDecoderPlugin.java:30)
        at org.embulk.spi.util.Decoders$RecursiveControl.transaction(org/embulk/spi/util/Decoders.java:68)
        at org.embulk.spi.util.Decoders.transaction(org/embulk/spi/util/Decoders.java:33)
        at org.embulk.spi.FileInputRunner$RunnerControl.run(org/embulk/spi/FileInputRunner.java:114)
        at org.embulk.exec.BufferFileInputPlugin.transaction(org/embulk/exec/BufferFileInputPlugin.java:26)
        at org.embulk.spi.FileInputRunner.transaction(org/embulk/spi/FileInputRunner.java:65)
        at org.embulk.exec.PreviewExecutor.doPreview(org/embulk/exec/PreviewExecutor.java:108)
        at org.embulk.exec.PreviewExecutor.doPreview(org/embulk/exec/PreviewExecutor.java:98)
        at org.embulk.exec.PreviewExecutor.access$000(org/embulk/exec/PreviewExecutor.java:30)
        at org.embulk.exec.PreviewExecutor$1.run(org/embulk/exec/PreviewExecutor.java:70)
        at org.embulk.exec.PreviewExecutor$1.run(org/embulk/exec/PreviewExecutor.java:66)
        at org.embulk.spi.Exec.doWith(org/embulk/spi/Exec.java:25)
        at org.embulk.exec.PreviewExecutor.preview(org/embulk/exec/PreviewExecutor.java:66)
        at org.embulk.EmbulkEmbed.preview(org/embulk/EmbulkEmbed.java:169)
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453)
        at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314)

Add feature of skipping invalid record

Hi
In the case of invalid record coming, it raise error with following stack trace.
Please support stop_on_invalid_record option to skip it or stop immediately.

org.embulk.exec.PartialExecutionException: org.jruby.exceptions.RaiseException: (ParserError) unexpected token at ''
	at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(org/embulk/exec/BulkLoader.java:373)
	at org.embulk.exec.BulkLoader.doRun(org/embulk/exec/BulkLoader.java:591)
	at org.embulk.exec.BulkLoader.access$000(org/embulk/exec/BulkLoader.java:33)
	at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:389)
	at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:385)
	at org.embulk.spi.Exec.doWith(org/embulk/spi/Exec.java:25)
	at org.embulk.exec.BulkLoader.run(org/embulk/exec/BulkLoader.java:385)
	at org.embulk.EmbulkEmbed.run(org/embulk/EmbulkEmbed.java:180)
	at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
	at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453)
	at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314)
	at RUBY.run(/usr/local/bin/embulk!/embulk/runner.rb:84)
	at RUBY.run(/usr/local/bin/embulk!/embulk/command/embulk_run.rb:307)
	at RUBY.<main>(/usr/local/bin/embulk!/embulk/command/embulk_main.rb:2)
	at org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:850)
	at org.jruby.Ruby.loadFile(org/jruby/Ruby.java:2976)
	at org.jruby.RubyKernel.requireCommon(org/jruby/RubyKernel.java:963)
	at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:956)
	at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(org/jruby/RubyKernel$INVOKER$s$1$0$require19.gen)
	at RUBY.(root)(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1)
	at usr.local.bin.embulk.embulk.command.embulk_bundle.invokeOther66:require(usr/local/bin/embulk/embulk/command/file:/usr/local/bin/embulk!/embulk/command/embulk_bundle.rb:51)
	at usr.local.bin.embulk.embulk.command.embulk_bundle.<main>(file:/usr/local/bin/embulk!/embulk/command/embulk_bundle.rb:51)
	at java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)
	at org.jruby.Ruby.runScript(org/jruby/Ruby.java:834)
	at org.jruby.Ruby.runNormally(org/jruby/Ruby.java:749)
	at org.jruby.Ruby.runNormally(org/jruby/Ruby.java:767)
	at org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:580)
	at org.jruby.Main.doRunFromMain(org/jruby/Main.java:425)
	at org.jruby.Main.internalRun(org/jruby/Main.java:313)
	at org.jruby.Main.run(org/jruby/Main.java:242)
	at org.jruby.Main.main(org/jruby/Main.java:204)
	at org.embulk.cli.Main.main(org/embulk/cli/Main.java:23)
Caused by: org.jruby.exceptions.RaiseException: (ParserError) unexpected token at ''
	at json.ext.Parser.parse(json/ext/Parser.java:251)
	at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.json.common.parse(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/json/common.rb:155)
	at root.$_dot_embulk.jruby.$2_dot_3_dot_0.gems.embulk_minus_parser_minus_json_minus_0_dot_0_dot_5.lib.embulk.parser.jsonpath.run(/root/.embulk/jruby/2.3.0/gems/embulk-parser-json-0.0.5/lib/embulk/parser/jsonpath.rb:25)
	at ocal.bin.embulk.embulk.parser_plugin.run(/usr/local/bin/embulk!/embulk/parser_plugin.rb:56)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.