Giter Club home page Giter Club logo

ex_ndjson's Introduction

ExNdjson

Build Status Hex.pm Version Hex.pm Download Total

ExNdjson is a Newline-delimited JSON library for Elixir that implements encoding and decoding to/from NDJSON as described in the NDJSON Spec.

Requirements

  • Elixir 1.6 or later

Installation

First, Add ExNdjson to you mix.exs dependencies:

def deps do
  [
    {:ex_ndjson, "~> 0.3.2"}
  ]
end

Then, update your dependencies:

mix deps.get

Usage

ExNdjson.marshal!([%{"some" => "thing"}, %{"bar" => false, "foo" => 17, "quux" => true}])
#=> "{\"some\":\"thing\"}\n{\"quux\":true,\"foo\":17,\"bar\":false}\n"

ExNdjson.marshal_into_file!([%{id: 1}, [1, 2, 3]], "/path/to/dump.ndjson")
#=> :ok

ExNdjson.unmarshal('{"some": "thing"}\n{"quux":true, "foo":17, "bar": false}\r\n')
#=> [%{"some" => "thing"}, %{"bar" => false, "foo" => 17, "quux" => true}]

ExNdjson.unmarshal(<<123, 125, 10>>)
#=> [%{}]

ExNdjson.unmarshal_from_file!("/path/to/ndjson/file")
#=> [%{"id" => "1"}, [1, 2, 3]]

Configuration

You can customize the library used for JSON encoding/decoding:

config :ex_ndjson, json_library: Poison # Defaults to Jason

Documentation

Full documentation can be found at https://hexdocs.pm/ex_ndjson.

License

The library is available as open source under the terms of the MIT License.

ex_ndjson's People

Contributors

amrfaissal avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

marceldegraaf

ex_ndjson's Issues

NdJson not being parsed

First of all thanks for this library. I'm trying to parse an NdJson from stdout (output from gherkin c implementation). I've tried writing it to a file as well, but this didn't help.

Here is the ndjson:

{"source":{"media":{"encoding":"UTF8","contentType":"text/x.cucumber.gherkin+plain"},"uri":"/home/wannes/Documents/temp/gherkin_parser/gherkin_c/testdata/good/minimal-example.feature","data":"Feature: Minimal\n\n  Example: minimalistic\n    Given the minimalism\n"}}
{"gherkinDocument":{"uri":"/home/wannes/Documents/temp/gherkin_parser/gherkin_c/testdata/good/minimal-example.feature","feature":{"language":"en","keyword":"Feature","name":"Minimal","location":{"line":1,"column":1},"children":[{"scenario":{"keyword":"Example","name":"minimalistic","location":{"line":3,"column":3},"steps":[{"keyword":"Given ","text":"the minimalism","location":{"line":4,"column":5}}]}}]}}}
{"pickle":{"uri":"/home/wannes/Documents/temp/gherkin_parser/gherkin_c/testdata/good/minimal-example.feature","language":"en","id":"93ce054b56a540becc47afcbe729ea7f87c74443","name":"minimalistic","locations":[{"line":3,"column":3}],"steps":[{"locations":[{"line":4,"column":11}],"text":"the minimalism"}]}}

If relevant, here is the output in string format (iex shell):

"{\"source\":{\"media\":{\"encoding\":\"UTF8\",\"contentType\":\"text/x.cucumber.gherkin+plain\"},\"uri\":\"/home/wannes/Documents/temp/gherkin_parser/gherkin_c/testdata/good/minimal-example.feature\",\"data\":\"Feature: Minimal\\n\\n  Example: minimalistic\\n    Given the minimalism\\n\"}}\n{\"gherkinDocument\":{\"uri\":\"/home/wannes/Documents/temp/gherkin_parser/gherkin_c/testdata/good/minimal-example.feature\",\"feature\":{\"language\":\"en\",\"keyword\":\"Feature\",\"name\":\"Minimal\",\"location\":{\"line\":1,\"column\":1},\"children\":[{\"scenario\":{\"keyword\":\"Example\",\"name\":\"minimalistic\",\"location\":{\"line\":3,\"column\":3},\"steps\":[{\"keyword\":\"Given \",\"text\":\"the minimalism\",\"location\":{\"line\":4,\"column\":5}}]}}]}}}\n{\"pickle\":{\"uri\":\"/home/wannes/Documents/temp/gherkin_parser/gherkin_c/testdata/good/minimal-example.feature\",\"language\":\"en\",\"id\":\"93ce054b56a540becc47afcbe729ea7f87c74443\",\"name\":\"minimalistic\",\"locations\":[{\"line\":3,\"column\":3}],\"steps\":[{\"locations\":[{\"line\":4,\"column\":11}],\"text\":\"the minimalism\"}]}}\n"

This output is able to be formatted by the "jq" linux program.
Elixir version: 1.10
OS: 18.04.1-Ubuntu

I am able to parse it manually though with the following code (though i am aware that this is definitely not best practices.):

output_string
    |> String.replace("}\n{", "}SPLIT HERE{")
    |> String.split("SPLIT HERE")
    |> Enum.map(&Jason.decode!/1)

Stacktrace (snipping string contents):

[error] GenServer ExNdjson.Worker terminating
** (UndefinedFunctionError) function nil.decode/1 is undefined
    nil.decode("{\"source\":{\"med [CONTENT SNIPPED] lism\\n\"}}")
    (elixir 1.10.3) lib/enum.ex:1396: Enum."-map/2-lists^map/1-0-"/2
    (ex_ndjson 0.3.2) lib/ex_ndjson/parser.ex:39: ExNdjson.NdJSONParser.parse/1
    (ex_ndjson 0.3.2) lib/ex_ndjson/worker.ex:37: ExNdjson.Worker.handle_call/3
    (stdlib 3.12.1) gen_server.erl:661: :gen_server.try_handle_call/4
    (stdlib 3.12.1) gen_server.erl:690: :gen_server.handle_msg/6
    (stdlib 3.12.1) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.411.0>): {:unmarshal, "{\"sourc [CONTENT SNIPPED] m\"}]}}\n"}
State: []
Client #PID<0.411.0> is alive

    (stdlib 3.12.1) gen.erl:167: :gen.do_call/4
    (elixir 1.10.3) lib/gen_server.ex:1020: GenServer.call/3
    (stdlib 3.12.1) erl_eval.erl:680: :erl_eval.do_apply/6
    (elixir 1.10.3) src/elixir.erl:278: :elixir.recur_eval/3
    (elixir 1.10.3) src/elixir.erl:263: :elixir.eval_forms/3
    (iex 1.10.3) lib/iex/evaluator.ex:258: IEx.Evaluator.handle_eval/5
    (iex 1.10.3) lib/iex/evaluator.ex:239: IEx.Evaluator.do_eval/3
    (iex 1.10.3) lib/iex/evaluator.ex:217: IEx.Evaluator.eval/3
** (exit) exited in: GenServer.call(ExNdjson.Worker, {:unmarshal, "{\"source\":{\"media\" [CONTENT SNIPPED]  }}\n"}, 5000)
    ** (EXIT) an exception was raised:
        ** (UndefinedFunctionError) function nil.decode/1 is undefined
            nil.decode("{\"source\":{\"media\":{\"encoding\":\"UTF8\",\"conten [CONTENT SNIPPED] m\\n\"}}")
            (elixir 1.10.3) lib/enum.ex:1396: Enum."-map/2-lists^map/1-0-"/2
            (ex_ndjson 0.3.2) lib/ex_ndjson/parser.ex:39: ExNdjson.NdJSONParser.parse/1
            (ex_ndjson 0.3.2) lib/ex_ndjson/worker.ex:37: ExNdjson.Worker.handle_call/3
            (stdlib 3.12.1) gen_server.erl:661: :gen_server.try_handle_call/4
            (stdlib 3.12.1) gen_server.erl:690: :gen_server.handle_msg/6
            (stdlib 3.12.1) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
    (elixir 1.10.3) lib/gen_server.ex:1023: GenServer.call/3

Thank you in advance for your support.

Parsing Issue

Certain characters break the string in the marshal method.

Here's an example:

Actual result:

─ iex -S mix                                                                                                                                                                                                              
Erlang/OTP 22 [erts-10.6.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Compiling 3 files (.ex)
Interactive Elixir (1.9.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> covid_19 = %{pandemic: "{sucks}"}
%{pandemic: "{sucks}"}
iex(2)> ExNdjson.marshal!([covid_19])
"{\"pandemic\":{sucks}}\n"

Expected result:

─ iex -S mix                                                                                                                                                                                                               
Erlang/OTP 22 [erts-10.6.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Interactive Elixir (1.9.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> covid_19 = %{pandemic: "{sucks}"}
%{pandemic: "{sucks}"}
iex(2)> ExNdjson.marshal!([covid_19])
"{\"pandemic\":\"{sucks}\"}\n"

By simply deleting the cleanup_string function, everything seems to work. What is the objective of this method?

Proof:

image

image (1)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.