bson-json-to-csv's Introduction

bson-json-to-csv

Creates a flat csv files out of a nested bson/json file

Click here to download

Usage

Works with stream of json/bson objects. For example,

{
  "name": "john",
  "location": {
    "city": "mumbai",
    "country": "india"
  }
}
{
  "name": "jane",
  "location": "delhi",
  "tags": ["scala", "java", "big data"]
}

Or with a comma

{
  "name": "john",
  "location": {
    "city": "mumbai",
    "country": "india"
  }
},
{
  "name": "jane",
  "location": "delhi",
  "tags": ["scala", "java", "big data"]
}

Version 1.4.0 also adds supports for top level single large array. All first level items in the array result into an individual csv row.

Example:

[
  {
    "name": "john",
    "location": {
      "city": "mumbai",
      "country": "india"
    }
  },
  {
    "name": "jane",
    "location": "delhi",
    "tags": ["scala", "java", "big data"]
  }
]

Each top level json object results into an individual csv row.

The above json will generate a csv with a header and two csv records

.name,.location.city,.location.country,.tags[1],.location,.tags[0],.tags[2]
john,mumbai,india,,,,
jane,,,java,delhi,scala,big data

The headers are json-paths of corresponding json fields

.name	.location.city	.location.country	.tags[1]	.location	.tags[0]	.tags[2]
john	mumbai	india
jane			java	delhi	scala	big data

Notice that as shown in above example, the order of json fields (columns) may not reflect exactly same in the csv. However the indexes of arrays will be maintained correctly.

Order of rows will be same as json records.

bson-json-to-csv's People

Contributors

Stargazers

Watchers

bson-json-to-csv's Issues

Hardcoded timeout of 5 seconds

    val params = schema
      .generate(source)
      .via(viaIndex(x => print(s"\rfound ${x._2 + 1} columns...")))
      .runWith(Sink.collection)
      .recover {
        case NonFatal(_: FramingException) =>
          println("Invalid JSON encountered")
          system.terminate().block()
          sys.exit(1)
        case NonFatal(err) =>
          err.printStackTrace()
          system.terminate().block()
          sys.exit(1)
      }
      .block() // todo: this is an issue!!!
      .toList

  extension [T](f: Future[T]) {
    def block(duration: FiniteDuration = 5.seconds): T =
      Await.result(f, duration)
  }