Hi, I've got a csv file that I wanted to convert to json using <code class="notran

Please read this <a href="https://miller.readthedocs.io/en/6.11.0/reference-main-flag-

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

`mlr --icsv --ojson cat < mlr.bug.csv` drops some columns about miller HOT 5 CLOSED

SamuelLarkin commented on June 8, 2024

`mlr --icsv --ojson cat < mlr.bug.csv` drops some columns

from miller.

Comments (5)

SamuelLarkin commented on June 8, 2024 2

🤦🏼‍♂️
I was going to write that I found a solution but it looks like I'm slightly too late.

Thanks for your answer.

mlr --icsv --ojson --flatsep : cat < mlr.bug.2.csv

[
{
  "expt_name": "lr=1.0e-5_e=16/000",
  "A": 40.93,
  "B.last_checkpoint": 39.75,
  "C.001": 44.05,
  "C.001.last_checkpoint": 43.13
},
{
  "expt_name": "lr=1.0e-5_e=16/001",
  "A": 41.09,
  "B.last_checkpoint": 41.11,
  "C.001": 43.81,
  "C.001.last_checkpoint": 43.82
},
{
  "expt_name": "lr=1.0e-5_e=16/002",
  "A": 40.84,
  "B.last_checkpoint": 40.81,
  "C.001": 43.87,
  "C.001.last_checkpoint": 43.72
}
]

from miller.

SamuelLarkin commented on June 8, 2024

Here's another example where we see the . hierarchy shadowing substrings.

expt_name,A,B.last_checkpoint,C.001,C.001.last_checkpoint
lr=1.0e-5_e=16/000,40.93,39.75,44.05,43.13
lr=1.0e-5_e=16/001,41.09,41.11,43.81,43.82
lr=1.0e-5_e=16/002,40.84,40.81,43.87,43.72

mlr --icsv --ojson cat < mlr.bug.2.csv

[
{
  "expt_name": "lr=1.0e-5_e=16/000",
  "A": 40.93,
  "B": {
    "last_checkpoint": 39.75
  },
  "C": {
    "001": {
      "last_checkpoint": 43.13
    }
  }
},
{
  "expt_name": "lr=1.0e-5_e=16/001",
  "A": 41.09,
  "B": {
    "last_checkpoint": 41.11
  },
  "C": {
    "001": {
      "last_checkpoint": 43.82
    }
  }
},
{
  "expt_name": "lr=1.0e-5_e=16/002",
  "A": 40.84,
  "B": {
    "last_checkpoint": 40.81
  },
  "C": {
    "001": {
      "last_checkpoint": 43.72
    }
  }
}
]

from miller.

aborruso commented on June 8, 2024

Please read this https://miller.readthedocs.io/en/6.11.0/reference-main-flag-list/index.html#flatten-unflatten-flags

The way to do it, is to use --no-auto-unflatten flag. Running

mlr --c2j --no-auto-unflatten cat input.csv

you get

[
  {
    "expt_name": "lr=1.0e-5_e=16/000",
    "baseline-final-test": 40.93,
    "baseline-final-test.last_checkpoint": 39.75,
    "stage-all.001": 44.05,
    "stage-all.001.last_checkpoint": 43.13
  },
  {
    "expt_name": "lr=1.0e-5_e=16/001",
    "baseline-final-test": 41.09,
    "baseline-final-test.last_checkpoint": 41.11,
    "stage-all.001": 43.81,
    "stage-all.001.last_checkpoint": 43.82
  },
  {
    "expt_name": "lr=1.0e-5_e=16/002",
    "baseline-final-test": 40.84,
    "baseline-final-test.last_checkpoint": 40.81,
    "stage-all.001": 43.87,
    "stage-all.001.last_checkpoint": 43.72
  }
]