Problem with boolean and decimal values about nodejs-bigtable HOT 7 CLOSED

I was able to reproduce.

cc @kolodny

from nodejs-bigtable.

I was able to repro too. This bug doesn't seem new though since I was able to repro as far back as v0.10.2. It appears the only difference is that having bools in the data used to throw TypeError: Illegal buffer and after #35 it just puts it bools in as empty strings.

What is the current state with booleans in Bigtable? Does it get encoded in some special way that we can encode/decode when writing/reading that value?

cc @sduskis

from nodejs-bigtable.

Here's the gapic ReadRowsResponse for inserting a decimal, 1.23:

{
  "chunks": [
    {
      "labels": [],
      "rowKey": {
        "type": "Buffer",
        "data": [
          103,
          119,
          97,
          115,
          104,
          105,
          110,
          103,
          116,
          111,
          110
        ]
      },
      "familyName": {
        "value": "follows"
      },
      "qualifier": {
        "value": {
          "type": "Buffer",
          "data": [
            106,
            97,
            100,
            97,
            109,
            115
          ]
        }
      },
      "timestampMicros": "1524072042604000",
      "value": {
        "type": "Buffer",
        "data": [
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          1
        ]
      },
      "valueSize": 0,
      "commitRow": true,
      "rowStatus": "commitRow"
    }
  ],
  "lastScannedRowKey": []
}

And for true:

{
  "chunks": [
    {
      "labels": [],
      "rowKey": {
        "type": "Buffer",
        "data": [
          103,
          119,
          97,
          115,
          104,
          105,
          110,
          103,
          116,
          111,
          110
        ]
      },
      "familyName": {
        "value": "follows"
      },
      "qualifier": {
        "value": {
          "type": "Buffer",
          "data": [
            106,
            97,
            100,
            97,
            109,
            115
          ]
        }
      },
      "timestampMicros": "1524071811968000",
      "value": [],
      "valueSize": 0,
      "commitRow": true,
      "rowStatus": "commitRow"
    }
  ],
  "lastScannedRowKey": []
}

At this point, it seems impossible to derive 1.23 from value: [0, 0, 0, 0, 0, 0, 0, 1] and true from value: [].

from nodejs-bigtable.

Bigtable stores all it's fields in binary with no encoding information. The typical way to put data inside bigtable or extract it out is by manually doing the encoding/decoding. The library currently helps out by converting numbers to Buffers otherwise it just passes whatever the value is to Buffer.from and hopes for the best. Ideally every field should be encoded before being passed to table.mutate.

In the future we may consider allowing passing some Mutation like API to simplify the encoding/decoding actions to and from the table

from nodejs-bigtable.

One thing to note based on the above, any data previously entered in that fashion should be considered bad and should be deleted

I'll close this issue for now, feel free to reopen if you have any followup questions or issues

from nodejs-bigtable.

Thanks. I would consider throwing on everything except string buffer and null/undefined instead of potentially storing bad values. After testing to see that string/int works you may move on thinking that also bool and decimal works (like I almost did).

from nodejs-bigtable.

Thanks @moander, I opened #102 to get a discussion about that point started.

from nodejs-bigtable.