Add format transformation for json-like input for BigQueryConverters about dataflowtemplates HOT 4 OPEN

I see the Context.OUTER being deprecated, is there any alternative to be used for this? Please help.

from dataflowtemplates.

A small callout for making change to the convertJsonToTableRow function....
Sometimes double-quote as special characters may be part of json key/value as well...
example like this ... https://developer.salesforce.com/forums/?id=906F0000000B3HuIAK

from dataflowtemplates.

Created a pubsub to GCS plain text pipeline to compare the pipeline that work and that doesn't.

BigQuery cannot deal with the non-standard json which was output from Kafka and then to pubsub
{SPEED=71.2, FREEWAY_ID=163, TIMESTAMP=2008-11-01 00:00:00, LONGITUDE=-117.155519, FREEWAY_DIR=S, LATITUDE=32.749679, LANE=399999}

Directly pushing standard json to pubsub will have the pipeline work fine when BQ received correct format of json
{"LANE": "3", "SPEED": "72", "TIMESTAMP": "2008-11-01 00:00:00", "LONGITUDE": "-117.089026", "LATITUDE": "32.780922", "FREEWAY_ID": "8", "FREEWAY_DIR": "W"}

from dataflowtemplates.

One guess is that we configured the following for Kafka configuration files, which might have caused Kafka to received correct format of json and output in non-standard format (and encoded in base64).

key.converter.schemas.enable=false
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.storage.StringConverter

from dataflowtemplates.