[clinical] Fix out of order columns in the per-collection tables about etl HOT 5 CLOSED

I had originally detected the column types using pandas but I'll try auto-detection in BigQuery.

from etl.

So auto-detect in BigQuery will not work because the clinical data is sparse. Some patients rows have missing columns. But I figured out how to do this using the predetermined schema. This column order is now fixed in idc-dev-etl.clinical dataset

from etl.

So auto-detect in BigQuery will not work because the clinical data is sparse. Some patients rows have missing columns.

I don't understand that. I understand individual patients may have empty values in certain columns, but the columns are defined per-table, how can they be missing?

from etl.

In the JSON array used to populate the tables, patient rows will have missing attributes . The first patient will have one set of defined attributes, and the 2nd patient will have a somewhat different set etc. This makes it tricky to auto-detect the column order. I expect that, using auto-detect, the columns are created in the order the corresponding attributes are first encountered.

from etl.

This has been resolved.

from etl.