uswitch / big-replicate Goto Github PK
View Code? Open in Web Editor NEWReplicates data between Google Cloud BigQuery projects
Replicates data between Google Cloud BigQuery projects
Currently just the project is overwritten, it might be useful to also be able to set a different dataset (perhaps for copying data within a project from a US to an EU located dataset).
When attempting to replicate a table with a DATE type field, the type is not found in the Field.Type enum. Assuming missing support for the newer Standard SQL datatypes in the version of BigQuery library:
exception #error {
:cause No enum constant com.google.cloud.bigquery.Field.Type.Value.DATE
:via
[{:type java.lang.IllegalArgumentException
:message No enum constant com.google.cloud.bigquery.Field.Type.Value.DATE
:at [java.lang.Enum valueOf Enum.java 238]}]
:trace
[[java.lang.Enum valueOf Enum.java 238]
[com.google.cloud.bigquery.Field$Type$Value valueOf Field.java 75]
[com.google.cloud.bigquery.Field fromPb Field.java 368]
[com.google.cloud.bigquery.Field$1 apply Field.java 46]
[com.google.cloud.bigquery.Field$1 apply Field.java 43]
[com.google.common.collect.Lists$TransformingRandomAccessList$1 transform Lists.java 640]
[com.google.common.collect.TransformedIterator next TransformedIterator.java 48]
[java.util.AbstractCollection toArray AbstractCollection.java 141]
[java.util.ArrayList <init> ArrayList.java 177]
[com.google.common.collect.Lists newArrayList Lists.java 146]
[com.google.cloud.bigquery.Schema$Builder fields Schema.java 78]
[com.google.cloud.bigquery.Schema of Schema.java 151]
[com.google.cloud.bigquery.Schema fromPb Schema.java 159]
[com.google.cloud.bigquery.TableDefinition$Builder <init> TableDefinition.java 88]
[com.google.cloud.bigquery.StandardTableDefinition$Builder <init> StandardTableDefinition.java 140]
[com.google.cloud.bigquery.StandardTableDefinition$Builder <init> StandardTableDefinition.java 119]
[com.google.cloud.bigquery.StandardTableDefinition fromPb StandardTableDefinition.java 283]
[com.google.cloud.bigquery.TableDefinition fromPb TableDefinition.java 172]
[com.google.cloud.bigquery.TableInfo$BuilderImpl <init> TableInfo.java 157]
[com.google.cloud.bigquery.Table fromPb Table.java 348]
[com.google.cloud.bigquery.BigQueryImpl getTable BigQueryImpl.java 353]
[sun.reflect.GeneratedMethodAccessor18 invoke nil -1]
[sun.reflect.DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl.java 43]
[java.lang.reflect.Method invoke Method.java 498]
[clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
[clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
[gclouj.bigquery$table invokeStatic bigquery.clj 142]
[gclouj.bigquery$table invoke bigquery.clj 141]
[uswitch.big_replicate.sync$load_table invokeStatic sync.clj 92]
[uswitch.big_replicate.sync$load_table invoke sync.clj 91]
[uswitch.big_replicate.sync$progress$fn__7851 invoke sync.clj 129]
[uswitch.big_replicate.sync$progress invokeStatic sync.clj 129]
[uswitch.big_replicate.sync$progress invoke sync.clj 121]
[uswitch.big_replicate.sync$replicator_agent$fn__7855 invoke sync.clj 138]
[clojure.core.async$thread_call$fn__6122 invoke async.clj 439]
[clojure.lang.AFn run AFn.java 22]
[java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1142]
[java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 617]
[java.lang.Thread run Thread.java 745]]}
09:13:56.977 [async-thread-macro-1] INFO uswitch.big-replicate.sync - syncing 0 tables:
09:13:56.977 [main] INFO uswitch.big-replicate.sync - creating 8 replicator agents
Creates new releases without pushing tags.
Staging data is not automatically deleted once its been loaded into the destination table. It'll gradually fill up over time so it'd be nice to automatically tidy once the load has finished successfully.
The BigQuery API has a Copy job that will copy data between tables. It should be possible to merge the use of separate extract + load jobs into just the copy job. It'll also mean there's no need to manage Cloud Storage used currently for intermediate data.
gclouj will need to change to add support for building the job configuration: CopyJobConfiguration.
The tool currently assumes its replicating only Google Analytics exported data. It would be nice to change this to allow a table regexp to be specified on the cli so its less specific to GA data.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.