Comments (5)
nevillelyh
commented on May 18, 2024
This is the API I'm referring to, which allows saving a single PCollection into multiple BigQuery tables with a BoundedWindow => TableReference function.
from scio.
nevillelyh
commented on May 18, 2024
Not sure if this should be part of Scio though, since it breaks the JobTest semantics. Plus the user can do SCollection#saveAsCustomOutput now.
from scio.
andrewsmartin
commented on May 18, 2024
Can we close this?
from scio.
nevillelyh
commented on May 18, 2024
Let's leave it open for a bit longer. I might think about a way to do this properly.
from scio.
nevillelyh
commented on May 18, 2024
@samschlegel also found this: apache/beam@35db745
We can maybe add these as methods on a implicit class and to skip JobTest support.
from scio.
Related Issues (20)
- new StateBaseAsyncDoFn with State and Timers HOT 5
- Document Avro version support on GH site HOT 1
- Drop java 8 support
- Add java 21 Support HOT 1
- Support Taps for SMB writes/transforms HOT 1
- Support projections in ParquetAvroFileOperations/ParquetAvroSortedBucketIO HOT 1
- Flaky test in SCollectionTest
- Flaky test in ParquetExampleIOTest
- Expose extendedErrorInfo in all BQ APIs
- Document BigTableDoFn
- Using sparkey largeHashIntersectByKey blocks pipeline on Dataflow v1
- ParquetAvroTap does not set projection or dataSupplier HOT 2
- ParquetAvroFileOperations does not set dataSupplier
- Make SplittableDoFn the default Parquet read implementation HOT 2
- StorageUtil.toTableSchema fails on BigNumeric BQ column
- com.spotify.scio.bigquery.client.BigQuery.defaultInstance().run does not respect destinationTable=null setting for UML HOT 1
- ParquetAvroDynamicTest is flaky HOT 2
- ByteArrayCoder sometimes produces encodings that match Deflate Codec header
- Deadlock in AsyncLookupDoFn HOT 1
- Support setting numShards > 1 in SMB transforms
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scio.