Comments (4)
Hi @cesaraugusto98, Yes, both bronze and silver schemas are managed by Spark (one logical db). The gold schema is managed by Postgres. You need to run CTAS queries that read data from Iceberg and store data in Postgres when moving data from silver to gold. Trino is much better for such federated queries.
from ngods-stocks.
Hi @zsvoboda. How about using only Trino and a data lake framework (Delta, Iceberg) for everything (bronze, silver, gold). Or am I missing something that only Spark would be capable of?
from ngods-stocks.
This is certainly an option. However, Spark is more resilient for larger datasets (10s of billions of rows). I was running into memory issues with Trino myself.
from ngods-stocks.
from ngods-stocks.
Related Issues (16)
- would you please provide complete docker-compose.yaml with all components(like trino,dbt)? HOT 18
- new
- can not run sql in cube via jdbc HOT 1
- Kyuubi-1.6.0 not supported HOT 1
- Datahub wont start in the docker compose setup HOT 1
- Custom CSV
- CDC - Is it part of Dagster or somewhere else?
- Metabase stuck in a loop HOT 3
- Error starting containers
- Thoughts on using software defined assets and a Trino IO Manager?
- requirements.txt - Python dependency resolution stops with an error HOT 3
- Error when re-running the workflows HOT 8
- exec /usr/bin/entrypoint.sh: no such file or directory HOT 3
- [Question] Is iceberg a valid file format? HOT 1
- Trino Error HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ngods-stocks.