arquivei / goduck Goto Github PK
View Code? Open in Web Editor NEWSimple and extensible stream/pool consumer
License: BSD 3-Clause "New" or "Revised" License
Simple and extensible stream/pool consumer
License: BSD 3-Clause "New" or "Revised" License
A lot of messaging systems allow the user to send custom metadata attached to the message. But with the current goduck implementation, it is impossible to read those values, since the RawMessage interface allows to extract only the bytes from the content of the message.
Specifically for Streams, there is usually a special metadata called "key", from which the ordering guarantee comes. For now, it is not possible to read a message's key, which could be limiting.
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These problems occurred while renovating this repository. View logs.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
cloud.google.com/go/bigquery
, cloud.google.com/go/pubsub
, cloud.google.com/go/storage
, github.com/IBM/sarama
, github.com/arquivei/foundationkit
, github.com/confluentinc/confluent-kafka-go/v2
, github.com/googleapis/gax-go/v2
, golang.org/x/sync
).github/workflows/go.yml
actions/checkout v4
actions/setup-go v5
.github/workflows/golangci-lint.yml
actions/setup-go v5
actions/checkout v4
golangci/golangci-lint-action v6
go.mod
go 1.21
cloud.google.com/go/bigquery v1.61.0
cloud.google.com/go/pubsub v1.38.0
github.com/IBM/sarama v1.43.2
github.com/arquivei/foundationkit v0.9.0
github.com/confluentinc/confluent-kafka-go/v2 v2.4.0
github.com/go-kit/kit v0.13.0
github.com/imkira/go-observer v1.0.3
github.com/olivere/elastic/v7 v7.0.32
github.com/rs/zerolog v1.33.0
github.com/segmentio/kafka-go v0.4.47
github.com/stretchr/testify v1.9.0
cloud.google.com/go/storage v1.41.0
github.com/googleapis/gax-go/v2 v2.12.4
golang.org/x/sync v0.7.0
As soon as #12 is implemented, it is possible to implement a custom engine, built on top of the StreamEngine, that transforms a (key,value) stream into a in-memory table.
This would be similar to the KTable in kafka streams. The stream is materialized into a RocksDB table, which can be stored both in the memory and in the disk.
Things to bear in mind:
To configure goduckStreams, there is a need to instantiate kafka (config and producer) using the same version of kafka in the goduck project, which ends up generating overhead and the need to change structures in several projects to update dependencies.
Proposal for a solution: creation of a Goduck.ConfigMap as type alias of confluent kafka types, so that the dependency of kafka types from confluent is only on the side of goduck and not in external projects.
Elasticsink already has an option pattern. Use this pattern to create another option to limit the bulkRequest
length in Store
method.
https://github.com/arquivei/goduck/blob/master/pipeline/pubsubsink/adapter_pubsub.go#L20
This functions obtains the pubsub topic for a given topic name, but doesn't cache the information. For high-throughput pipelines, this causes RATE_LIMIT_EXCEEDED for administratorPerMinutePerProject
The first implementation of a stream backed by kafka was done using segmentio's library, in the folder goduck/impl/implstream/
. But now that there are other implementations, it should be done in goduck/impl/implstream/kafkasegmentio
.
This should be done in a backward compatible way, marking the old code as deprecated before deletion.
pubsubsink waits for the last message to be completely delivered before sending the next one.
This is inefficient, because if we don't batch the requests, the throughput is limited to 1/write_latency
.
A solution would be to .Get
the result only once all messages have been published.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.