Comments (9)
Reviewed all the inputs and added metadata from the ones I felt made sense.
from connect.
I've been working on a version of the amqp input that includes all but Acknowledger and MessageCount from the amqp.Delivery struct. Still working out the nesting of amqp.Table in Delivery.Headers. I'll send you a PR when I'm done to see if you can use any of it.
from connect.
I've just pushed an API change that allows per message metadata in batches and updated all the existing components to use it.
from connect.
What are your thoughts on allowing metadata on message
s in addition to part
s? I have a processor that is turning a multi-part message into N new messages (basically performing a group-by) and it'd be handy if I could enrich each outgoing message
with metadata about the grouping.
from connect.
Hey @seanlaff, how would you intend on using the metadata? For cases like that I would usually either suggest setting and referring to the batch wide metadata on the first part, or just set the field on all parts.
from connect.
@Jeffail Inspecting the metadata on the first part is the solution I opted for in the end, and works pretty well. My use case may be a little contrived, but this is what it looks like:
Data is streamed from pubsub to a benthos batch processor, and then to a database sink. However the database table that the data is written into depends on a dynamic field in the message. Additionally, the database does not support writing to multiple tables in one request.
My first solution was to spin up goroutines in the output to handle writing to the N discrete tables concurrently- but that felt like work I could relegate to benthos itself. Instead, I opted to create a processor (executed after the batch processor) that creates N new multi-part message, with each message containing all parts that share the same keying field. Then in my output I inspect the metadata of the first part, and make a singular database write for the correct table for all parts of that message at once (thus leaning on benthos to do the concurrent work I was doing by hand before)
This may be a non-idomatic use of benthos- and is only a minor inconvenience since I can achieve the same by checking the metadata on the first part
from connect.
@seanlaff, sounds like a reasonable approach. Let me know if you hit any issues with doing it this way and I'll take another look.
from connect.
@Jeffail I've raised a PR for a candidate implementation of metadata with the Kafka output type - happy to address any comments there.
from connect.
Closing this as it's become stale. The bulk of the work is done, anything else can be tackled if/when people need it.
from connect.
Related Issues (20)
- output component fallback not work HOT 1
- [Feature Request] Trim whitespaces from columns in CSV scanner/input HOT 1
- aws_s3: Scanner and backing reader not closed on non `io.EOF` error
- 4.28.0 should really be 5.0.0 HOT 3
- Document change in licensing HOT 2
- Inconsistent behavior with Javascript processing HOT 1
- JavaScript processor unable to handle asynchronous code execution HOT 1
- Iceberg support HOT 1
- Esto serΓ‘ facil
- MwM
- Exclude enterprise licensed plugins from the all package HOT 2
- main.go seems to be Redpanda Enterprise licensed HOT 2
- Document workflow/result_map
- CLI references the wrong binary name
- Collaborate on a Benthos processor for a Conduit pipeline?
- Dependency Licensing issue caused by couchbase/gocbcoreps HOT 3
- aws_kinesis input: shards are not processed if they are closed HOT 4
- public free bundle missing the xml package import HOT 2
- Pass along bloblang/yaml error context
- kafka_franz: No connection errors if `consumer_group: ""`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from connect.