Comments (15)
@marceloboeira I have an example plugin you can check out here: https://github.com/Jeffail/benthos/blob/feature/plugin-example/lib/input/plugins/mysql.go#L48
Let me know what you think of the RegisterPlugin
and DocumentPlugin
calls, it's fairly experimental right now so I'm looking for general feedback.
from connect.
Since this is a fairly high demand topic I'm repurposing this issue for a general revisit of SQL CDC within Benthos. CDC is handled very differently across databases and therefore we have two options:
- Lean heavily into a generalized CDC connector library such as https://github.com/wal-g/wal-g, this would require a significant amount of work/collaboration in order to get the integration right. From Benthos we could expose this as a general
sql_cdc
input similar to thesql
processor. - Have individual inputs per database such as
postgresql_wal
, allowing us to have a more granular configuration spec for each implementation.
Ideally I wouldn't want to burden the Benthos project with actively maintaining these inputs, and would therefore prefer to utilize a library that is intended to be consumed similar to a typical stream client library.
To start this one off I'd like for us to do another search for Go libraries in this space since a fair amount of time has passed.
from connect.
Also look at Conduit project and their postgres connector.
from connect.
Hey @marceloboeira, this sounds very similar to another MySQL based input we're looking at: #76
The current plan is to add it in as a plugin, the API for which is ready to use but some plumbing is still pending for generating configuration examples and documentation etc. I'd be interested in adding this as another plugin so your configuration would look something like this:
input:
type: pgsql
plugin:
... pgsql config fields here ...
If you can wait a few days for me to get the MySQL input merged you'll end up with a good example to base yours on.
from connect.
I've been testing out wal-listener quite extensively the last 24h. I think it would be an incredibly nice feature in benthos to have a similar interface.
from connect.
Think implementing CDC in Benthos would be a great step forward and it would attract new users. I was looking at some projects and here are some I liked:
- https://github.com/ihippik/wal-listener
- https://github.com/Nextdoor/pg-bifrost
- https://github.com/hasura/pgdeltastream
- https://github.com/jackc/pglogrepl
from connect.
Another repo to check out (from #570): https://github.com/tmc/pqstream
from connect.
In the direction of per database connections, here is another lib that handles postgres logical decoding well:
https://github.com/kyleconroy/pgoutput
I liked what pgdeltastream is doing as well.
Main difference between the two is pgoutput creates the replication slot with the plugin pgoutput
, the standard logical replication decoder. While pgdeltastream uses the plugin wal2json
, which has the benefit of being immediately human readable, but the downside is it needs to be installed (though many postgres providers like aws rds have it installed out of the box).
from connect.
Blocked on: #83
from connect.
Hey
I am trying to use Benthos to parse database logs ( WAL)...for Oracle, mySQL, PostreSQL
@marceloboeira I have an example plugin you can check out here: https://github.com/Jeffail/benthos/blob/feature/plugin-example/lib/input/plugins/mysql.go#L48
The link above is dead. "Shes dead jim, shes dead"
Let me know what you think of the
RegisterPlugin
andDocumentPlugin
calls, it's fairly experimental right now so I'm looking for general feedback.
from connect.
Hey @gedw99, the plugin examples can now be found here: https://github.com/benthosdev/benthos-plugin-example, but there isn't one for WAL I'm afraid.
from connect.
I guess the WAL part we'll probably figure it out with wal-g and similars, and then have a plugin to hook it to benthos. Thanks @Jeffail for the update.
from connect.
thanks all for the update.
Yeah playing with the following stuff to do this over at https://github.com/14-bits/cdc. The plan is to get it working with benthos, but lots of database parsing to do first.
Playing around with these libs to do CDC ( Change Capture Control) of the databases to feed into Benthos.
- universal sql parser.
https://github.com/wal-g/wal-g
- postresql and mysql cdc wal parsing
https://github.com/moiot/gravity
- mysql and tidb cdc wal parsing
- uses kafka, but easy to add NATS
from connect.
I don't need support for this (yet), but did some quick digging and thought that mixing Benthos with https://github.com/prest/prest might be interesting.
from connect.
Likewise conduit's vitess/mysql connector
https://github.com/conduitio-labs/conduit-connector-vitess
from connect.
Related Issues (20)
- redis_streams: support for `XAUTOCLAIM`
- Docs typo in Configuration: Templating
- sql_insert - high CPU usage mainly due to GC cycles and allocations. HOT 6
- Global options no longer work via rpk connect HOT 5
- Elasticsearch output backoff should honor HTTP code `429`
- Log rotation is extra aggressive on removing older log files
- Kafka_franz info HOT 1
- Emit `kafka_lag` metadata for the `kafka_franz` input similarly to the `kafka` input
- Add connector support levels to the connector source and templates
- S3 output missing header since aws-sdk-go-v2 upgrade HOT 1
- Contributing link leads nowhere
- Stored procedure output HOT 2
- How to overwrite the value of tracparent in tracing_span().traceparent
- Bloblang `with` method is not properly filtering arrays HOT 3
- Enhance the http_client output with additional error checking HOT 4
- Incorrect Usage: flag provided but not defined: -r when in streams mode HOT 2
- Failed to recover from broker unavailable using kafka_franz HOT 2
- Is there any option to retrieve error result from underlying output component that is wrapper inside Retry output component? HOT 1
- sql_insert ORA-01461: can bind a LONG value only for insert into a LONG column HOT 3
- add name to the mongo input/output for tracking w/ mongodb
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from connect.