Comments (14)
Adapted: https://github.com/Jeffail/benthos/blob/feature/plugin-example/lib/input/plugins/mysql.go#L48
from connect.
Changing direction with this, the implementation of a plugin will remain the same but the way in which they are built and run is ideally going to fit into one of two categories:
Building Public Plugins
Shared plugins won't be merged into the main Benthos codebase, and therefore won't be part of the main distributed binary. Instead, you will be recommended to create a plugin project (the template for this is TBD) where the compiled plugin (a .so
file) can simply be dropped into a plugins
directory where a Benthos binary can read and use them.
The ability for Benthos to do this has already been added in: 128e896. However, there are issues in creating these plugins that need to be resolved.
Building Private Plugins
If you are building plugins intended for private use then advice is going to be to compile them into either a fork or use Benthos as a framework in a new project. Framework is recommended since it means easier upgrades to track the main project.
However, if you intend to also use public shared plugins you should create them the same way (and just don't share them.)
from connect.
Issues
I've been reading up on Go plugins and experimenting with Benthos and it seems as though the versioning of a plugin is extremely strict. Not only does a plugin need to compile against the exact same dependency tree as the Benthos version it is loaded by, but it is also not able to use vendoring for this purpose, and neither can Benthos. Even the smallest difference (like, for example, the value of GOPATH) between the Benthos build and the plugin build seem to break the plugin.
It seems as though the Go team have primarily implemented plugins with the assumption that they are compiled along with the binary they are loaded with. Since that completely defeats the purpose of our use of plugins we will have to find a way of working around that. I imagine this will require isolating the Benthos build system so that it can be exactly reproduced by any teams building a plugin.
Short term goals
I need to change the Benthos build system to no longer use vendoring. The vendor directory isn't yet checked into master, it was my intention to do this eventually but since we might have to scrap vendoring I'll simply proceed as if that was never the case.
The build system also needs to be designed to make it extremely easy for other teams to reproduce the exact conditions of the packaged builds. This needs to be the case for both the binary distributions and the docker images.
Long term goals
Work out how we're going to advise people to distribute their plugins, considering that versioning is so crucial here this won't be trivial
from connect.
Wow, I guess that’s why terraform uses a custom plugin implementation https://github.com/hashicorp/go-plugin. What are your thoughts on their approach? I’m not sure it fits the benthos use case, but the issues you’ve described above with native plugins seem to make this approach a no go from my perspective. The only way it seems feasible is if all plugins were included in the benthos repo and built along with the core, which kind of defeats the point. Guess I need to think about this some more
from connect.
I'd be worried about the overhead of using RPC, could always give it a go. I've got an example of a plugin using official Go plugins: https://github.com/Jeffail/benthos-plugin-example, it's actually very simple, would be great if we could find a nice way of doing the builds.
from connect.
Don't get me wrong, the example implementation looks awesome! and I agree with the RPC concerns. However, after doing some more reading on go plugins, the limitations seem to be a pretty difficult problem to solve.
from connect.
Hi, where I can see this plugin example: https://github.com/Jeffail/benthos/blob/feature/plugin-example/lib/input/plugins/mysql.go#L48 - this link is expired
from connect.
Hey @silentnull, sorry, I've updated the first post as there's now a project demoing this: https://github.com/Jeffail/benthos-plugin-example
from connect.
Hey @Jeffail . Yep, I saw this repo, but there is only simple benthos plugin example, but I will extended example with mysql :) Is possible get source code of plugin for mysql ?
from connect.
@silentnull ah right, I've pushed that example branch back so you can see, the link should work again now. I'll make sure it exists in the example project before I delete again.
from connect.
Hey @Jeffail . Yes, now the link works 👍 Thank you.
from connect.
Hey @Jeffail . I have a question to Manager type. It has function GetCache. Function works as Get property and returns Cache object. Where this object is declared ?
The second question is that in MySql sample is returned input.NewReader object, but in plugins example project (separetly repo) your return the "Example" object instance ? Why ?
from connect.
@silentnull, the caches are defined in the resources.caches
section of a config file, the list of types is here: https://github.com/Jeffail/benthos/tree/master/docs/caches
The plugin example implements the types.Input
interface. The MySQL type implements https://github.com/Jeffail/benthos/blob/master/lib/input/reader/interface.go#L31 which is simpler, and then it gets turned into a types.Input
with input.NewReader
.
The choice is yours really, the main advantage of using input.NewReader
is that it handles reconnections, retries, metrics and logging for you. The advantage of implementing types.Input
directly is that it gives you more granular control.
from connect.
Closing this for now, we have https://github.com/benthosdev/benthos-plugin-example and my advice from here onwards is going to be to maintain your own main
func and use Benthos as a library. Go plugins would have been great but it doesn't look like they're going to be what we need, at least in the foreseeable future.
from connect.
Related Issues (20)
- redis_streams: support for `XAUTOCLAIM`
- Docs typo in Configuration: Templating
- sql_insert - high CPU usage mainly due to GC cycles and allocations. HOT 6
- Global options no longer work via rpk connect HOT 5
- Elasticsearch output backoff should honor HTTP code `429`
- Log rotation is extra aggressive on removing older log files
- Kafka_franz info HOT 1
- Emit `kafka_lag` metadata for the `kafka_franz` input similarly to the `kafka` input
- Add connector support levels to the connector source and templates
- S3 output missing header since aws-sdk-go-v2 upgrade HOT 1
- Contributing link leads nowhere
- Stored procedure output HOT 2
- How to overwrite the value of tracparent in tracing_span().traceparent
- Bloblang `with` method is not properly filtering arrays HOT 3
- Enhance the http_client output with additional error checking HOT 4
- Incorrect Usage: flag provided but not defined: -r when in streams mode HOT 2
- Failed to recover from broker unavailable using kafka_franz HOT 2
- Is there any option to retrieve error result from underlying output component that is wrapper inside Retry output component? HOT 1
- sql_insert ORA-01461: can bind a LONG value only for insert into a LONG column HOT 3
- add name to the mongo input/output for tracking w/ mongodb
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from connect.