Comments (18)
Hey @mintbridge, if your custom processor is to be applied to all messages in your pipeline you can use Benthos as a framework like this: https://godoc.org/github.com/Jeffail/benthos/lib/stream#example-package--Base64Encoder, which would likely be the least amount of work for you.
Alternatively, an encrypt/decrypt processor sounds like it might be generally useful, in which case I'd be happy to include it within Benthos. It's slightly more work as you'd need to make a config driven processor similar to encode (https://github.com/Jeffail/benthos/blob/master/lib/processor/encode.go), but I'm happy to work with you to get it merged in.
from connect.
Awesome, yeah I was looking at the encode/decode functions as an example. Our use case is a PGP encrypted message, not sure if that will be too complex?
from connect.
If you could summarise the config fields you'd need for your use case I'll try and find commonalities with other types of encryption we might also want to offer. If there's a general set of parameters that isn't too large then it sounds like a good fit.
I imagine right now it could be stuff like private_key
, password
, public_key
, where the key fields would be read as paths. If we were then to make a hypothetical AES option it could use private_key
to mean key
and would expect it to be the raw contents, in which case maybe private_key
should be renamed key
.
It might be worth building a custom processor without worrying about config to start with just to get you unblocked, and then if you're willing to share the implementation afterwards we can discuss the configuration parts later.
from connect.
cool, yep ill get one built then push it up for discussion
from connect.
Hey @mintbridge, pull request looks good so far, the way you're using parts
is fine. No passphrase fields for now is fine, we can always add it if we need it later. I imagine it's reasonable to read the key once, but I'm a security novice. I'll ask around over the next few days, if you do the same then maybe someone can enlighten us, otherwise I say read once.
from connect.
ok, the key is read in once, not sure what else is needed now? I guess some more tests would be good but not entirely sure the best way to test it?
from connect.
Hey @mintbridge, yeah looks good. More unit tests would be good, maybe generate some small encrypted sample files and a mock key in a testdata
subdirectory, then for the decrypt
processor try and get the original contents and for the encrypt
processor try and match the encrypted files by giving it the original contents.
from connect.
added a test, but i'm not sure i'm understanding the parts properly, i think for this i need to combine them all to decrypt the full message?
from connect.
The lingo for messages is a little confusing, but each message part is a discrete message payload within a batch. So if you receive a message of three parts that would be synonymous with a batch of three messages. Sorry for the confusion, it's because multipart messages and message batches are the same concept within Benthos.
from connect.
Figured out what the issue is, the delimit param defaults to new line, which breaks up the ascii armoured message into parts and so cant be decrypted. The files input works ok as it reads the whole file as one. Is there a way to set the delimiter to EOF, go has io.EOF but we would need to work out a way to pass that from yaml/json. Maybe easier to default to EOF and let people set it as newline?
from connect.
Hey @mintbridge, for the use case you're describing, where you want to read a file in its entirety as a single message, the files
input would be my recommended way of doing that. The file
input is specifically for when a file contains a delimited set of payloads.
I should have time today to take a look at your pull request but it looked good to me last time I checked, it's just the unit tests needed some small adjustments.
from connect.
Hi @Jeffail @mintbridge, I was following along with the thread and wondered how you had made out. Is this processor on a separate branch or are there plans to merge it into master?
from connect.
Hey @homer6, I have a branch that is able to be merged here: https://github.com/Jeffail/benthos/tree/mintbridge-decrypt-processor, there's a PR from @mintbridge here: #62 but I closed it due to inactivity.
If you're interested in this would you be able to discuss your use cases? I'm hesitant to add something like this into Benthos unless I know there are active use cases for it.
from connect.
That makes sense. I don't have a use case for it at the moment. I was looking at it as an example of how to add components that are as merge-friendly as possible.
I'll open an issue with the design and use case of the component that I had in mind.
from connect.
Hey @mintbridge, closing this for now, feel free to reopen if you want to get this merged in. As far as I'm concerned the feature is ready to add in but I'm holding off until we have active stakeholders.
from connect.
Was this ever merged? we are interested on encrypting specific fields on a message
from connect.
@callmegar there's an encrypt method in bloblang now: https://www.benthos.dev/docs/guides/bloblang/methods#encrypt_aes
from connect.
The current encryption method doesn't include "GCM" which uses an IV (nonce) of 12 bytes - For now, I pad the IV with 00 to make it 16 bytes to pass the size check in Bloblang.
From my understanding, "GCM" should not be streamed to preserve its authenticity and has to be processed by block. Hence the current implementation which uses XORKeyStream
won't work.
@Jeffail do you think it's possible to include a new gcm
check in the main repository with an update something like below? I can do a pull request if required.
var schemeFn func([]byte) ([]byte, error)
switch schemeStr {
case "gcm":
schemeFn = func(b []byte) ([]byte, error) {
aesgcm, _ := cipher.NewGCM(block)
plaintext, _ := aesgcm.Open(nil, iv[:12], b, nil)
return plaintext, nil
}
case "ctr":
schemeFn = func(b []byte) ([]byte, error) {
plaintext := make([]byte, len(b))
stream := cipher.NewCTR(block, iv)
stream.XORKeyStream(plaintext, b)
return plaintext, nil
}
...
@callmegar there's an encrypt method in bloblang now: https://www.benthos.dev/docs/guides/bloblang/methods#encrypt_aes
from connect.
Related Issues (20)
- Pass along bloblang/yaml error context
- kafka_franz: No connection errors if `consumer_group: ""`
- redis_streams: support for `XAUTOCLAIM`
- Docs typo in Configuration: Templating
- sql_insert - high CPU usage mainly due to GC cycles and allocations. HOT 6
- Global options no longer work via rpk connect HOT 5
- Elasticsearch output backoff should honor HTTP code `429`
- Log rotation is extra aggressive on removing older log files
- Kafka_franz info HOT 1
- Emit `kafka_lag` metadata for the `kafka_franz` input similarly to the `kafka` input
- Add connector support levels to the connector source and templates
- S3 output missing header since aws-sdk-go-v2 upgrade HOT 1
- Contributing link leads nowhere
- Stored procedure output HOT 2
- How to overwrite the value of tracparent in tracing_span().traceparent
- Bloblang `with` method is not properly filtering arrays HOT 3
- Enhance the http_client output with additional error checking HOT 4
- Incorrect Usage: flag provided but not defined: -r when in streams mode HOT 2
- Failed to recover from broker unavailable using kafka_franz HOT 2
- Is there any option to retrieve error result from underlying output component that is wrapper inside Retry output component? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from connect.