Integrate the sdk with rdkafka, creating a new integration crate cloudevents-sdk-rdkafka

More info: https://pipelines.actions.githubusercontent.com/6C9UN8OKv3LCAcpoFIbcGjz0OJvuOblmKW8cJH8an47RVH3iPp/_apis/pipelines/1/runs/203/signedlogcontent/3?urlExpires=2020-08-06T13%3A14%3A53.0443581Z&urlSigningMethod=HMACV1&urlSignature=JhF9VJoZZVAHzPL5Puyr%2BbV3%2Bi0Ndszgjn%2F1pv96KEM%3D

Before going 0.1, we need to make source and dataschema attributes typed:

• source is a RFC3986 URI Ref
• dataschema is a RFC3986 URI

We need a library that implements RFC3986 where the types are owned.

@Lazzaretti @linuxbasic any hints?

At the moment our "subcrates" doesn't have any docs in the main crate file lib.rs, we should add a doc with a little code example to explain the features of the crate and some pointers to the main doc of cloudevents-sdk

Assuming you have an event with:

{
  "datacontenttype": "application/json",
  "data_base64": "… non JSON …",
}

Then it fails to deserialize the event, as it tries to parse the value as JSON:

Citing the spec:

When a CloudEvents is deserialized from JSON, the presence of the data_base64 member clearly indicates that the value is a Base64 encoded binary data, which the serializer MUST decode into a binary runtime data type.

So parsing this into JSON is OK I guess, but a fallback of Binary must also be possible IMHO if that fails.

Hi All
Testing the tide websocket implementation using the example message from
https://github.com/cloudevents/spec/blob/master/json-format.md#32-examples

{
    "specversion" : "1.0",
    "type" : "com.example.someevent",
    "source" : "/mycontext",
    "id" : "A234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "unsetextension": null,
    "datacontenttype" : "text/xml",
    "data" : "<much wow=\"xml\"/>"
}

Specifically this line 205e16a#diff-97633925b0ac8233888bd3bdc2313999cd86c04f1e7e56a62a8cb1e8d58ba5eaR50

let v: Event = serde_json::from_str(&message).unwrap(); 

fails with

thread 'async-std/runtime' panicked at 'called `Result::unwrap()` on an 
`Err` value: Error("data did not match any variant of untagged enum ExtensionValue", 
line: 0, column: 0)', src/main.rs:50:71

If I remove the property set to null "unsetextension": null, the message works as expected.

{
    "specversion" : "1.0",
    "type" : "com.example.someevent",
    "source" : "/mycontext",
    "id" : "A234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "datacontenttype" : "text/xml",
    "data" : "<much wow=\"xml\"/>"
}

According to the docs null should be treated as unset or omitted
https://github.com/cloudevents/spec/blob/70823088e52b393b4251abd1ab03b6cfa233d427/json-format.md#22-type-system-mapping.

Do you have any preferences for which it should be?

There is this nice checklist on things to do to develop a nice api in rust, we should check it out and see if there is room of improvements in our APIs https://rust-lang.github.io/api-guidelines/checklist.html

Rust API Guidelines Checklist

• Naming (crate aligns with Rust naming conventions)
  • Casing conforms to RFC 430 ([C-CASE])
  • Ad-hoc conversions follow as_, to_, into_ conventions ([C-CONV]) #87
  • Getter names follow Rust convention ([C-GETTER]) #88
  • Methods on collections that produce iterators follow iter, iter_mut, into_iter ([C-ITER])
  • Iterator type names match the methods that produce them ([C-ITER-TY]) #89
  • Feature names are free of placeholder words ([C-FEATURE])
  • Names use a consistent word order ([C-WORD-ORDER]) #90
• Interoperability (crate interacts nicely with other library functionality)
  • Types eagerly implement common traits ([C-COMMON-TRAITS])
    • Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Debug,
      Display, Default #91
  • Conversions use the standard traits From, AsRef, AsMut ([C-CONV-TRAITS])
  • Collections implement FromIterator and Extend ([C-COLLECT])
  • Data structures implement Serde's Serialize, Deserialize ([C-SERDE])
  • Types are Send and Sync where possible ([C-SEND-SYNC]) #90
  • Error types are meaningful and well-behaved ([C-GOOD-ERR]) #90
  • Binary number types provide Hex, Octal, Binary formatting ([C-NUM-FMT])
  • Generic reader/writer functions take R: Read and W: Write by value ([C-RW-VALUE])
• Macros (crate presents well-behaved macros)
  • Input syntax is evocative of the output ([C-EVOCATIVE])
  • Macros compose well with attributes ([C-MACRO-ATTR])
  • Item macros work anywhere that items are allowed ([C-ANYWHERE])
  • Item macros support visibility specifiers ([C-MACRO-VIS])
  • Type fragments are flexible ([C-MACRO-TY])
• Documentation (crate is abundantly documented)
  • Crate level docs are thorough and include examples ([C-CRATE-DOC]) #95
  • All items have a rustdoc example ([C-EXAMPLE])
  • Examples use ?, not try!, not unwrap ([C-QUESTION-MARK]) #95
  • Function docs include error, panic, and safety considerations ([C-FAILURE]) #95
  • Prose contains hyperlinks to relevant things ([C-LINK]) #95
  • Cargo.toml includes all common metadata ([C-METADATA]) #95
    • authors, description, license, homepage, documentation, repository,
      readme, keywords, categories
  • Crate sets html_root_url attribute "https://docs.rs/CRATE/X.Y.Z" ([C-HTML-ROOT]) #95
  • Release notes document all significant changes ([C-RELNOTES])
  • Rustdoc does not show unhelpful implementation details ([C-HIDDEN]) #95
• Predictability (crate enables legible code that acts how it looks)
  • Smart pointers do not add inherent methods ([C-SMART-PTR])
  • Conversions live on the most specific type involved ([C-CONV-SPECIFIC])
  • Functions with a clear receiver are methods ([C-METHOD])
  • Functions do not take out-parameters ([C-NO-OUT])
  • Operator overloads are unsurprising ([C-OVERLOAD])
  • Only smart pointers implement Deref and DerefMut ([C-DEREF])
  • Constructors are static, inherent methods ([C-CTOR])
• Flexibility (crate supports diverse real-world use cases)
  • Functions expose intermediate results to avoid duplicate work ([C-INTERMEDIATE])
  • Caller decides where to copy and place data ([C-CALLER-CONTROL])
  • Functions minimize assumptions about parameters by using generics ([C-GENERIC])
  • Traits are object-safe if they may be useful as a trait object ([C-OBJECT])
• Type safety (crate leverages the type system effectively)
  • Newtypes provide static distinctions ([C-NEWTYPE])
  • Arguments convey meaning through types, not bool or Option ([C-CUSTOM-TYPE])
  • Types for a set of flags are bitflags, not enums ([C-BITFLAG])
  • Builders enable construction of complex values ([C-BUILDER])
• Dependability (crate is unlikely to do the wrong thing)
  • Functions validate their arguments ([C-VALIDATE])
  • Destructors never fail ([C-DTOR-FAIL])
  • Destructors that may block have alternatives ([C-DTOR-BLOCK])
• Debuggability (crate is conducive to easy debugging)
  • All public types implement Debug ([C-DEBUG]) #91
  • Debug representation is never empty ([C-DEBUG-NONEMPTY]) #91
• Future proofing (crate is free to improve without breaking users' code)
  • Sealed traits protect against downstream implementations ([C-SEALED]) #95
  • Structs have private fields ([C-STRUCT-PRIVATE])
  • Newtypes encapsulate implementation details ([C-NEWTYPE-HIDE])
  • Data structures do not duplicate derived trait bounds ([C-STRUCT-BOUNDS])
• Necessities (to whom they matter, they really matter)
  • Public dependencies of a stable crate are stable ([C-STABLE])
  • Crate and its dependencies have a permissive license ([C-PERMISSIVE])

Emojis are not shown in crates.io: https://crates.io/crates/cloudevents-sdk

Updating serde to latest breaks cloudevents-sdk

cargo build
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.58
   Compiling autocfg v1.0.1
   Compiling libc v0.2.82
   Compiling serde_derive v1.0.119
   Compiling serde v1.0.119
   Compiling tinyvec_macros v0.1.0
   Compiling matches v0.1.8
   Compiling doc-comment v0.3.3
   Compiling getrandom v0.2.1
   Compiling ryu v1.0.5
   Compiling serde_json v1.0.61
   Compiling percent-encoding v2.1.0
   Compiling cfg-if v1.0.0
   Compiling match_cfg v0.1.0
   Compiling itoa v0.4.7
   Compiling base64 v0.12.3
   Compiling unicode-bidi v0.3.4
   Compiling tinyvec v1.1.0
   Compiling form_urlencoded v1.0.0
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling quote v1.0.8
   Compiling time v0.1.43
   Compiling hostname v0.3.1
   Compiling uuid v0.8.2
   Compiling unicode-normalization v0.1.16
   Compiling ordered-float v1.1.1
   Compiling idna v0.2.0
   Compiling snafu-derive v0.6.10
   Compiling delegate-attr v0.2.9
   Compiling snafu v0.6.10
   Compiling url v2.2.0
   Compiling chrono v0.4.19
   Compiling serde-value v0.6.0
   Compiling cloudevents-sdk v0.3.0
error[E0603]: module `export` is private
   --> /home/awebber/.cargo/registry/src/github.com-1ecc6299db9ec823/cloudevents-sdk-0.3.0/src/event/data.rs:1:12
    |
1   | use serde::export::Formatter;
    |            ^^^^^^ private module
    |
note: the module `export` is defined here
   --> /home/awebber/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.119/src/lib.rs:275:5
    |
275 | use self::__private as export;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0603]: module `export` is private
   --> /home/awebber/.cargo/registry/src/github.com-1ecc6299db9ec823/cloudevents-sdk-0.3.0/src/event/spec_version.rs:2:12
    |
2   | use serde::export::Formatter;
    |            ^^^^^^ private module
    |
note: the module `export` is defined here
   --> /home/awebber/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.119/src/lib.rs:275:5
    |
275 | use self::__private as export;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0603`.
error: could not compile `cloudevents-sdk`

To learn more, run the command again with --verbose.

Current workaround is rolling back to serde version 1.0.118

See related fix teloxide/teloxide#328

[dependencies]
cloudevents-sdk = "0.3.0"
serde = { version="=1.0.118", features = ["derive"] }

The actual code coverage of rdkafka integration is poor, we should increase it and hopefully add some tests bootstrapping kafka on CI too

Event should expose a function that allows to iterate over available attributes

At the moment all the examples doesn't compile on CI. We should at least run cargo check when somebody push a new PR to check if the examples break.

For some reason, some docs intra links seems broken: Here only some works https://docs.rs/cloudevents-sdk-actix-web/0.2.0/cloudevents_sdk_actix_web/ while here all of them works https://docs.rs/cloudevents-sdk/0.2.0/cloudevents/

We need those links fixed in our next release

The actual APIs makes really hard to destructure the Event data structure and take ownership of attributes, extensions and data separately. We should think to new facilities to improve the ergonomics of the Event API.

There is a lot of redundant logic around headers for each binding that we can probably consolidate into a shared lib.

@slinkydeveloper I'm thinking I can replace the ATTRIBUTES_TO_HEADERS map with a simple function that returns strings rather than HeaderNames. That way, I can eliminate all the lazy_static! stuff. It would simplify the code considerably and I can't imagine it would have any significant impact on performance. Agree?

• attributes_iter should be renamed to iter_attributes()
• get_extensions should return an iter (not a vec) and should be renamed to iter_extensions()

Right now the APIs are:

event_to_request(input, client.post(&url))
  .unwrap()
  .send()

I think it would be nicer to have something like:

client.post(&url)
  .event(input).unwrap() // This still need to fail somehow
  .send()

This applies both to request and responses for both actix-web and reqwest modules

• actix-web example that just logs the received events
• reqwest example with WASM that sends events from an HTML form

example event

{"id":"f8e5ce6e-001e-0041-3e8c-bb82c006f3fc","source":"/subscriptions/5df","specversion":"1.0","type":"Microsoft.Storage.BlobCreated","dataschema":"#","subject":"blob","time":"2021-10-07T15:03:03.0109072Z","data":{}}

Reimplement BinarySerializer and StructuredSerializer for EventBuilder, removing the implementation for Event

Document that explains how to contribute to this project

I would love to look at implementing https://github.com/cloudevents/spec/blob/master/websockets-protocol-binding.md for one of our integrations (maybe actix?)

Although actix-web 4 is only released as beta yet, it is required for the switch to tokio 1.

If/when the actix-web dependency is updated to v3, the dependency in Cargo.toml should ideally be:

- actix-web = "3.0.2"
+ actix-web = { version = "3.0.2", default-features = false }

In order to take advantage of actix-web's Extractors and Responders, cloudevents::Event would have to implement a few actix-web traits, but that's impossible due to rustc --explain E0117: Only traits defined in the current crate can be implemented for arbitrary types.

After chatting with @slinkydeveloper and @ctron, one possible solution would be to combine all of the current web runtimes into a single crate where the user's desired runtime would be conditionally compiled in using a cargo feature.

This issue is to track the feasibility of doing that.

At the moment, to generate library errors, we're using snafu. This library is more powerful than we need, we should investigate https://docs.rs/thiserror as alternative, since it provides only the features we really need

Hello there, brand new to Rust so take this with a pinch of salt.

I have an incredibly simple actix-web setup and as I was testing it I noticed that the ce-source/source header must be passed through as a full absolute URL, including scheme because this is being parsed via the URL module. From what I can tell from the CloudEvents spec, it seems like the source does not need to include the scheme or even the domain (looking at the 'Exemplary JSON Value' for the source attribute).

It appears though that when I pass a relative URL to the to_event function I see the following response:

Error while parsing a url: relative URL without a base

Absolute URL (including scheme) work fine.

In case you need it to replicate, attaching my code here:

use actix_web::{post, web, App, HttpRequest, HttpServer};
use cloudevents::{EventBuilder, EventBuilderV10};
use cloudevents_sdk_actix_web::{HttpResponseBuilderExt, HttpRequestExt};

#[post("/")]
async fn handler(req: HttpRequest, payload: web::Payload) -> Result<String, actix_web::Error> {
    let event = req.to_event(payload).await?;
    println!("Received Event: {:?}", event);
    Ok(format!("{:?}", event))
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| App::new().service(handler))
        .bind("127.0.0.1:8080")?
        .run()
        .await
}

[dependencies]
actix-web = "^3"
cloudevents-sdk = "0.3.0"
cloudevents-sdk-actix-web = "0.3.0"

curl -v "http://127.0.0.1:8080" \
  -X POST \
  -H "Ce-Specversion: 1.0" \
  -H "Ce-Type: dev.knative.samples.helloworld" \
  -H "Ce-Source: /helloworldsource" \
  -H "Ce-Id: 536808d3-88be-4077-9d7a-a3f162705f79" \
   -H "Content-Type: application/json" \
  -d '{"msg":"Hello Knative!"}'

I know its early days with Axum. However from a security point of view (using latest tokio) Axum is the current horse I'm betting on (Comparing Actix-web 4, Rocket 5, warp and the tower ecosystem).

I have an example pull request that can be found here
https://github.com/andrewwebber/cloudevents-axum

We should check if it's doable for the main crate cloudevents-sdk to compile in no_std environments, in order to use this sdk in restricted environments too. An example use case could be: send a cloudevent using mqtt from an iot device

Add blog post about sdk-rust first release and change the nav bar.

EventBuilder now is implemented using Event. This is wrong because event always have default values for the mandatory fields, we should convert EventBuilders to store all attributes as optional (even the mandatory ones) and fail on build() if there are some missing mandatory attributes.

Additional context

Now both v1 and v03 modules has a module called builder which exposes this builder struct:

pub struct EventBuilder {
    event: Event,
}

This builder will always set the default values for Event, never failing when the EventBuilder is not filled with mandatory fields:

pub fn build(self) -> Event {
        self.event
    }

The EventBuilder should in fact look like (below example for v10, but similar should be for v03):

pub struct EventBuilder {
    id: Option<String>,
    ty: Option<String>,
    source: Option<Url>,
    datacontenttype: Option<String>,
    dataschema: Option<Url>,
    subject: Option<String>,
    time: Option<DateTime<Utc>>,
    data: Option<Data>,
    extensions: HashMap<String, ExtensionValue>,
}

And the build signature should be like:

pub fn build(self) -> Result<Event, BuildError>

Where build error explicit the missing mandatory fields

Data structures to represent CloudEvents that can be extended in future to support different cloudevents versions. Some examples:

• https://github.com/cloudevents/sdk-go/blob/master/pkg/cloudevents/event.go#L12
• https://github.com/kichristensen/rust-cloudevents/blob/master/cloudevents/src/event.rs#L8

We also need easy accessors to read cloudevents metadata, extensions and payload. An example to read and parse payload: https://github.com/slinkydeveloper/cloudevents-rust/blob/master/cloudevents-rust/src/event/payload.rs

Data structure could be immutable, so we builders to build them. For builders we can try to use https://docs.rs/crate/derive_builder/0.9.0

I propose to add as submodule encoders for HTTP, abstract as much as possible from HTTP libraries data structures. Some experiments are here https://github.com/slinkydeveloper/cloudevents-rust/tree/master/cloudevents-actix-web

Similar to how go-sdk provides the EventContextConverter feature https://github.com/cloudevents/sdk-go/blob/master/pkg/cloudevents/eventcontext.go#L90

Serde value is blocking the #72, because it doesn't support no_std. Do we really need it? Can we get rid of it?

All the Event::remove_* methods today doesn't return the removed value. We should modify the signature and implementation of those methods to return the owned value removed from the event.

We need to bump tokio to 1.0 and check for something broken