Giter Club home page Giter Club logo

temporal-saga-grpc's Introduction

temporal-saga-grpc

Lines Of Code

./docs/public/diagrams/architecture.svg

This project demonstrates using Temporal to orchestrate a saga (effectively a distributed transaction) that interacts with multiple services and has a robust, edge-case-proof rollback strategy, as well as durable function execution. Temporal abstracts away failures.

Getting started

Step 1: Spin everything up

You can spin everything up with:

make

Note

Under the hood, we use a bunch of tools (which you can read about here), but to run things seamlessly locally, the one tool you will need is pkgx.

Step 2: Observe the workflow

Let's get ready to observe this thing in action!

Step 3: Start a Temporal Workflow

curl -v http://localhost:8081/temporal.v1beta1.TemporalService/CreateOnboardingWorkflow \
  -H "Content-Type: application/json" \
  --data-binary @- <<EOF
  {
    "license": {
      "start": "2023-11-16T12:00:00Z",
      "end": "2024-01-16T12:00:00Z"
    },
    "org": {
      "name": "Org 1"
    },
    "profile": {
      "full_name": "Kevin Chen"
    }
  }
EOF

temporal-saga-grpc's People

Contributors

dependabot[bot] avatar kevinmichaelchen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

anhgeeky

temporal-saga-grpc's Issues

OTel Baggage

Baggage

Baggage refers to contextual information that’s passed between spans.

Imagine you want to have a CustomerId attribute on every span in your trace, which involves multiple services; however, CustomerId is only available in one specific service. To accomplish your goal, you can use OpenTelemetry Baggage to propagate this value across your system.

In OpenTelemetry, “Baggage” refers to contextual information that’s passed between spans. It’s a key-value store that resides within a trace context, making values available to any span created within that trace.

OpenTelemetry uses Context Propagation to pass Baggage around, and each of the different library implementations has propagators that will parse and make that Baggage available without you needing to explicitly implement it.

otel-baggage-1

What should OTel Baggage be used for?

OTel Baggage should be used for non-sensitive data that you’re okay with potentially exposing to third parties.

Common use cases include information that’s only accessible further up a stack. This can include things like Account Identification, User Ids, Product Ids, and origin IPs, for example. Passing these down your stack allows you to then add them to your Spans in downstream services to make it easier to filter when you’re searching in your Observability back-end.

There are no built-in integrity checks to ensure that the Baggage items are yours, so exercise caution when working with Baggage.

Baggage != Span attributes

One important thing to note about Baggage is that it is not a subset of the Span Attributes. When you add something as Baggage, it does not automatically end up on the Attributes of the child system’s spans. You must explicitly take something out of Baggage and append it as Attributes.

Sources

gRPC Reflection

https://github.com/connectrpc/grpcreflect-go

grpcreflect-go adds support for gRPC's server reflection API to any net/http server — including those built with Connect. With server reflection enabled, ad-hoc debugging tools can call your gRPC-compatible handlers and print the responses without a copy of the schema.

The exposed reflection API is wire compatible with Google's gRPC implementations, so it works with grpcurl, grpcui, BloomRPC, and many other tools.

Earthly — Makefile + Dockerfile

We can have an Earthfile alongside each main.go.

Instead of running docker build -t my-image:latest ., we'd have an Earthfile:

FROM DOCKERFILE .
SAVE IMAGE svc-a:latest

We can then run with:

svc-a:
    FROM earthly/dind:alpine
    WITH DOCKER --pull svc-a
        RUN docker run svc-a
    END

How is Earthly different from Dockerfiles?

Dockerfiles were designed for specifying the make-up of Docker images and that's where Dockerfiles stop. Earthly takes some key principles of Dockerfiles (like layer caching), but expands on the use-cases. For example, Earthly can output regular artifacts, run unit and integration tests, and create several Docker images at a time - all outside the scope of Dockerfiles.

It is possible to use Dockerfiles in combination with other technologies (e.g., Makefiles or bash files) to solve such use-cases. However, these combinations are difficult to parallelize, challenging to scale across repositories as they lack a robust import system and also they often vary in style from one team to another. Earthly does not have these limitations as it was designed as a general-purpose build system.

For example, Earthly introduces a richer target, artifact and image referencing system, allowing for better reuse in complex builds spanning a single large repository or multiple repositories. Because Dockerfiles are only meant to describe one image at a time, such features are outside the scope of applicability of Dockerfiles.

GraphQL

It would be nice to turn our Connect APIs into a GraphQL API, but there are some challenges.

API Composition Support

Before we can offer a GraphQL API, we must first build something that most GraphQL federation technologies can interoperate with.

  • Many federation solutions have well-tested support for composing REST APIs;
  • none of them have any idea what Connect RPC is; and
  • very few support composition of gRPC APIs (Wundergraph has it on their roadmap; Tailcall supports it, but it's fairly new and has rough edges with imports; graphql-mesh has it, but it's very JS/YAML heavy).

REST is a prerequisite

Given the maturity of solutions in the existing ecosystem, REST is the clear winner.

Before we can offer GraphQL from Connect, we must first offer REST from Connect.

This is possible with grpc-gateway, but there's a newer officially-supported-by-Buf way via the Vanguard library (see #82).

Tailcall is another winner

Tailcall is capable of federating and composing REST APIs.

REST Support

Support REST, gRPC, gRPC-Web, and Connect clients with one server.

Why Vanguard?

Vanguard offers a range of compelling use cases that make it an invaluable addition
to your services:

  1. RESTful Transformation: By leveraging HTTP transcoding annotations, you can effortlessly support REST clients. This feature is especially handy during the migration from a REST API to a schema-driven RPC API. With the right annotations, your existing REST clients can seamlessly access your API, even as you transition your server implementations to Protobuf and RPC.

  2. Efficiency and Code Generation: Unlike traditional approaches like gRPC-Gateway, Vanguard operates efficiently within Go servers, compatible with various servers such as Connect and gRPC. It doesn't rely on extensive code generation, eliminating the need for additional code generation steps. This flexibility ensures that your code can adapt dynamically, loading service definitions from configuration, schema registries, or via gRPC Server Reflection, making it a perfect fit for proxies without the hassle of recompilation and redeployment each time an RPC service schema changes.

  3. Legacy Compatibility: The HTTP transcoding annotations also empower you to support legacy REST API servers when clients are accustomed to using Protobuf RPC. This lets you embrace RPC in specific teams, such as for web or mobile clients, without the prerequisite of migrating all backend API services.

  4. Seamless Protocol Bridging: If your organization is transitioning from gRPC to Connect, Vanguard acts as a bridge between the protocols. This facilitates the use of your existing gRPC service handlers with Connect clients, allowing you to smoothly adapt to Connect's enhanced usability and inspectability with web browsers and mobile devices. No need to overhaul your server handler logic before migrating clients to Connect.

Caching with ReadySet

https://readyset.io/

Add ReadySet to your Postgres or MySQL database and serve millions of requests per second, at sub-millisecond latencies. All with zero code changes.

Forget TTLs or complicated eviction logic. ReadySet uses your database's replication stream to automatically refresh the cache as updates happen.

Use OTel Collector + Fix Trace Export

2023/11/16 19:31:26 traces export: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tcp/14268/api/traces: nodename nor servname provided, or not known"

Simulated failure

  • Consider allowing failure probabilities (0-100) in the request protos to simulate probabilistic failures.
  • Add simulated latency to request protos to simulate timeout failures.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.