harlow / go-micro-services Goto Github PK

HTTP up front, Protobufs in the rear

License: MIT License

Go 88.66% Makefile 1.55% HTML 8.34% CSS 0.52% Dockerfile 0.92%

golang microservice grpc protobuf go jaeger consul

go-micro-services's Issues

Any example for using service discovery (e.g. consul) with this code?

From my understanding, the current design does not use service discovery, instead each service is deployed in a single docker container. I was wondering how to integrate consul into this implementation, and how to run multiple instances for each service.

Thanks

Some extension ideas :)

thanks for this example - really easy to follow along.

I am wanting to put a Message queue within this architecture and wanted to tun it by you and well the whole internet :)
I am still trying to work out the best way, without introducing too much boiler plate into it.

Why do this ?
Because then a GRPC Service can also call other GRPC services as and when they need to.
This allows building larger systems of Microservices where the Choreography is emergent. Bottom up rather than top down. Essentially its the reflux pattern being used for Microservices. Nothing ground breaking but an important principle i feel.

Here is what i intend to do:
The API code where the REST / JSON layer is exposed would put JSON messages onto a Message queue topic.
Then the GRPC Service would listen for them on the right topic.
Nothing really to it.

a Coupe of design choices thogh :)

GRPC-gateway
Why not just use the GRPC-gateway to expose the GRPC services as Resftul JSON services ? The pro is that you get code gen for free. Your API code is much less boiler plate and so much less brittle.
I think also you get websockets for free based on the latest code there.

NATS or NSQ ?
Apart from HA style semantics, the thing that is also important is how easily it can handle websocket. The reason being because then a Microservice Or the APi layer can also push to the client (s). Dont knwo about you, but for me, its an important this.
I like to design (and think about the client) as just being another service behind a message queue.
SO the centre of everything is your API layer where everything comes in and out of.
There are not many examples of using NQS or NATS with web sockets from what i can find.

Would be cool if you have any thoughts on this.
I am happy to make a PR for this and contrib it back to this repo too. There are not enough examples of this around i feel.

Question: what if you had to use a relational database

I was wondering how would you structure the code if you had to work with data stored in a relational database?
Would you use a single database? One database for each service? Any thoughts on that? Do you know any real world scenario and how they actually store data when using micro services?

Cheers!

Question about the GRPC protoc version

Hi, I wanna convert the .proto file to the .pb.go file, but use the latest protoc version of 3.6.1, but generate a totally different .pb.go file with your original .proto file.

Add lightstep instrumentation to project

https://github.com/harlow/go-micro-services/blob/master/cmd/geo/main.go#L50

Add README to each of the Services

In general it would be nice to document how to call each of the services and give more details around their implementations.

Are there any benefits to use consul instead of Kubernetes for service discovery?

I wonder whether there are any benefits to use consul instead of Kubernetes services for service discovery?

Microservices fail to start without Stackdriver variables in .env file.

I tried getting this project running from the README. Ran into two problems:

1: Had to go get go-bindata

go get -u github.com/jteeuwen/go-bindata/...

2: I then got the following error on make run:
Couldn't find env file

3: I added a blank .env file. I don't want to play with Tracing using Google Stackdriver at this time.

❯ make run
...

docker-compose up
Creating gomicroservices_www_1
Creating gomicroservices_geo_1
Creating gomicroservices_profile_1
Creating gomicroservices_rate_1
Creating gomicroservices_api_1
Attaching to gomicroservices_profile_1, gomicroservices_www_1, gomicroservices_geo_1, gomicroservices_rate_1, gomicroservices_api_1
profile_1  | 2017/03/15 22:05:34 google config from json error: unexpected end of JSON input
www_1      | + exec app
geo_1      | 2017/03/15 22:05:34 google config from json error: unexpected end of JSON input
rate_1     | 2017/03/15 22:05:34 google config from json error: unexpected end of JSON input
gomicroservices_profile_1 exited with code 1
gomicroservices_geo_1 exited with code 1
api_1      | 2017/03/15 22:05:36 google config from json error: unexpected end of JSON input
gomicroservices_rate_1 exited with code 1
gomicroservices_api_1 exited with code 1

So it appears that this project won't run without setting up Google Stackdriver?

Service API error?

Hi!
Below the details:

stalker@zombie ~/workspace/src/github.com/harlow/go-micro-services > $ sudo docker-compose ps
Name                Command      State     Ports   
------------------------------------------------------------
gomicroservices_api_1       /app/api       Exit 1            
gomicroservices_auth_1      /app/auth      Up       8080/tcp 
gomicroservices_geo_1       /app/geo       Up       8080/tcp 
gomicroservices_profile_1   /app/profile   Up       8080/tcp 
gomicroservices_rate_1      /app/rate      Up       8080/tcp 

stalker@zombie ~/workspace/src/github.com/harlow/go-micro-services > $ sudo docker-compose logs
Attaching to gomicroservices_api_1, gomicroservices_auth_1, gomicroservices_geo_1, gomicroservices_rate_1, gomicroservices_profile_1
api_1     | 2015/11/18 11:48:31 dial failed: grpc: no transport security set (use grpc.WithInsecure() explicitly or set credentials)
api_1     | 2015/11/18 11:48:48 dial failed: grpc: no transport security set (use grpc.WithInsecure() explicitly or set credentials)

I think in API service (main.go) should present grpc.WithInsecure() call?

38 // mustDial ensures the tcp connection to specified address.
39 func mustDial(addr *string) *grpc.ClientConn {
40         var opts []grpc.DialOption
41 
42         opts = append(opts, grpc.WithInsecure())
43 
44         conn, err := grpc.Dial(*addr, opts...)
45         if err != nil {
46                 log.Fatalf("dial failed: %s", err)
47                 panic(err)
48         }
49         return conn
50 }

Thanx!

Implement client-side loadbalancing

Topical reading:
https://grpc.io/blog/loadbalancing
https://github.com/grpc/grpc/blob/master/doc/load-balancing.md

Library that implements the client side balancing:
https://github.com/bsm/grpclb

Refactor loop in code path

Think it will also be worth re-visiting this line and see what other alternatives there are:
https://github.com/harlow/go-micro-services/blob/master/cmd/api/main.go#L83

Use database instead of data binding from JSON

Hi @harlow, I want to contribute this project with use Database instead of binding data from JSON files. Is this make senses?

Add back in Trace Visualization

During the last PR we had a bunch of merge conflicts and I intentionally left some code out of master to make it easier.

Add back the visualization: 7904945

grpc dialer issue

Hi,
I'm running a DeathStarBench application built on top of this project. Or, at least I'm trying to. I have been trying to run all of this with Docker Compose in Ubuntu, but I keep hitting this dependency error that seems like more of an issue with this project than with DeathStarBench and I don't know what I'd like to do. I'll also mention this is part of PhD research I'm doing at Tufts University. Anyway, here's the error:

go: finding module for package google.golang.org/grpc/naming
go: downloading google.golang.org/grpc v1.46.0
github.com/harlow/go-micro-services/dialer imports github.com/olivere/grpc/lb/consul imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.46.0), but does not contain package google.golang.org/grpc/naming
The command '/bin/sh -c go mod vendor' returned a non-zero code: 1
ERROR: Service 'frontend' failed to build : Build failed

error in generated protobuf go code?

Hello @harlow ! I'm new to go, trying to get this demo up and running. I'm getting the following error when I run foreman start.

go-micro-services (master) $ go version
go version go1.4.2 darwin/amd64

go-micro-services (master) $ foreman start
23:36:57 web.1     | started with pid 76450
23:36:57 auth.1    | started with pid 76451
23:36:57 geo.1     | started with pid 76452
23:36:57 profile.1 | started with pid 76453
23:36:57 rate.1    | started with pid 76454
23:36:58 web.1     | # go-micro-services/service.auth/proto
23:36:58 geo.1     | # go-micro-services/service.geo/proto
23:36:58 profile.1 | # go-micro-services/service.profile/proto
23:36:58 rate.1    | # go-micro-services/service.rate/proto
23:36:58 web.1     | service.auth/proto/auth.pb.go:104: cannot use _Auth_VerifyToken_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 geo.1     | service.geo/proto/geo.pb.go:152: cannot use _Geo_BoundedBox_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 profile.1 | service.profile/proto/profile.pb.go:163: cannot use _Profile_GetProfiles_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 rate.1    | service.rate/proto/rate.pb.go:148: cannot use _Rate_GetRates_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 web.1     | # go-micro-services/service.geo/proto
23:36:58 auth.1    | # go-micro-services/service.auth/proto
23:36:58 web.1     | service.geo/proto/geo.pb.go:152: cannot use _Geo_BoundedBox_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 auth.1    | service.auth/proto/auth.pb.go:104: cannot use _Auth_VerifyToken_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 web.1     | # go-micro-services/service.profile/proto
23:36:58 web.1     | service.profile/proto/profile.pb.go:163: cannot use _Profile_GetProfiles_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 rate.1    | exited with code 2
23:36:58 system    | sending SIGTERM to all processes
23:36:58 profile.1 | exited with code 2
SIGTERM received
23:36:58 auth.1    | exited with code 2
23:36:58 geo.1     | exited with code 2
23:36:58 web.1     | exited with code 2

Looks like a type error from the files generated by protobuf-gen-go, which I installed using go get -u github.com/golang/protobuf/{proto,protoc-gen-go}. Any ideas?

Add tracing service for generating UML sequence diagrams

This will keep the logs cleaner and give a query interface for request sequences

.env.sample is missing

Great job on this super useful example!

The README mentions the following in the Run step:

$ cp .env.sample .env

But the .env.sample file is missing from the repository.

Failed to build due to GRPC Protobuf

I know this issue may relate to my own setting GRPC protobuf installation. However, i just can't figure it out. Since I have followed the manual instruction in README.md. I also did protoc --version which gave me output libprotoc 3.1.0. However, when I did make build. It gave me the following errors:

⇒  protoc --version
libprotoc 3.1.0
⇒  make pb
for f in pb/**/*.proto; do \
		protoc --go_out=plugins=grpc:. $f; \
		echo compiled: $f; \
	done
compiled: pb/auth/auth.proto
compiled: pb/geo/geo.proto
compiled: pb/profile/profile.proto
compiled: pb/rate/rate.proto
⇒  make build
./bin/build.sh
building cmd/api
# github.com/harlow/go-micro-services/pb/auth
../../../../golang/src/github.com/harlow/go-micro-services/pb/auth/auth.pb.go:128: cannot use _Auth_VerifyToken_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
# github.com/harlow/go-micro-services/pb/rate
../../../../golang/src/github.com/harlow/go-micro-services/pb/rate/rate.pb.go:158: cannot use _Rate_GetRates_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
# github.com/harlow/go-micro-services/pb/profile
../../../../golang/src/github.com/harlow/go-micro-services/pb/profile/profile.pb.go:175: cannot use _Profile_GetProfiles_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
# github.com/harlow/go-micro-services/pb/geo
../../../../golang/src/github.com/harlow/go-micro-services/pb/geo/geo.pb.go:113: cannot use _Geo_Nearby_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
building cmd/auth
# github.com/harlow/go-micro-services/pb/auth
../../../../golang/src/github.com/harlow/go-micro-services/pb/auth/auth.pb.go:128: cannot use _Auth_VerifyToken_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
building cmd/geo
# github.com/harlow/go-micro-services/pb/geo
../../../../golang/src/github.com/harlow/go-micro-services/pb/geo/geo.pb.go:113: cannot use _Geo_Nearby_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
building cmd/profile
# github.com/harlow/go-micro-services/pb/profile
../../../../golang/src/github.com/harlow/go-micro-services/pb/profile/profile.pb.go:175: cannot use _Profile_GetProfiles_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
building cmd/rate
# github.com/harlow/go-micro-services/pb/rate
../../../../golang/src/github.com/harlow/go-micro-services/pb/rate/rate.pb.go:158: cannot use _Rate_GetRates_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value

Can anyone help me to get this right? Or if you have any prebuild docker in Docker Hub/ Marketplace that'd help me a lot.

Thank you!

access microservices

how we access multiple microservices?

Alternative to Google Stackdriver

Hello,

I was following the readme and tried to get a trial account for Google Stackdriver, sadly they restricted the service to companies only here in Europe, I currently don't have one. Could it be possible to offer/suggest an alternatives ?

Thank you very much :)

is there a plan to support k8s

hello,

This project is Very helpful, is there a plan to support k8s ?

codahale/hdrhistogram repo url has been transferred under the github HdrHstogram umbrella

Problem

The codahale/hdrhistogram repo has been transferred under the github HdrHstogram umbrella with the help from the original author in Sept 2020 (new repo url https://github.com/HdrHistogram/hdrhistogram-go). The main reasons are to group all implementations under the same roof and to provide more active contribution from the community as the original repository was archived several years ago.

The dependency URL should be modified to point to the new repository URL. The tag "v0.9.0" was applied at the point of transfer and will reflect the exact code that was frozen in the original repository.

If you are using Go modules, you can update to the exact point of transfer using the @v0.9.0 tag in your go get command.

go mod edit -replace github.com/codahale/hdrhistogram=github.com/HdrHistogram/[email protected]

Performance Improvements

From the point of transfer, up until now (mon 16 aug 2021), we've released 3 versions that aim support the standard HdrHistogram serialization/exposition formats, and deeply improve READ performance.
We recommend to update to the latest version.

Optimize the build process w/ Docker compose

It feels a little strange having the cmd directories each having their own vendor directory. However when I try to bring the vendor directory down to root I get an error when trying to build multiple applications at once.

Build problems with metadata errors

Hi!
I'm using protobuf-3.0.0-beta1

Here is my build log of go-micro-services:

stalker@zombie ~/workspace/src/github.com/harlow/go-micro-services  > $ make -s
compiled: proto/auth/auth.proto
compiled: proto/geo/geo.proto
compiled: proto/profile/profile.proto
compiled: proto/rate/rate.proto

stalker@zombie ~/workspace/src/github.com/harlow/go-micro-services > $ make build
./build.sh
building cmd/api
building cmd/auth
# github.com/harlow/go-micro-services/cmd/auth
./main.go:37: cannot use md["traceID"] (type []string) as type string in field value
building cmd/geo
# github.com/harlow/go-micro-services/cmd/geo
./main.go:41: cannot use md["traceID"] (type []string) as type string in field value
building cmd/profile
# github.com/harlow/go-micro-services/cmd/profile
./main.go:35: cannot use md["traceID"] (type []string) as type string in field value
building cmd/rate
# github.com/harlow/go-micro-services/cmd/rate
./main.go:41: cannot use md["traceID"] (type []string) as type string in field value

Here is the code of cmd/auth.go:

33 // VerifyToken finds a customer by authentication token.
34 func (s *authServer) VerifyToken(ctx context.Context, args *auth.Args) (*auth.Customer, error) {
35         md, _ := metadata.FromContext(ctx)
36 
37         t := trace.Tracer{TraceID: md["traceID"]}
38         t.In(s.serverName, md["fromName"])
39         defer t.Out(md["fromName"], s.serverName, time.Now())
40 
41         customer := s.customers[args.AuthToken]
42         if customer == nil {
43                 return &auth.Customer{}, errors.New("Invalid Token")
44         }
45 
46         return customer, nil
47 }

The metadata["traceID"], metadata["fromName"] and so on...
obviously the []string array, so, it should be fixed?
Thanx, Alex!

harlow / go-micro-services Goto Github PK

go-micro-services's Issues

Problem

Performance Improvements

Recommend Projects

Recommend Topics

Recommend Org