harlow / go-micro-services Goto Github PK

HTTP up front, Protobufs in the rear

License: MIT License

Go 88.66% Makefile 1.55% HTML 8.34% CSS 0.52% Dockerfile 0.92%

golang microservice grpc protobuf go jaeger consul

go-micro-services's Introduction

Golang Microservices Example

A demonstration of Golang micro-services that expose a HTTP/JSON frontend and then leverages gRPC for inter-service communication.

Services written in Golang
gRPC for inter-service communication
Jaeger for request tracing

The example application plots Hotel locations on a Google map:

The web page makes an HTTP request to the API Endpoint which in turn spawns a number of RPC requests to the backend services.

Data for each of the services is stored in JSON flat files under the data/ directory. In reality each of the services could choose their own specialty datastore. The Geo service for example could use PostGis or any other database specializing in geospacial queries.

Installation

Setup

Docker is required for running the services https://docs.docker.com/engine/installation.

Protobuf v3 are required:

$ brew install protobuf

Install the protoc-gen libraries:

$ go get -u github.com/golang/protobuf/{proto,protoc-gen-go}

Clone the repository:

$ git clone [email protected]:harlow/go-micro-services.git

Run

To make the demo as straigforward as possible; Docker Compose is used to run all the services at once (In a production environment each of the services would be run (and scaled) independently).

$ make run

Vist the web page in a browser:

http://localhost:5000/

cURL the API endpoint and receive GeoJSON response:

$ curl "http://localhost:5000/hotels?inDate=2015-04-09&outDate=2015-04-10"

The JSON response:

{
	"type": "FeatureCollection",
	"features": [{
		"id": "5",
		"type": "Feature",
		"properties": {
			"name": "Phoenix Hotel",
			"phone_number": "(415) 776-1380"
		},
		"geometry": {
			"type": "Point",
			"coordinates": [-122.4181, 37.7831]
		}
	}, {
		"id": "3",
		"type": "Feature",
		"properties": {
			"name": "Hotel Zetta",
			"phone_number": "(415) 543-8555"
		},
		"geometry": {
			"type": "Point",
			"coordinates": [-122.4071, 37.7834]
		}
	}]
}

Request Tracing

The Jaeger Tracing project is used for tracing inter-service requests. The tracing package is used initialize a new service tracer:

tracer, err := tracing.Init("serviceName", jaegeraddr)
if err != nil {
    fmt.Fprintf(os.Stderr, "%v\n", err)
    os.Exit(1)
}

View dashboard: http://localhost:16686/search

Protobufs

If changes are made to the Protocol Buffer files use the Makefile to regenerate:

$ make proto

Bindata

The example app data is stored in flat files in the /data directory. When any of the data files are manually editied the bindata must be regenerated.

Install the go-bindata libraries:

$ go get -u github.com/go-bindata/go-bindata/...

If changes are made to any of the raw JSON data files use the Makefile to regenerate:

$ make data

Credits

Thanks to all the contributors. This codebase was heavily inspired by the following talks and repositories:

go-micro-services's People

Contributors

Stargazers

Watchers

Forkers

ltzmaxwell jonahglover isaiah erophames beauhoyt bobbwu s4uliu5 sguzwf mozhonghua ritoon ibmendoza ravikumarmaddi timezstyle houcy indexfromzero jim3ma codesword wesovilabs markoradinovic qgymje dbs6713 wlads pourer zysyyz joeblew99 abhay440 arjun-rao irnes gophersgang gsiegfried tuanquanghpvn windperson dung13890 davidbuenonnoleto zhouchuanwen up1 chusri mhockelberg wlwr cricketue bizsbuzz wkharold luigi-riefolo prolog-opslab huangjialin halokid dwethmar radu-matei wang2467 ecochainadmin hadokencode haotianli89 sittipongwork qiangmzsx alexnimoy cemilcelik jeffersbaxter monjovi ashwingrao dengyingtuo yamaszone bigzun iporsut benhawker thanhniencung tannineo etsangsplk xfyecn hariprasadraja boyone chaitanyaphalak sagarrakshe xfanhsl gy1005 treycheng pyjava1984 infra-modules jiushi506 asatap davitron liuzhl518 forging2012 africhild linecode letubert jordybaylac yuchou rvojjala wacr2008 infigenie hackerrithm codelingobot rbandara pavlelee tullo zenling codezeros-dev satnami myproject-0722 tuongntk

go-micro-services's Issues

Any example for using service discovery (e.g. consul) with this code?

From my understanding, the current design does not use service discovery, instead each service is deployed in a single docker container. I was wondering how to integrate consul into this implementation, and how to run multiple instances for each service.

Thanks

Add README to each of the Services

In general it would be nice to document how to call each of the services and give more details around their implementations.

access microservices

how we access multiple microservices?

Build problems with metadata errors

Hi!
I'm using protobuf-3.0.0-beta1

Here is my build log of go-micro-services:

stalker@zombie ~/workspace/src/github.com/harlow/go-micro-services  > $ make -s
compiled: proto/auth/auth.proto
compiled: proto/geo/geo.proto
compiled: proto/profile/profile.proto
compiled: proto/rate/rate.proto

stalker@zombie ~/workspace/src/github.com/harlow/go-micro-services > $ make build
./build.sh
building cmd/api
building cmd/auth
# github.com/harlow/go-micro-services/cmd/auth
./main.go:37: cannot use md["traceID"] (type []string) as type string in field value
building cmd/geo
# github.com/harlow/go-micro-services/cmd/geo
./main.go:41: cannot use md["traceID"] (type []string) as type string in field value
building cmd/profile
# github.com/harlow/go-micro-services/cmd/profile
./main.go:35: cannot use md["traceID"] (type []string) as type string in field value
building cmd/rate
# github.com/harlow/go-micro-services/cmd/rate
./main.go:41: cannot use md["traceID"] (type []string) as type string in field value

Here is the code of cmd/auth.go:

33 // VerifyToken finds a customer by authentication token.
34 func (s *authServer) VerifyToken(ctx context.Context, args *auth.Args) (*auth.Customer, error) {
35         md, _ := metadata.FromContext(ctx)
36 
37         t := trace.Tracer{TraceID: md["traceID"]}
38         t.In(s.serverName, md["fromName"])
39         defer t.Out(md["fromName"], s.serverName, time.Now())
40 
41         customer := s.customers[args.AuthToken]
42         if customer == nil {
43                 return &auth.Customer{}, errors.New("Invalid Token")
44         }
45 
46         return customer, nil
47 }

The metadata["traceID"], metadata["fromName"] and so on...
obviously the []string array, so, it should be fixed?
Thanx, Alex!

codahale/hdrhistogram repo url has been transferred under the github HdrHstogram umbrella

Problem

The codahale/hdrhistogram repo has been transferred under the github HdrHstogram umbrella with the help from the original author in Sept 2020 (new repo url https://github.com/HdrHistogram/hdrhistogram-go). The main reasons are to group all implementations under the same roof and to provide more active contribution from the community as the original repository was archived several years ago.

The dependency URL should be modified to point to the new repository URL. The tag "v0.9.0" was applied at the point of transfer and will reflect the exact code that was frozen in the original repository.

If you are using Go modules, you can update to the exact point of transfer using the @v0.9.0 tag in your go get command.

go mod edit -replace github.com/codahale/hdrhistogram=github.com/HdrHistogram/[email protected]

Performance Improvements

From the point of transfer, up until now (mon 16 aug 2021), we've released 3 versions that aim support the standard HdrHistogram serialization/exposition formats, and deeply improve READ performance.
We recommend to update to the latest version.

Failed to build due to GRPC Protobuf

I know this issue may relate to my own setting GRPC protobuf installation. However, i just can't figure it out. Since I have followed the manual instruction in README.md. I also did protoc --version which gave me output libprotoc 3.1.0. However, when I did make build. It gave me the following errors:

⇒  protoc --version
libprotoc 3.1.0
⇒  make pb
for f in pb/**/*.proto; do \
		protoc --go_out=plugins=grpc:. $f; \
		echo compiled: $f; \
	done
compiled: pb/auth/auth.proto
compiled: pb/geo/geo.proto
compiled: pb/profile/profile.proto
compiled: pb/rate/rate.proto
⇒  make build
./bin/build.sh
building cmd/api
# github.com/harlow/go-micro-services/pb/auth
../../../../golang/src/github.com/harlow/go-micro-services/pb/auth/auth.pb.go:128: cannot use _Auth_VerifyToken_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
# github.com/harlow/go-micro-services/pb/rate
../../../../golang/src/github.com/harlow/go-micro-services/pb/rate/rate.pb.go:158: cannot use _Rate_GetRates_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
# github.com/harlow/go-micro-services/pb/profile
../../../../golang/src/github.com/harlow/go-micro-services/pb/profile/profile.pb.go:175: cannot use _Profile_GetProfiles_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
# github.com/harlow/go-micro-services/pb/geo
../../../../golang/src/github.com/harlow/go-micro-services/pb/geo/geo.pb.go:113: cannot use _Geo_Nearby_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
building cmd/auth
# github.com/harlow/go-micro-services/pb/auth
../../../../golang/src/github.com/harlow/go-micro-services/pb/auth/auth.pb.go:128: cannot use _Auth_VerifyToken_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
building cmd/geo
# github.com/harlow/go-micro-services/pb/geo
../../../../golang/src/github.com/harlow/go-micro-services/pb/geo/geo.pb.go:113: cannot use _Geo_Nearby_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
building cmd/profile
# github.com/harlow/go-micro-services/pb/profile
../../../../golang/src/github.com/harlow/go-micro-services/pb/profile/profile.pb.go:175: cannot use _Profile_GetProfiles_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
building cmd/rate
# github.com/harlow/go-micro-services/pb/rate
../../../../golang/src/github.com/harlow/go-micro-services/pb/rate/rate.pb.go:158: cannot use _Rate_GetRates_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value

Can anyone help me to get this right? Or if you have any prebuild docker in Docker Hub/ Marketplace that'd help me a lot.

Thank you!

Some extension ideas :)

thanks for this example - really easy to follow along.

I am wanting to put a Message queue within this architecture and wanted to tun it by you and well the whole internet :)
I am still trying to work out the best way, without introducing too much boiler plate into it.

Why do this ?
Because then a GRPC Service can also call other GRPC services as and when they need to.
This allows building larger systems of Microservices where the Choreography is emergent. Bottom up rather than top down. Essentially its the reflux pattern being used for Microservices. Nothing ground breaking but an important principle i feel.

Here is what i intend to do:
The API code where the REST / JSON layer is exposed would put JSON messages onto a Message queue topic.
Then the GRPC Service would listen for them on the right topic.
Nothing really to it.

a Coupe of design choices thogh :)

GRPC-gateway
Why not just use the GRPC-gateway to expose the GRPC services as Resftul JSON services ? The pro is that you get code gen for free. Your API code is much less boiler plate and so much less brittle.
I think also you get websockets for free based on the latest code there.

NATS or NSQ ?
Apart from HA style semantics, the thing that is also important is how easily it can handle websocket. The reason being because then a Microservice Or the APi layer can also push to the client (s). Dont knwo about you, but for me, its an important this.
I like to design (and think about the client) as just being another service behind a message queue.
SO the centre of everything is your API layer where everything comes in and out of.
There are not many examples of using NQS or NATS with web sockets from what i can find.

Would be cool if you have any thoughts on this.
I am happy to make a PR for this and contrib it back to this repo too. There are not enough examples of this around i feel.

Add back in Trace Visualization

During the last PR we had a bunch of merge conflicts and I intentionally left some code out of master to make it easier.

Add back the visualization: 7904945

is there a plan to support k8s

hello,

This project is Very helpful, is there a plan to support k8s ?

Add tracing service for generating UML sequence diagrams

This will keep the logs cleaner and give a query interface for request sequences

Service API error?

Hi!
Below the details:

stalker@zombie ~/workspace/src/github.com/harlow/go-micro-services > $ sudo docker-compose ps
Name                Command      State     Ports   
------------------------------------------------------------
gomicroservices_api_1       /app/api       Exit 1            
gomicroservices_auth_1      /app/auth      Up       8080/tcp 
gomicroservices_geo_1       /app/geo       Up       8080/tcp 
gomicroservices_profile_1   /app/profile   Up       8080/tcp 
gomicroservices_rate_1      /app/rate      Up       8080/tcp 

stalker@zombie ~/workspace/src/github.com/harlow/go-micro-services > $ sudo docker-compose logs
Attaching to gomicroservices_api_1, gomicroservices_auth_1, gomicroservices_geo_1, gomicroservices_rate_1, gomicroservices_profile_1
api_1     | 2015/11/18 11:48:31 dial failed: grpc: no transport security set (use grpc.WithInsecure() explicitly or set credentials)
api_1     | 2015/11/18 11:48:48 dial failed: grpc: no transport security set (use grpc.WithInsecure() explicitly or set credentials)

I think in API service (main.go) should present grpc.WithInsecure() call?

38 // mustDial ensures the tcp connection to specified address.
39 func mustDial(addr *string) *grpc.ClientConn {
40         var opts []grpc.DialOption
41 
42         opts = append(opts, grpc.WithInsecure())
43 
44         conn, err := grpc.Dial(*addr, opts...)
45         if err != nil {
46                 log.Fatalf("dial failed: %s", err)
47                 panic(err)
48         }
49         return conn
50 }

Thanx!

grpc dialer issue

Hi,
I'm running a DeathStarBench application built on top of this project. Or, at least I'm trying to. I have been trying to run all of this with Docker Compose in Ubuntu, but I keep hitting this dependency error that seems like more of an issue with this project than with DeathStarBench and I don't know what I'd like to do. I'll also mention this is part of PhD research I'm doing at Tufts University. Anyway, here's the error:

go: finding module for package google.golang.org/grpc/naming
go: downloading google.golang.org/grpc v1.46.0
github.com/harlow/go-micro-services/dialer imports github.com/olivere/grpc/lb/consul imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.46.0), but does not contain package google.golang.org/grpc/naming
The command '/bin/sh -c go mod vendor' returned a non-zero code: 1
ERROR: Service 'frontend' failed to build : Build failed

Optimize the build process w/ Docker compose

It feels a little strange having the cmd directories each having their own vendor directory. However when I try to bring the vendor directory down to root I get an error when trying to build multiple applications at once.

Add lightstep instrumentation to project

https://github.com/harlow/go-micro-services/blob/master/cmd/geo/main.go#L50

.env.sample is missing

Great job on this super useful example!

The README mentions the following in the Run step:

$ cp .env.sample .env

But the .env.sample file is missing from the repository.

Microservices fail to start without Stackdriver variables in .env file.

I tried getting this project running from the README. Ran into two problems:

1: Had to go get go-bindata

go get -u github.com/jteeuwen/go-bindata/...

2: I then got the following error on make run:
Couldn't find env file

3: I added a blank .env file. I don't want to play with Tracing using Google Stackdriver at this time.

❯ make run
...

docker-compose up
Creating gomicroservices_www_1
Creating gomicroservices_geo_1
Creating gomicroservices_profile_1
Creating gomicroservices_rate_1
Creating gomicroservices_api_1
Attaching to gomicroservices_profile_1, gomicroservices_www_1, gomicroservices_geo_1, gomicroservices_rate_1, gomicroservices_api_1
profile_1  | 2017/03/15 22:05:34 google config from json error: unexpected end of JSON input
www_1      | + exec app
geo_1      | 2017/03/15 22:05:34 google config from json error: unexpected end of JSON input
rate_1     | 2017/03/15 22:05:34 google config from json error: unexpected end of JSON input
gomicroservices_profile_1 exited with code 1
gomicroservices_geo_1 exited with code 1
api_1      | 2017/03/15 22:05:36 google config from json error: unexpected end of JSON input
gomicroservices_rate_1 exited with code 1
gomicroservices_api_1 exited with code 1

So it appears that this project won't run without setting up Google Stackdriver?

Question about the GRPC protoc version

Hi, I wanna convert the .proto file to the .pb.go file, but use the latest protoc version of 3.6.1, but generate a totally different .pb.go file with your original .proto file.

Alternative to Google Stackdriver

Hello,

I was following the readme and tried to get a trial account for Google Stackdriver, sadly they restricted the service to companies only here in Europe, I currently don't have one. Could it be possible to offer/suggest an alternatives ?

Thank you very much :)

error in generated protobuf go code?

Hello @harlow ! I'm new to go, trying to get this demo up and running. I'm getting the following error when I run foreman start.

go-micro-services (master) $ go version
go version go1.4.2 darwin/amd64

go-micro-services (master) $ foreman start
23:36:57 web.1     | started with pid 76450
23:36:57 auth.1    | started with pid 76451
23:36:57 geo.1     | started with pid 76452
23:36:57 profile.1 | started with pid 76453
23:36:57 rate.1    | started with pid 76454
23:36:58 web.1     | # go-micro-services/service.auth/proto
23:36:58 geo.1     | # go-micro-services/service.geo/proto
23:36:58 profile.1 | # go-micro-services/service.profile/proto
23:36:58 rate.1    | # go-micro-services/service.rate/proto
23:36:58 web.1     | service.auth/proto/auth.pb.go:104: cannot use _Auth_VerifyToken_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 geo.1     | service.geo/proto/geo.pb.go:152: cannot use _Geo_BoundedBox_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 profile.1 | service.profile/proto/profile.pb.go:163: cannot use _Profile_GetProfiles_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 rate.1    | service.rate/proto/rate.pb.go:148: cannot use _Rate_GetRates_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 web.1     | # go-micro-services/service.geo/proto
23:36:58 auth.1    | # go-micro-services/service.auth/proto
23:36:58 web.1     | service.geo/proto/geo.pb.go:152: cannot use _Geo_BoundedBox_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 auth.1    | service.auth/proto/auth.pb.go:104: cannot use _Auth_VerifyToken_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 web.1     | # go-micro-services/service.profile/proto
23:36:58 web.1     | service.profile/proto/profile.pb.go:163: cannot use _Profile_GetProfiles_Handler (type func(interface {}, context.Context, []byte) (interface {}, error)) as type grpc.methodHandler in field value
23:36:58 rate.1    | exited with code 2
23:36:58 system    | sending SIGTERM to all processes
23:36:58 profile.1 | exited with code 2
SIGTERM received
23:36:58 auth.1    | exited with code 2
23:36:58 geo.1     | exited with code 2
23:36:58 web.1     | exited with code 2

Looks like a type error from the files generated by protobuf-gen-go, which I installed using go get -u github.com/golang/protobuf/{proto,protoc-gen-go}. Any ideas?

Question: what if you had to use a relational database

I was wondering how would you structure the code if you had to work with data stored in a relational database?
Would you use a single database? One database for each service? Any thoughts on that? Do you know any real world scenario and how they actually store data when using micro services?

Cheers!