petabridge / akkadotnet-cluster-workshop Goto Github PK

I posted this in case anyone has the same problem I had with the Akka.CQRS.Pricing.Web container constantly restarting.

To resolve it, I added "app.UseRouting(); "in the Startup.cs file before app.UseEndpoints.

The Dockerfile in the dev branch has changed, the Lesson 2 Dockerfile source listing should now read:

FROM microsoft/dotnet:2.1-sdk AS base
WORKDIR /app

# should be a comma-delimited list
ENV CLUSTER_SEEDS "[]"
ENV CLUSTER_IP ""
ENV CLUSTER_PORT "5054"

# 9110 - Petabridge.Cmd
# 5055 - Akka.Cluster
EXPOSE 9110 5054

# Install Petabridge.Cmd client
RUN dotnet tool install --global pbm 

COPY ./bin/Release/netcoreapp2.1/publish/ /app

FROM microsoft/dotnet:2.1-runtime AS app
WORKDIR /app

COPY --from=base /app /app

# copy .NET Core global tool
COPY --from=base /root/.dotnet /root/.dotnet/

# Needed because https://stackoverflow.com/questions/51977474/install-dotnet-core-tool-dockerfile
ENV PATH="${PATH}:/root/.dotnet/tools"

# RUN pbm help

CMD ["dotnet", "Akka.CQRS.TradePlacers.Service.dll"]

Hi

I am trying to complete lesson 2 of the tutorial and I have read and re-read the tutorial to see what I am doing wrong but cannot spot the mistake.

Everytime I bring up the service stack and look at the logs for the tradeprocessor service I see the following error message

src-tradeprocessor-1 | [ERROR][10/05/2022 07:36:42][Thread 0042][akka.tcp://AkkaTrader@9951bfd18690:5110/user/orderbooks/APPL] Persistence failure when replaying events for persistenceId [APPL-orderBook]. Last known sequence number [0]src-tradeprocessor-1 | Cause: Akka.Pattern.OpenCircuitException: Circuit Breaker is open; calls are failing fast
src-tradeprocessor-1 | ---> System.ArgumentNullException: Value cannot be null. (Parameter 'name')
src-tradeprocessor-1 | at MongoDB.Driver.Core.Misc.Ensure.IsNotNullOrEmpty(String value, String paramName)
src-tradeprocessor-1 | at MongoDB.Driver.MongoDatabaseImpl.GetCollection[TDocument](String name, MongoCollectionSettings settings)
src-tradeprocessor-1 | at Akka.Persistence.MongoDb.Journal.MongoDbJournal.b__12_2()
src-tradeprocessor-1 | at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode) src-tradeprocessor-1 | --- End of stack trace from previous location where exception was thrown --- src-tradeprocessor-1 | at System.Lazy1.CreateValue()
src-tradeprocessor-1 | at Akka.Persistence.MongoDb.Journal.MongoDbJournal.ReadHighestSequenceNrAsync(String persistenceId, Int64 fromSequenceNr)
src-tradeprocessor-1 | at Akka.Util.Internal.AtomicState.CallThrough[T](Func1 task) src-tradeprocessor-1 | at Akka.Util.Internal.AtomicState.CallThrough[T](Func1 task)
src-tradeprocessor-1 | at Akka.Pattern.HalfOpen.Invoke[T](Func`1 body)
src-tradeprocessor-1 | at Akka.Persistence.Journal.AsyncWriteJournal.<>c__DisplayClass19_0.<g__ExecuteHighestSequenceNr|0>d.MoveNext()
src-tradeprocessor-1 | --- End of inner exception stack trace ---

In the traders service I have following

src-traders-1 | [ERROR][10/05/2022 07:41:17][Thread 0069][akka.tcp://AkkaTrader@e4100dc4a6f7:5110/user/$rc] Error while waiting for SubscribeAck for [SAVE-Fill] - retrying in 5s.
src-traders-1 | Cause: Akka.Actor.AskTimeoutException: Timeout after 00:00:03 seconds
src-traders-1 | at Akka.CQRS.Subscriptions.Actor.ActorTradeSubscriptionManager.Subscribe(String tickerSymbol, TradeEventType[] events, IActorRef subscriber) in C:\training\akka.net\cluster\src\Akka.CQRS.Subscriptions\Actor\ActorTradeSubscriptionManager.cs:line 22

Any help would be much appreciated

This lab will be upgraded to AKKA.NET 1.5 ???

#892 need to be backported to all the lesson branches

While attempting to run ./build.cmd docker the script fails when attempting to build the Akka.CQRS.Pricing.Service project and BuildDockerImages reports failure.

Running build failed.
Error:
System.Exception: docker build failed. D:\Dev\akkadotnet-cluster-workshop\src\Akka.CQRS.Pricing.Service\Akka.CQRS.Pricing.Service.csproj
at Microsoft.FSharp.Core.Operators.FailWith[T](String message)
at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc2 action, IEnumerable1 source)
at [email protected](Unit _arg1)
at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in C:\code\fake\src\app\FakeLib\TargetHelper.fs:line 626

I cloned the repo on my mac and tried to "dotnet restore" but the restore failed restoring Akka.Cluster.Sharding of Akka.CQRS.Infrastructure project.

I removed the -beta* part of the declaration hoping this will solve my problem. But was not possible to restore the package again.

Next I removed the package and run "dotnet add package Akka.Cluster.Sharding --version=1.4.9" command to add this package but even this command failed with the message below:

>dotnet add package Akka.Cluster.Sharding --version=1.4.9
  Determining projects to restore...
  Writing /var/folders/82/s0hwd2h14qb8b04pw98zq2ch0000gn/T/tmpXP9u1I.tmp
info : Adding PackageReference for package 'Akka.Cluster.Sharding' into project '/Users/pyrros/Akka-samples/akkadotnet-cluster-workshop/src/Akka.CQRS.Infrastructure/Akka.CQRS.Infrastructure.csproj'.
info : Restoring packages for /Users/pyrros/Akka-samples/akkadotnet-cluster-workshop/src/Akka.CQRS.Infrastructure/Akka.CQRS.Infrastructure.csproj...
info :   GET https://api.nuget.org/v3-flatcontainer/akka.cluster.sharding/index.json
info :   OK https://api.nuget.org/v3-flatcontainer/akka.cluster.sharding/index.json 524ms
error: Unable to find package Akka.Cluster.Sharding with version (>= 1.4.9)
error:   - Found 57 version(s) in nuget.org [ Nearest version: 1.4.0-beta3 ]
error: Package 'Akka.Cluster.Sharding' is incompatible with 'all' frameworks in project '/Users/pyrros/Akka-samples/akkadotnet-cluster-workshop/src/Akka.CQRS.Infrastructure/Akka.CQRS.Infrastructure.csproj'.

Before the release of Akka.Cluster.Sharding --version=1.4.9 I was able to run the entire project on my mac inside K8s too (I walked through all the lessons successfully).

Is there a example of using Generic Host Builder in .NET Core 3.1? Dotnet does not seem to be listening to the same port as the cluster.

The docker manager in the lessons should be updated to use Docker Desktop Dashboard.

There's a discrepancy between the lesson and the git result of lesson 4 in Akka.CQRS.Pricing.Service app.conf.

Discrepancy 1

Lesson 4 asks the student to add these lines into the HOCON config:

	cluster {
		#will inject this node as a self-seed node at run-time
		seed-nodes = ["akka.tcp://[email protected]:6055"] 
		roles = ["pricing-engine" , "trade-events"]
		pub-sub{
			role = "trade-events"
		}
		sharding{
			role = "pricing-engine"
			state-store-mode = ddata
		}
		client.receptionist.role = pricing-engine # stops ClusterClient gossip from going to the worker nodes
	}

but the app.conf in the git repo for lesson 5 contains these:

	cluster {
		#will inject this node as a self-seed node at run-time
		seed-nodes = ["akka.tcp://[email protected]:6055"] 
		roles = ["pricing-engine" , "trade-events"]
		pub-sub{
			role = "trade-events"
		}
		sharding{
			role = "pricing-engine"
			state-store-mode = ddata
		}
		client.receptionist.role = pricing-engine # stops ClusterClient gossip from going to the worker nodes
		price-singleton{
			singleton-name = "price-initiator"
			role = "pricing-engine"
			hand-over-retry-interval = 1s
			min-number-of-hand-over-retries = 10
		}
	}

The price-singleton sub config was missing from the lesson.

Discrepancy 2

The Akka.CQRS.Pricing.Service/Program.cs file has been updated with the price initiator update (#57), the source listing on the lesson needs to be updated with the var priceInitiatorActor line:

var shardRegion = sharding.Start("priceAggregator",
                s => Props.Create(() => new MatchAggregator(s)),
                ClusterShardingSettings.Create(actorSystem),
                new StockShardMsgRouter());

            var priceInitiatorActor = actorSystem.ActorOf(ClusterSingletonManager.Props(Props.Create(() => new PriceInitiatorActor(shardRegion)),
                ClusterSingletonManagerSettings.Create(actorSystem).WithRole("pricing-engine").WithSingletonName("priceInitiator")), "priceInitiator");

            var clientHandler =
                actorSystem.ActorOf(Props.Create(() => new ClientHandlerActor(shardRegion)), "subscriptions");

Discrepancy 3

The Akka.CQRS.TradeProcessor.Service/Program.cs has been updated, the source listing on the lesson needs to be updated.
The lesson asks the student to replace this line:

var orderBooks = actorSystem.ActorOf(Props.Create(() => new OrderBookMasterActor()), "orderbooks");

with this:

var shardRegion = sharding.Start("orderBook", s => OrderBookActor.PropsFor(s), ClusterShardingSettings.Create(actorSystem),
        new StockShardMsgRouter());

but on the dev branch, the code has been updated to:

            Cluster.Cluster.Get(actorSystem).RegisterOnMemberUp(() =>
            {
                var sharding = ClusterSharding.Get(actorSystem);

                var shardRegion = sharding.Start("orderBook", s => OrderBookActor.PropsFor(s), ClusterShardingSettings.Create(actorSystem),
                    new StockShardMsgRouter());
            });

Akka.CQRS.Infrastructure\Ops\phobos.conf

In the dev branch, the HOCON config misses the phobos.monitoring.sample-rate = 0.1 and phobos.tracing.sample-rate = 0.1 properties.

When I ran the command, it outputs this error.

unable to recognize no matches for kind "Deployment" in version "extensions/v1beta1"

It did create some resources in the cluster. At this point I don't know if it is a problem or not. I will look around for an updated YAML.

Need to port these changes #92 to lesson6 branch.

HI,

Has anyone been able to host an Akka actor inside a DaemonSet? What was your YAML configuration? Any example would be much appreciated!

We were successful at deploying multiple Akka services in their own StatefulSet pods but we have not been able to find the correct configuration to switch one of our service to a DaemonSet instead.

The pod that is now deployed in a DaemonSet is no longer reachable by the LightHouse pods. See error below

[WARNING][02/09/2021 22:04:17][Thread 0014][akka.tcp://[email protected]:9600/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fedgraphjobs%jobprocessor-svc-v1-s9899.jobprocessor-svc%3A9600-9690/endpointWriter] AssociationError [akka.tcp://[email protected]:9600] -> akka.tcp://[email protected]:9600: Error [Association failed with akka.tcp://[email protected]:9600] []
[WARNING][02/09/2021 22:04:17][Thread 0014][remoting (akka://edgraphjobs)] Tried to associate with unreachable remote address [akka.tcp://[email protected]:9600]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Association failed with akka.tcp://[email protected]:9600] Caused by: [System.AggregateException: One or more errors occurred. (Name or service not known)
 ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
   at System.Net.Dns.InternalGetHostByName(String hostName)
   at System.Net.Dns.ResolveCallback(Object context)

Here is the YAML configuration for the JobProcessor service (which was previously working as a StatefulSet)

apiVersion: v1
kind: Service
metadata:
  name: jobprocessor-svc
  namespace: edgraph
  labels:
    app: jobprocessor-svc
    service: jobprocessor-svc
spec:
  clusterIP: None
  ports:
  - port: 9600
    name: akka-tcp
  selector:
    app: jobprocessor-svc
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: edgraph-jobprocessor-svc
  namespace: edgraph
  labels:
    account: jobprocessor-svc
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: jobprocessor-svc-v1
  namespace: edgraph
  labels:
    app: jobprocessor-svc
    version: v1
spec:
  selector:
    matchLabels:
      app: jobprocessor-svc
      version: v1
  template:
    metadata:
      labels:
        app: jobprocessor-svc
        version: v1
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: agentpool
                operator: In
                values:
                - jobsnodepool
      serviceAccountName: edgraph-jobprocessor-svc
      terminationGracePeriodSeconds: 35
      containers:
      - name: jobprocessor-svc
        image: jobprocessor-svc:1.0.0
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "pbm 127.0.0.1:9110 cluster leave"]
        ports:
        - containerPort: 9600
          protocol: TCP
        env:
          - name: Environment__Name
            value: "Development"
          - name: ASPNETCORE_ENVIRONMENT
            value: "Development"
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: CLUSTER_IP
            value: "$(POD_NAME).datasync-jobprocessor-svc"
          - name: CLUSTER_PORT
            value: "9600"
          - name: CLUSTER_SEEDS
            value: "akka.tcp://[email protected]:9100,akka.tcp://[email protected]:9100,akka.tcp://[email protected]:9100"
        livenessProbe:
          tcpSocket:
            port: 9600
      nodeSelector:
        kubernetes.io/os: linux
---

The solution repeatedly failed to build locally for me until I installed nuget.exe from https://dist.nuget.org/win-x86-commandline/latest/nuget.exe and added it to my PATH. This was an issue in both VS 2019 and VS Code. In Visual Studio, after opening the solution, building the project would fail due to a failure to restore any nuget packages. Attempting to navigate to Options -> Nuget Package Manager -> Package Sources simply displayed an error with no helpful information. This problem occurred for me only with this solution; I just worked through several of the akka.net boot camp solutions (as well as solutions related to my job) without ever experiencing it.

Might be good to note in the readme, as it threw me for quite a loop!

Why Coordinated Shutdown is not used in this repository? This repo is the only example (at least I was not able to find more) that contains examples of Akka.NET k8s deployment

According to @Aaronontheweb answer: here usage of Coordinated Shutdown

... is the recommended way of doing things since Akka.NET 1.3.2

Need to do this so we can cache the parts of the image that don't change frequently in the Docker build stack, namely the pbm install and environment variable declarations. Code changes should come at the very end.

The pricing-engine definition inside docker-compose.yaml didn't get explained/added in lesson 4, it suddenly appears in the lesson 5 .yaml file.

Shouldn't pull grafana:latest - everything should be tagged against a specific version.

Hi,
are we losing asks and bids here?