License: Apache License 2.0
If you've followed a link to this repository, but are not really sure what it contains or how to use it, head over to Industrial Edge for additional context and installation instructions.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
After deploying IE 2.0 on OCP 4.8, JupyterHub pods are not running in project manuela-ml-workspace as indicated in the instructions.
I did deploy the RHPDS fully deployed demo today.
Problem Statement:
manuela line dashboard does not show data, not events being shipped to kafka.
All pods in the manuela-stormshift-machine-sensor namespace are crash looping with error message:
java.net.UnknownHostException: broker-amq-mqtt-all-0-svc-rte-manuela-stormshift-messaging.apps.edge-kfsmk.kfsmk.sandbox230.opentlc.com
Root cause: the config is wrong, the host name "...edge...." is incorrect, it needs to be "...cluster..." in the RHPDS fully deployed version.
Here is the correct hostname from the route in manuel-stormshift-messaging:
broker-amq-mqtt-all-0-svc-rte-manuela-stormshift-messaging.apps.cluster-kfsmk.kfsmk.sandbox230.opentlc.com
Workaround:
Change the ConfigMap "machine-sensor-1|" in namespace from ...edge... to ...cluster...
The common folder is not cloned by default since it is a submodule. You need to run the git submodule update --init command after doing the initial git clone... on your repository.
[clouds]➜ manufacturing-edge-ai-ml git:(main) git version
git version 2.33.0
[clouds]➜ manufacturing-edge-ai-ml git:(main) git submodule
4011679f228aa96830ef1522d55dc8b82949d772 common (4011679)
[clouds]➜ manufacturing-edge-ai-ml git:(main) git submodule update --init
Submodule 'common' (https://github.com/hybrid-cloud-patterns/common.git) registered for path 'common'
Cloning into '/Users/nstephan/gpte/manufacturing-edge-ai-ml/common'...
Submodule path 'common': checked out '4011679f228aa96830ef1522d55dc8b82949d772'
This causes the machine-sensor pods to not function correctly.
Recently the 7.x AMQ broker channel moved to v7.9 of the AMQ broker - we were expecting 7.8.
Plan of attack is to specify v7.8 in the subscription spec (7.8 is a long term support release of AMQ), and rebuild the cluster, seeing if that allows the machine sensor pods to come up.
Error logs from the machine sensor pods:
Starting the Java application using /opt/jboss/container/java/run/run-java.sh ...
52
INFO exec java -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/iot-software-sensor-0.0.1-SNAPSHOT.jar
51
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
50
49
. ____ _ __ _ _2022-04-19T20:34:02.728374348Z
48
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \2022-04-19T20:34:02.728442581Z
47
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \2022-04-19T20:34:02.728501767Z
46
\\/ ___)| |_)| | | | | || (_| | ) ) ) )2022-04-19T20:34:02.728566219Z
45
' |____| .__|_| |_|_| |_\__, | / / / /2022-04-19T20:34:02.728953111Z
44
=========|_|==============|___/=/_/_/_/2022-04-19T20:34:02.729039178Z
43
:: Spring Boot :: (v2.2.1.RELEASE)2022-04-19T20:34:02.733999368Z
42
41
I> No access restrictor found, access to any MBean is allowed2022-04-19T20:34:02.832289331Z
40
Jolokia: Agent started with URL https://10.131.0.66:8778/jolokia/
39
2022-04-19 20:34:02.888 INFO 1 --- [ main] com.redhat.examples.iot.Application : Starting Application v0.0.1-SNAPSHOT on machine-sensor-1-568464bfdc-tz6jv with PID 1 (/deployments/iot-software-sensor-0.0.1-SNAPSHOT.jar started by jboss in /deployments)
38
2022-04-19 20:34:02.891 INFO 1 --- [ main] com.redhat.examples.iot.Application : No active profile set, falling back to default profiles: default
37
2022-04-19 20:34:05.282 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
36
2022-04-19 20:34:05.447 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
35
2022-04-19 20:34:05.448 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.27]
34
2022-04-19 20:34:05.511 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
33
2022-04-19 20:34:05.511 INFO 1 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 2568 ms
32
2022-04-19 20:34:06.303 INFO 1 --- [ main] c.redhat.examples.iot.SchedulerManager : Starting up...
31
2022-04-19 20:34:06.303 INFO 1 --- [ main] c.redhat.examples.iot.SchedulerManager : ---------------------------
30
2022-04-19 20:34:06.303 INFO 1 --- [ main] c.redhat.examples.iot.SchedulerManager : Application Name: iot-sensor
29
2022-04-19 20:34:06.305 INFO 1 --- [ main] c.redhat.examples.iot.SchedulerManager : Device ID: pump-1
28
2022-04-19 20:34:06.305 INFO 1 --- [ main] c.redhat.examples.iot.SchedulerManager : ---------------------------
27
2022-04-19 20:34:06.310 INFO 1 --- [ main] c.redhat.examples.iot.SchedulerManager : Sensor type temperature disabled
26
2022-04-19 20:34:06.356 INFO 1 --- [ main] c.redhat.examples.iot.mqtt.MqttProducer : MQTT Connect options
25
2022-04-19 20:34:06.358 INFO 1 --- [ main] c.redhat.examples.iot.mqtt.MqttProducer :
24
============== Connection options ==============
23
CleanSession : true
22
SocketFactory : null
21
MqttVersion : 0
20
KeepAliveInterval : 60
19
ConTimeout : 30
18
UserName : iotuser
17
SSLProperties : null
16
WillDestination : null
15
==========================================
14
13
2022-04-19 20:34:06.389 ERROR 1 --- [ main] c.redhat.examples.iot.mqtt.MqttProducer : MqttException
12
11
org.eclipse.paho.client.mqttv3.MqttException: MqttException
10
at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:38) ~[org.eclipse.paho.client.mqttv3-1.2.2.jar!/:na]
9
at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:736) ~[org.eclipse.paho.client.mqttv3-1.2.2.jar!/:na]
8
at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_322]
7
Caused by: java.io.IOException: WebSocket Response header: Incorrect connection header
6
at org.eclipse.paho.client.mqttv3.internal.websocket.WebSocketHandshake.receiveHandshakeResponse(WebSocketHandshake.java:157) ~[org.eclipse.paho.client.mqttv3-1.2.2.jar!/:na]
5
at org.eclipse.paho.client.mqttv3.internal.websocket.WebSocketHandshake.execute(WebSocketHandshake.java:80) ~[org.eclipse.paho.client.mqttv3-1.2.2.jar!/:na]
4
at org.eclipse.paho.client.mqttv3.internal.websocket.WebSocketNetworkModule.start(WebSocketNetworkModule.java:67) ~[org.eclipse.paho.client.mqttv3-1.2.2.jar!/:na]
3
at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:722) ~[org.eclipse.paho.client.mqttv3-1.2.2.jar!/:na]
2
... 1 common frames omitted
1
While deploying against a 4.10 cluster in a clean environment the Camel-K Operator tries to pull the wrong sha256 for its image..
Had to manually edit the CSV and change the sha from:
registry.redhat.io/integration/camel-k-rhel8-operator@sha256:d4a49a873879d2e89ed2f4ea732cdc2cbddcf2ea2e5c136bcf1937797f7ae908
to:
registry.access.redhat.com/integration/camel-k-rhel8-operator@sha256:98363f3f89bfc8308915282eefcf878b995e5abe49eaa43333296253c6c30e97
[camel-k-operator-97697fb5b-5x9dm](https://console-openshift-console.apps.cn-chengdu.sandbox283.opentlc.com/k8s/ns/manuela-stormshift-messaging/pods/camel-k-operator-97697fb5b-5x9dm)
NamespaceNS[manuela-stormshift-messaging](https://console-openshift-console.apps.cn-chengdu.sandbox283.opentlc.com/k8s/cluster/namespaces/manuela-stormshift-messaging)
8 minutes ago
Generated from kubelet on [ip-192-168-21-132.us-west-2.compute.internal](https://console-openshift-console.apps.cn-chengdu.sandbox283.opentlc.com/k8s/cluster/nodes/ip-192-168-21-132.us-west-2.compute.internal)
4 times in the last 10 minutes
Failed to pull image "registry.redhat.io/integration/camel-k-rhel8-operator@sha256:d4a49a873879d2e89ed2f4ea732cdc2cbddcf2ea2e5c136bcf1937797f7ae908": rpc error: code = Unknown desc = reading manifest sha256:d4a49a873879d2e89ed2f4ea732cdc2cbddcf2ea2e5c136bcf1937797f7ae908 in registry.redhat.io/integration/camel-k-rhel8-operator: unsupported: V2 schema 1 manifest digests are no longer supported for image pulls. Use the equivalent V2 schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332
The two applications named above show out of sync perpetually in ArgoCD. This is because there are RoleBindings and ClusterRoleBindings that are part of the installation that are managed by the operators and are not represented in the GitOps structure.
Sometimes pipeline tasks or steps fail with what seem like transient errors. Some examples:
level=error msg="Error while applying layer: ApplyLayer io: read/write on closed pipe stdout: {\"layerSize\":7301}\n stderr: "
error creating build container: Error committing the finished image: error adding layer with blob
time="2021-09-29T18:48:27Z" level=fatal msg="Error trying to reuse blob sha256:235f9e6f3559c04d5ee09b613dcab06dbc03ceb93b65ce364afe35c03fd53574 at destination: failed to read from destination repository martjack/iot-software-sensor: 500 (Internal Server Error)
I1006 22:07:47.908257 14 request.go:645] Throttling request took 1.195150708s, request: GET:https://172.30.0.1:443/apis/autoscaling.openshift.io/v1?timeout=32s
PipelineRun started: seed-iot-software-sensor-run-cpzzv
Waiting for logs to be available...
E1006 22:08:27.106369 14 runtime.go:78] Observed a panic: "send on closed channel" (send on closed channel)
goroutine 487 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1b40ee0, 0x1fe47b0)
/workspace/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/workspace/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x89
panic(0x1b40ee0, 0x1fe47b0)
Is there anything we can do to fix this?
Vibration anomalies are not detected in the line sensor dashboard even though messaging is configured accordingly. The anomaly-detection-predictor pods are not running in namespace manuela-tst-all.
Environment: OCP 4.8, deployed through RHPDS (OpenShift 4.8 Industrial Edge - Deployed).
After updating the sensor application and triggering the delivery pipeline using make build-and-test, the final pipeline run for stage-production fails during task github-pull-request. The error message reads:
declared workspace "github-secret" is required but has not been bound
Environment: OCP 4.8, deployed through RHPDS (OpenShift 4.8 Industrial Edge - Deployed).
This is a by-product of the way we are installing patterns. Ideally each application should "own" an installation as this kind of overlap creates noise in the system.
Hi,
I tried to follow the documentation.
Unfortunately it is completely outdated.
The referred scripts like make_common_subtree.sh in v2.1.1 are not available any more. And it seems they are missing in all the branches.
Can you please update the docs.
Cheers
Manfred
make[2]: Leaving directory '/home/martjack/gitwork/newprints/industrial-edge'
make[1]: Leaving directory '/home/martjack/gitwork/newprints/industrial-edge'
make argosecret
make[1]: Entering directory '/home/martjack/gitwork/newprints/industrial-edge'
echo "Delegating argosecret target"
Delegating argosecret target
make -f common/Makefile argosecret
make[2]: Entering directory '/home/martjack/gitwork/newprints/industrial-edge'
PATTERN="" TARGET_NAMESPACE="" COMPONENT="" SECRET_NAME="" common/scripts/secret.sh
Checking for namespace to exist: OK
Checking for secrets/-gitops-cluster to be populated in -: |^Cmake[2]: *** [common/Makefile:29: argosecret] Interrupt
make[1]: *** [Makefile:24: argosecret] Interrupt
make: *** [Makefile:35: install] Interrupt
martjack@martjack:~/gitwork/newprints/industrial-edge
2159db3f main % u= origin/main
% git checkout v2.1
Switched to branch 'v2.1'
Your branch is up to date with 'upstream/v2.1'.
martjack@martjack:~/gitwork/newprints/industrial-edge
a1843c71 v2.1 % u= upstream/v2.1
% make argosecret
make -f common/Makefile \
PATTERN=industrial-edge TARGET_NAMESPACE=manuela-ci \
SECRET_NAME="argocd-env" COMPONENT=datacenter argosecret
make[1]: Entering directory '/home/martjack/gitwork/newprints/industrial-edge'
PATTERN="industrial-edge" TARGET_NAMESPACE="manuela-ci" COMPONENT="datacenter" SECRET_NAME="argocd-env" common/scripts/secret.sh
Checking for namespace manuela-ci to exist: OK
Checking for secrets/datacenter-gitops-cluster to be populated in industrial-edge-datacenter: OK
Conditions met, managing secret argocd-env in manuela-ci
secret/argocd-env created
make[1]: Leaving directory '/home/martjack/gitwork/newprints/industrial-edge'
The workaround is to checkout v2.1 and run make argosecret. This seems to be because the PATTERN, TARGET_NAMESPACE, COMPONENT, and SECRET variables are not defined in the industrial-edge Makefile. (I signed off on removing them, sorry!).
A better question might be whether this should delegate to common at all, or whether we should move this code purely into industrial edge. It will have to be modified in any case, I think
I tried to run this today.
Ordered edge demo env from RHPDS.
Cloned the repos as instructed here,
https://github.com/hybrid-cloud-patterns/industrial-edge#start-here
But the environment doesn't get deployed correctly.
Difficult to know exactly where things go wrong, but one thing that might be relevant is that a pipelin run in manuela-ci fails,
"
PipelineRun manuela-ci/seed-run-9jr8h doesn't bind Pipeline manuela-ci/seed's Workspaces correctly: pipeline requires workspace with name "github-secret" be provided by pipelinerun
"
found that # Secret used for S3 functionality in manuela-data-lake is required on values-secret.yaml. May I know it is a must as I don't have it. thank you
Ray
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.