Hi! ๐
I've tried deploying the everest-demo
container images to a Kubernetes cluster (specifically Amazon EKS), and although mqtt-server
and node-red
deployed successfully, manager
kept crashing with the following error:
[2024-06-05 13:53:18.883199] [0x00007f730994bb48] [info] Main manager process exits because of caught exception:
stoi
I wrote the following script to convert the Docker Compose file to a Helm chart for deployment to Kubernetes:
#!/bin/sh
# This script requires the following tools, all of which can be installed with Homebrew:
# - wget
# - kompose
# - kubectl
# - helm
# - kubectx
export EVEREST_MANAGER_CPUS='2.0'
export EVEREST_MANAGER_MEMORY='1536mb'
export TAG='0.0.16'
# Change the following value to the proper Kubernetes context alias in your configuration
K8S_CONTEXT_ALIAS=k8s-tooling-cluster
mkdir -p ./tmp
rm -rf ./tmp/*
cd ./tmp || exit
wget https://raw.githubusercontent.com/EVerest/everest-demo/main/docker-compose.iso15118-dc.yml
kompose -f docker-compose.iso15118-dc.yml convert -c
cd ..
helm lint ./tmp/docker-compose.iso15118-dc || exit
kubectx ${K8S_CONTEXT_ALIAS} || exit
helm upgrade everest ./tmp/docker-compose.iso15118-dc --cleanup-on-fail --create-namespace --description "EVerest demo" --dry-run=client --install --namespace everest || exit
helm upgrade everest ./tmp/docker-compose.iso15118-dc --cleanup-on-fail --create-namespace --description "EVerest demo" --dry-run=server --install --namespace everest || exit
echo "Helm dry-run on server successful. To actually deploy to the tooling cluster, run the following command:"
echo ""
echo "kubectx ${K8S_CONTEXT_ALIAS} && helm upgrade everest ./tmp/docker-compose.iso15118-dc --cleanup-on-fail --create-namespace --description \"EVerest demo\" --install --namespace everest"
echo ""
I ran this script, and then I ran the command that the script printed out in the end and the deployment initially appeared to be successful, until I noticed the pod belonging to the deployment manager
constantly restarting and failing and eventually going into CrashLoopBackOff
.
A kubectl -n everest logs [pod name]
yielded the "stoi" error that I mentioned further above.
When I changed spec.template.spec.containers[0].command
in the generated manager-deployment.yaml
file as below and then ran the same helm upgrade
command again, I managed to get the pod to start successfully, so I could log into it and try some troubleshooting:
spec:
containers:
- command: [ "/bin/sh" ]
args: [ "-c", "while true; do echo hello; sleep 10;done" ]
When I ran the helm upgrade
command again to apply this, the manager
pod started successfully, and then I could log into it as follows (note that you need to replace the [manager-pod-name]
part, because that changes every time the deployment is updated and a new pod is spun up):
kubectl -n everest exec pod/[manager-pod-name] -it -- /bin/sh
In this console, it was easy to recreate the error:
/ext/source/build/run-scripts/run-sil-dc.sh
I noticed that I could run manager
with the --help
option just fine (I looked in in run-sil-dc.sh
to see how I could run it) :
LD_LIBRARY_PATH=/ext/source/build/dist/lib:$LD_LIBRARY_PATH PATH=/ext/source/build/dist/bin:$PATH manager --help
...But whenever I would try any of the configurations in /ext/source/config
, I would get that weird stoi
(string to integer conversion?) exception, regardless of whether or not I included the --check
option:
LD_LIBRARY_PATH=/ext/source/build/dist/lib:$LD_LIBRARY_PATH PATH=/ext/source/build/dist/bin:$PATH manager --check --conf /ext/source/config/config-sil-dc.yaml
LD_LIBRARY_PATH=/ext/source/build/dist/lib:$LD_LIBRARY_PATH PATH=/ext/source/build/dist/bin:$PATH manager --check --conf /ext/source/config/config-example.yaml
LD_LIBRARY_PATH=/ext/source/build/dist/lib:$LD_LIBRARY_PATH PATH=/ext/source/build/dist/bin:$PATH manager --conf /ext/source/config/config-example.yaml
I took at look at the manager.cpp source code, but it wasn't very clear where exactly the exception was being thrown, because no other hints were being given other than the exception message stoi
.
It appears to be happening somewhere in the int boot(...)
function, before the splash banner is printed with EVLOG_info
.
Strangely enough, when I run the same Docker container image locally, I can't reproduce this issue:
docker run --rm -it --platform linux/amd64 --entrypoint sh ghcr.io/everest/everest-demo/manager:0.0.16
LD_LIBRARY_PATH=/ext/source/build/dist/lib:$LD_LIBRARY_PATH PATH=/ext/source/build/dist/bin:$PATH manager --check --conf /ext/source/config/config-sil-dc.yaml
The result in that case, on an Apple Silicon MacBook, running the Docker container in x86 emulation mode:
2024-06-05 14:21:36.996736 [INFO] manager :: ________ __ _
2024-06-05 14:21:37.001297 [INFO] manager :: | ____\ \ / / | |
2024-06-05 14:21:37.001332 [INFO] manager :: | |__ \ \ / /__ _ __ ___ ___| |_
2024-06-05 14:21:37.001609 [INFO] manager :: | __| \ \/ / _ \ '__/ _ \/ __| __|
2024-06-05 14:21:37.001629 [INFO] manager :: | |____ \ / __/ | | __/\__ \ |_
2024-06-05 14:21:37.001643 [INFO] manager :: |______| \/ \___|_| \___||___/\__|
2024-06-05 14:21:37.001659 [INFO] manager ::
2024-06-05 14:21:37.001689 [INFO] manager :: Using MQTT broker localhost:1883
2024-06-05 14:21:37.016064 [ERRO] manager int main(int, char**) :: Main manager process exits because of caught exception:
Syscall pipe2() failed (Invalid argument), exiting
It's also an error, but at least a different one.
I'm a bit at a loss now.
Could you maybe help me getting this deployed to Kubernetes? (So far I've only tried AWS EKS, but I guess I could try this in a local minikube cluster or something too. Let me know if that would help.)
I also noticed that the everest-demo/manager
container images are not yet multi-platform, but the test cluster in which I tried to deploy it has nodes running on an x86_64 architecture, so that shouldn't be the problem.
Thank you kindly in advance with helping me getting this deployed to our test cluster for evaluation! ๐