Giter Club home page Giter Club logo

lf-edge / edge-home-orchestration-go Goto Github PK

View Code? Open in Web Editor NEW
91.0 13.0 48.0 5.21 MB

Home Edge Project in LF Edge - Edge Orchestration for home edge devices to enabling smart home use cases.

Home Page: https://www.lfedge.org/projects/homeedge/

License: Apache License 2.0

Shell 0.57% Go 97.66% Makefile 1.25% HTML 0.23% CSS 0.09% JavaScript 0.20%
homeedge edgecomputing lfedge iot orchestration containerization docker cluster-computing container node data-storage rbac hikey960 raspberry-pi home-iot

edge-home-orchestration-go's Introduction

Edge Orchestration

Goreport Build CodeQL Analysis Quality Gate Status CII Best Practices OpenSSF Scorecard RepoSize Release LICENSE

Introduction

The main purpose of Edge Orchestration project is to implement distributed computing between Docker Container enabled devices.

image

All the devices (TVs, fridges, washing machines, etc.) connected into Home Edge Network are considered Home Edge Devices. Some of the devices, possessing significant H/W resources and capable of running containerized applications, are called Home Edge Nodes. Tasks performed on devices of Home Edge Network are managed by Home Edge Orchestrator software. These tasks are called Home Edge Applications. They are assigned to specific Home Edge Nodes. As of now, Home Edge Orchestrator incessantly scans the Home Edge Network, forming lists of attached devices, and forms ratings of performance for them. Performance ratings are basis for deciding on which devices Home Edge Applications are to be run. If Home Edge Orchestrator cannot find device with rating higher than its own device rating, it will start the Home Edge Application locally. Home Edge Networks support distributed applications consisting of interacting Docker container instances. Docker containers offer quick deployment, easy management, safety and hardware independence. Applications that can run on Home Edge Network are deployed from cloud-based centralized storage as needed. Assortment of applications is determined by tasks that must be solved by Home Edge Network.

A description of all terms and abbreviations (italic) used in the Edge Orchestratiopn project can be found in the Home Edge Glossary


Roadmap and Release Notes

You can see our project roadmap and details about release notes since our project launch in 2019 from our official Wiki.

Platforms supported

x86-64 Linux Raspberry Pi3 HiKey960 Raspberry Pi3 Cluster
Orange Pi3 Add a new Platform

Quick start of the Edge Orchestration (w/ and w/o build)

Prerequisites

  • At least one device (see Platforms supported) that can be a Home Edge Device (preferably more than one)
  • Same network connected among the Home Edge Devices.
  • Same Authentication key in /var/edge-orchestration/user/orchestration_userID.txt
  • Edge Orchestration Docker image

The quick start guide contains instructions on where to find Edge Orchestration Docker image and how to create Authentication key

The Edge Orchestration workflow is as follows:

  1. Run the Edge Orchestration container at Home Edge Nodes.
  2. Home Edge Device generates and sends a Service Execution request via the REST API to a host machine (currently itself).
  3. Then the host machine becomes an Home Edge Orchestrator.
  4. Other Home Edge Nodes receive the Service Execution request from Home Edge Orchestrator via REST API.
  5. The Home Edge Node gets and compares the scores from the other Home Edge Nodes.
  6. The Service Execution on the Home Edge Node that has the highest score.

Home Edge Orchestrator also gets and compares scores and if it has the highest score, the service is executed on Home Edge Orchestrator.

Quick start guides for supported platforms

Please select the target platform in the table below and follow the link.

Platform Maintained Maintainer Remarks
x86_64 Linux dewberries @lf-edge/edge-home-orchestration-go-committers
Raspberry Pi3 dewberries @t25kim
HiKey960 dewberries @tdrozdovsky
Raspberry Pi3 Cluster dewberries @tdrozdovsky
Orange Pi3 dewberries @Vitalii55

Documentation

The Edge Orchestration documentation for the project is located in the docs folder. You can also familiarize yourself with other materials and plans by visiting the wiki.


License

The Edge Orchestration source code is distributed under the Apache License, Version 2.0 open source license.


Contributing

If you want to contribute to the Edge Orchestration project and make it better, your help is very welcome. Contributing is also a great way to learn more about social coding on Github, new technologies and and their ecosystems.
How to contribute code
How to report a security vulnerability


edge-home-orchestration-go's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

edge-home-orchestration-go's Issues

create Docker container fail

while executing the ./build.sh file in Ubuntu 16.04(x86_64). I got the following error


Create Docker container


edge-orchestration
Untagged: edge-orchestration:alpha
Deleted: sha256:c533b88f3523957580cf73b76219b17ba7db52ccaebd84f19da596fab3e063ec
Deleted: sha256:1f2237144efc2ffef4098f24147c617655c87f1db8d647421e482120e4ce285d
Deleted: sha256:27e58b236a7c06911b99229175562fc59df2e8512dfdea8b22468df74af91fa0
Deleted: sha256:ba7320ba8a6fe7a0763dd7bb6fd491af5f471c77e697be4a0eafd0e26086dead
docker build --tag edge-orchestration:alpha --file /home/Documents/mec/edge-home-orchestration-go/GoMain/Dockerfile .
Sending build context to Docker daemon 260.8MB
Step 1/16 : FROM ubuntu:16.04
---> 5e13f8dd4c1a
Step 2/16 : ENV TARGET_DIR=/edge-orchestration
---> Using cache
---> ac9bd4191d91
Step 3/16 : ENV HTTP_PORT=56001
---> Using cache
---> eaed1ac412ca
Step 4/16 : ENV MDNS_PORT=5353
---> Using cache
---> a7721d9a78e7
Step 5/16 : ENV ZEROCONF_PORT=42425
---> Using cache
---> bc930482e7d5
Step 6/16 : ENV APP_MAIN_DIR=GoMain
---> Using cache
---> a3ae31bec427
Step 7/16 : ENV APP_BIN_DIR=$APP_MAIN_DIR/bin
---> Using cache
---> c38738405bbb
Step 8/16 : ENV APP_NAME=edge-orchestration
---> Using cache
---> 6a80fbb4f830
Step 9/16 : COPY $APP_BIN_DIR/$APP_NAME $APP_MAIN_DIR/run.sh $TARGET_DIR/
---> d9d3c9b20f41
Step 10/16 : EXPOSE $HTTP_PORT $MDNS_PORT $ZEROCONF_PORT
---> Running in c3b48ac48abb
Removing intermediate container c3b48ac48abb
---> ac5ee5bac7b5
Step 11/16 : RUN mkdir -p /var
---> Running in 9aad0c66fd6c
OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused "write /proc/self/attr/keycreate: invalid argument"": unknown
Makefile:93: recipe for target 'build-container' failed
make: *** [build-container] Error 1

TestGetDeivceList invoked pending

=== RUN TestDeviceDetectionRoutine/Success/SuccessNewDevice
--- FAIL: TestDeviceDetectionRoutine (4.00s)
--- FAIL: TestDeviceDetectionRoutine/Success (4.00s)
--- PASS: TestDeviceDetectionRoutine/Success/SuccessClearMap (1.00s)
--- PASS: TestDeviceDetectionRoutine/Success/SuccessDeleteDevice (1.00s)
--- FAIL: TestDeviceDetectionRoutine/Success/SuccessUpdateInfo (1.00s)
discovery_test.go:189: Info Not Updated :: [1.1.1.1]
--- FAIL: TestDeviceDetectionRoutine/Success/SuccessNewDevice (1.00s)
discovery_test.go:197: Device Not Generated
=== RUN TestGetDeviceList
SIGQUIT: quit
PC=0x45d0f1 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x80b428, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7fffa3220028, 0x40b001, ...)
/usr/local/go/src/runtime/sys_linux_amd64.s:535 +0x21
runtime.futexsleep(0x80b428, 0x0, 0xffffffffffffffff)
/usr/local/go/src/runtime/os_linux.go:46 +0x4b
runtime.notesleep(0x80b428)
/usr/local/go/src/runtime/lock_futex.go:151 +0xa1
runtime.stopm()
/usr/local/go/src/runtime/proc.go:1936 +0xc1
runtime.findrunnable(0xc00002ca00, 0x0)
/usr/local/go/src/runtime/proc.go:2399 +0x54a
runtime.schedule()
/usr/local/go/src/runtime/proc.go:2525 +0x21c
runtime.park_m(0xc000001800)
/usr/local/go/src/runtime/proc.go:2605 +0xa1
runtime.mcall(0x0)
/usr/local/go/src/runtime/asm_amd64.s:299 +0x5b

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000138100, 0x650af5, 0x11, 0x65ab48, 0x474400)
/usr/local/go/src/testing/testing.go:917 +0x381
testing.runTests.func1(0xc0000dc000)
/usr/local/go/src/testing/testing.go:1157 +0x78
testing.tRunner(0xc0000dc000, 0xc0000abdf8)
/usr/local/go/src/testing/testing.go:865 +0xc0
testing.runTests(0xc00000e2c0, 0x806b80, 0x11, 0x11, 0x0)
/usr/local/go/src/testing/testing.go:1155 +0x2a9
testing.(*M).Run(0xc0000da000, 0x0)
/usr/local/go/src/testing/testing.go:1072 +0x162
main.main()
_testmain.go:126 +0x1cb

goroutine 36 [semacquire]:
sync.runtime_SemacquireMutex(0x80aa1c, 0x7f2df2200d00)
/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0x80aa18)
/usr/local/go/src/sync/mutex.go:134 +0x109
controller/discoverymgr.newDeviceHandler(0x656462, 0x22, 0xc000150000)
/home/wonny/worksapce/for_test/src/controller/discoverymgr/discovery.go:520 +0x3b
controller/discoverymgr.addDevice(0xc000044f01)
/home/wonny/worksapce/for_test/src/controller/discoverymgr/discovery_test.go:84 +0x1c0
controller/discoverymgr.TestGetDeviceList(0xc000138100)
/home/wonny/worksapce/for_test/src/controller/discoverymgr/discovery_test.go:209 +0x35
testing.tRunner(0xc000138100, 0x65ab48)
/usr/local/go/src/testing/testing.go:865 +0xc0
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:916 +0x35a

goroutine 10 [semacquire]:
sync.runtime_SemacquireMutex(0x80aa1c, 0x7f2df2200000)
/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0x80aa18)
/usr/local/go/src/sync/mutex.go:134 +0x109
controller/discoverymgr.updateInfoHandler(0x656462, 0x22, 0xc0000b4190)
/home/wonny/worksapce/for_test/src/controller/discoverymgr/discovery.go:503 +0x3b
controller/discoverymgr.deviceDetectionRoutine.func1(0xc000102000)
/home/wonny/worksapce/for_test/src/controller/discoverymgr/discovery.go:406 +0x338
created by controller/discoverymgr.deviceDetectionRoutine
/home/wonny/worksapce/for_test/src/controller/discoverymgr/discovery.go:386 +0xec

Use signals instead of timer

How about changing time.sleep functions to the code controlled by signal such as SIGINT, SIGTERM?
I think it could expand other scenarios such as

  • A node sends it's termination request to other neighbors if the node receives SIGTERM.

Build error occurred in arm architecture

Describe the bug
During $./build.sh container arm or $./build.sh container arm64, I got the following build error.

**********************************
 Create Docker container
**********************************

Error: No such container: edge-orchestration
Error: No such image: edge-orchestration:baobab
docker build --tag edge-orchestration:baobab --file /home/t25kim/work/edge-home-orchestration-go/GoMain/Dockerfile --build-arg PLATFORM=arm32v7 .
Sending build context to Docker daemon  301.4MB
Step 1/17 : ARG PLATFORM
Step 2/17 : FROM $PLATFORM/ubuntu:16.04
 ---> 8eedc387cb89
Step 3/17 : ENV TARGET_DIR=/edge-orchestration
 ---> Using cache
 ---> 5f0a61e388ca
Step 4/17 : ENV HTTP_PORT=56001
 ---> Using cache
 ---> b067bb7683b3
Step 5/17 : ENV MDNS_PORT=5353
 ---> Using cache
 ---> 075a5c1d1c28
Step 6/17 : ENV MNEDC_PORT=8000
 ---> Using cache
 ---> c2bc2afb2a0a
Step 7/17 : ENV MNEDC_BROADCAST_PORT=3333
 ---> Using cache
 ---> 5730d8bb2687
Step 8/17 : ENV ZEROCONF_PORT=42425
 ---> Using cache
 ---> b6b101e9d328
Step 9/17 : ENV APP_MAIN_DIR=GoMain
 ---> Using cache
 ---> b6d791035071
Step 10/17 : ENV APP_BIN_DIR=$APP_MAIN_DIR/bin
 ---> Using cache
 ---> e178739df475
Step 11/17 : ENV APP_NAME=edge-orchestration
 ---> Using cache
 ---> e75f9b70dc2c
Step 12/17 : RUN apt update
 ---> Running in 244a90de8ea7
standard_init_linux.go:211: exec user process caused "exec format error"
The command '/bin/sh -c apt update' returned a non-zero code: 1
Makefile:94: recipe for target 'build-container' failed
make: *** [build-container] Error 1

To Reproduce
Steps to reproduce the behavior:
$./build.sh container arm or $./build.sh container arm64

Secure Mode Judgement

Is your feature request related to a problem? Please describe.
The Edge Orchestration runs on the secure mode when it's built with "BUILD_TAGS='secure'"

Describe alternatives you've considered
How about deciding the secure mode if all the necessary files are in the edgeDir subpath?

  • example of necessary files : orchestration_userID.txt, passPhraseJWT.txt, app_rsa.pub etc

OrchestrationInit() failed for java binding / c binding due to recent security changes

Describe the bug
Due to recent changes https://github.com/lf-edge/edge-home-orchestration-go/pull/90/files which mandate verifier to be enabled for all the platforms: Container, native and android. It makes orchestrationInit() API to be failed for interfaces (src/interfaces/javaapi and src/interfaces/capi)

To Reproduce
Steps to reproduce the behavior:

  1. Generate java object.
  2. Include binding in application.
  3. Call orchestrationInit() API. (It is failing)

Expected behavior
OrchestrationInit() should work for all the platform.

Suggestion
File Name: orchestration.go
Line No: 162 (!o.isSetVerifierConf )

The above check should be done only for container platform. As this is common file for all the platform.
Please check the common files for different platform and put necessary conditional checks.

Build source code to x86_64 on RPi

Describe the bug
The RPi architecture is arm but it builds the source code with x86_64.

To Reproduce

  1. Download source code on RPi
  2. Build source code like below
    $ ./build

Screenshots

pi@raspberrypi:~/workspace/edge-home-orchestration-go $ uname -m
armv7l
pi@raspberrypi:~/workspace/edge-home-orchestration-go $ docker logs edge-orchestration
standard_init_linux.go:211: exec user process caused "exec format error"

Test environment configuration (please complete the following information):

  • Hardware: RPi2
  • Edge Orchestration Release: Baobab

Need to add popular platforms (for ex. Raspberry Pi x)

Is your feature request related to a problem? Please describe.
Lack of supported platforms other than x86_64

Describe the solution you'd like
It would be nice to add popular platforms like as Raspberry Pi x to attract more developers.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Any platform on which Linux can be run is suitable for this.

[Suggestion] README enhancement

Is your feature request related to a problem? Please describe.
We can improve the current README.md by simply adding several KPI barometers that are already available in public. We have also presented one example as "GoReportCard" in the current document.

Describe the solution you'd like
To employ the existing open source tools to present additional KPIs from the README.md

Describe alternatives you've considered
Currently, the suggestion is the simplest way.

Additional context
N/A

Ping to offline devices consistently to update RTT

Describe the bug
The Edge Orchestrator ping to offline devices to check RTT values consistently.

2020/07/16 22:58:50 discovery.go:374:
2020/07/16 22:58:54 helper.go:99: [http://10.113.71.53:56002/api/v1/ping] reqeust get failed !!, err = Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
2020/07/16 22:58:59 helper.go:99: [http://10.113.71.53:56002/api/v1/ping] reqeust get failed !!, err = Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
2020/07/16 22:59:04 helper.go:99: [http://10.113.71.53:56002/api/v1/ping] reqeust get failed !!, err = Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
2020/07/16 22:59:09 helper.go:99: [http://10.113.71.53:56002/api/v1/ping] reqeust get failed !!, err = Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
2020/07/16 22:59:14 helper.go:99: [http://10.113.71.53:56002/api/v1/ping] reqeust get failed !!, err = Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
2020/07/16 22:59:19 helper.go:99: [http://10.113.71.53:56002/api/v1/ping] reqeust get failed !!, err = Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
2020/07/16 22:59:24 helper.go:99: [http://10.113.71.53:56002/api/v1/ping] reqeust get failed !!, err = Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
2020/07/16 22:59:29 helper.go:99: [http://10.113.71.53:56002/api/v1/ping] reqeust get failed !!, err = Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
2020/07/16 22:59:34 helper.go:99: [http://10.113.71.53:56002/api/v1/ping] reqeust get failed !!, err = Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused
Get "http://10.113.71.53:56002/api/v1/ping": dial tcp 10.113.71.53:56002: connect: connection refused

To Reproduce

  1. Set up a connection between devices.
  2. Check logs after keeping a device offline

Expected behavior
The Edge Orchestrator should delete an offline device info from net DB and ping to online devices only.

Test environment configuration (please complete the following information):

  • Firmware version: Ubuntu 16.04
  • Hardware: x86-64
  • Edge Orchestration Release: Baobab

Launch of malicious containers

Describe the bug

A critical system security vulnerability was discovered. In case of hacking of any device and the possibility of transferring REST API to the LF Edge network, attcker can get full access to the system of any device. as an example, the case of access to the file system is described below.

To Reproduce
Steps to reproduce the behavior:

  1. Run Edge Orchestration container
$ docker run -it -d --privileged --network="host" --name edge-orchestration \
                -v /var/edge-orchestration/:/var/edge-orchestration/:rw \
                -v /var/run/docker.sock:/var/run/docker.sock:rw \
                -v /proc/:/process/:ro edge-orchestration:baobab
  1. Send the next REST API request
curl -X POST "127.0.0.1:56001/api/v1/orchestration/services" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"ServiceName\": \"Simple-server\", \"ServiceInfo\": [{ \"ExecutionType\": \"container\", \"ExecCmd\": [ \"docker\", \"run\", \"-v\", \"/var/run:/var/run:rw\", \"-v\", \"/:/app\", \"-p\", \"80:8000\", \"-p\", \"emazzotta/simple-fileserver-docker\"]}], \"StatusCallbackURI\": \"http://localhost:8888/api/v1/services/notification\"}"
  1. Start the browser and enter the ip address of the device on which the container was launched.
    After that, you will see the file system of the device under attack and get access to it

изображение

Expected behavior
Avoid launching unverified containers. Сreating a list of containers that can be run on the device.

Test environment configuration:

  • Software version: Ubuntu 16.04
  • Hardware: x86-64
  • Toolchain: Docker and Go recommended version
  • Edge Orchestration Release: Baobab

Two types of "service" in discoverymgr

I think there are two types of "service" in discoverymgr (discovery.go, wrapper.go, etc) which can confuse people.

  1. Service for mDNS/DNS-SD

    • Variables related to Zeroconf such as serviceEntry
  2. Service for Edge App

    • Most variables such as ServiceList in OrchestrationInformation

[Suggest] LICENSE file name

GitHub automatically parses a license file from a root repository into its project dashboard like "Apache2.0", "BSD-3.0" and etc., if and only if that file has the name of "LICENSE". Currently we have the file under the name of "LICENSE.Apache-2.0".

I would like to suggest to change the current license file name from "LICENSE.Apache-2.0" to "LICENSE". Does it make sense to all?

Balena colloboration with LF Home Edge

Balena:
Constraint: Balena OS should be preinstalled to use the Container Service.

OpenBalena is a stack of services which offer a combined service to manage balenaOS devices, like Raspberry Pi, Intel NUC etc. It is in no way a hypervisor or offering any kind of virtualization layers.
The issue observed how do Linux will be supported since LF Home Edge runs over Linux.
From the cloud side looks like there will be dependency on baleana cloud.

Build break due to golang version on RPi

Describe the bug
RPi instruction guides developers to install golang with command sudo apt install golang.
RPi install 1.11 version of golang and it fails to build the project.
Find the log below.

pi@raspberrypi:~/edge-home-orchestration-go $ go version
go version go1.11.6 linux/arm
pi@raspberrypi:~/edge-home-orchestration-go $ ./build.sh

-----------------------------------
 Install prerequisite packages
-----------------------------------
(1/7) go get -u github.com/axw/gocov/gocov: Done
(2/7) go get -u github.com/matm/gocov-html: Done
(3/7) go get -u golang.org/x/lint/golint: Done
(4/7) go get -u github.com/Songmu/make2help/cmd/make2help: Done
(5/7) go get -u golang.org/x/mobile/cmd/gomobile# golang.org/x/tools/go/packages
../go/src/golang.org/x/tools/go/packages/golist_overlay.go:547:7: undefined: strings.ReplaceAll

download fail

[Suggest] Go Report Card representation

Since this project has been written in GoLang, I would like to suggest to add the representation banner for "Go Report Card" from the "README.md", which is a famous index to most of the GoLang developers.

Just for your information, Home Edge project is currently evaluated as "A+" from the "Go Report" (reference to : https://goreportcard.com/report/github.com/lf-edge/edge-home-orchestration-go), which is a very encouraging information to the project.

You can check out how we are able to add such a index from the README.md with respect to the document from other project in LF Edge such as Project EVE and EdgeX Foundry. Here is an example.

[![CircleCI](https://circleci.com/gh/lf-edge/eve.svg?style=svg)](https://circleci.com/gh/lf-edge/eve)
[![Goreport](https://goreportcard.com/badge/github.com/lf-edge/eve)](https://goreportcard.com/report/github.com/lf-edge/eve)
[![Godoc](https://godoc.org/github.com/lf-edge/eve/pkg/pillar?status.svg)](https://godoc.org/github.com/lf-edge/eve/pkg/pillar)

RTT value initialization during device discovery

Describe the bug
The discoverymgr runs the deviceDetectionRoutine function to find devices at regular intervals.
During this period, it sets the RTT value to 0 even if the corresponding data is in the networkdb.

Expected behavior
If the networkdb doesn’t have corresponding data, the networkdb should insert or update the network information.

Screenshots
image

Test environment configuration (please complete the following information):

  • Firmware version: Ubuntu 16.04
  • Hardware: x86-64
  • Edge Orchestration Release: Baobab

[Discussion] Do we need to rectify SecurityMgr? (Sensitive Data handling)

I personally think that sensitive data is returned by an access to passPhraseJWTPath. What do you think? @tdrozdovsky

log.Panicf("Failed to create passPhraseJWTPath %s: %s\n", passPhraseJWTPath, err)

This might be the same potential security risk as follow.

log.Println(logPrefix, "cannot create "+passPhraseJWTFilePath+": ", err)

Additional tags for project promotion

I propose to add/remove/change tags to improve the indexing of our project and, therefore, increase the rating of the project in search engines.

Proposed to add tags:
orchestration
containerization
docker
cluster-computing
container
node
data-storage
rbac
hikey960
raspberry-pi

Proposed to remove tags:
edge-orchestration - name of repository

Proposed to change name tags:
homeiot -> iot

What do you think about it?

How about merging current configuration db and serviceInfo db into one?

How about merging the configuration DB and the serviceInfo DB as follows and using the serviceInfo DB for different purpose such as collecting performance data(cpu, memory), version and so on?

Current

type Configuration struct {
	ID       string `json:"id"`
	Platform string `json:"platform"`
	ExecType string `json:"executionType"`
}
type ServiceInfo struct {
	ID       string   `json:"id"`
	Services []string `json:"services"`
}

Future

type Configuration struct {
	ID       string `json:"id"`
	Platform string `json:"platform"`
	ExecType string `json:"executionType"`
	Services []string `json:"services"`
}

link issue

orchestration link is not responding. Is there any possibility to get access to that link or any other solution.

[Consideration] Policy for platform support

Currently, we have supported 1) x86-64 Linux, 2) Rpi 2, and 3) Hikey960 if we adopt the recent PR #125 . It is now a good timing to consider how we setup a policy to support those existing platform and additional ones in the future release including Coconut.

Any thoughts on this?

Error response from daemon: can not get logs from container which is dead or marked for removal

Docker image is running fine on my system but when run it on edge orchestration it shows me the errors:
Screenshot from 2020-06-13 10-40-25
Running perfect on my system:
Screenshot from 2020-06-13 10-42-01

Here is the command which I am running:
curl -X POST "192.168.43.11:56001/api/v1/orchestration/services" -H "accept: application/json" -H "Content-Type: application/json" -d "{ "ServiceName": "age_detection", "ServiceInfo": [{ "ExecutionType": "container", "ExecCmd": [ "docker", "run", "-it","--rm","--device=/dev/video0","-e","DISPLAY=$DISPLAY ","-v","$XSOCK:$XSOCK","-v","$XAUTH:$XAUTH","-e","XAUTHORITY=$XAUTH","-e"," QT_X11_NO_MITSHM=1","animeshj123/age_det_image:02"]}], "StatusCallbackURI": "http://192.168.43.11:8888/api/v1/services/notification\"}"

[WIP] Centralized management of edge devices and services

Is your feature request related to a problem? Please describe.
Current method of selecting an edge device for service execution is decentralized and complex. The edge orchestration as edge node collects the final score result from other edge devices and offloads services to the high-score device.

Describe the solution you'd like
Only one edge orchestrator controls every other edge nodes and services.
The following features are required.

  • Collect resource information from other devices
  • Metrics for selecting primary/secondary
  • Forward service requests from requester to edge orchestrator
  • Notify the service offloading result to requester

[WIP] Encryption of SecureMgr

Originally posted by @tdrozdovsky in #131 (comment)

I personally think that sensitive data is returned by an access to passPhraseJWTPath. What do you think? @tdrozdovsky

log.Panicf("Failed to create passPhraseJWTPath %s: %s\n", passPhraseJWTPath, err)

This might be the same potential security risk as follow.

log.Println(logPrefix, "cannot create "+passPhraseJWTFilePath+": ", err)

Good point, I know and remember this security issue.

These only informs about a failed attempt to create the passPhraseJWTFilePath file.
But of course, storing such information in files (passPhrase, edge-orchestration.key, etc) is a security risk.
I think in the future this should be solved with secure storage or with access control system such as: SeLinux, SMACK, etc.

Thank you for reminder

LF Home Edge colloboration with EdgeX foundry

Edge X foundry:

Migration from java to Go code will be very helpful for Home Edge since LFHomeEdge is Go language based framework.
EdgeXfoundry planned to migrate the database from Mongo DB to Bolt DB (LF Home Edge uses Bolt DB).
Its great opportunity for LF Home Edge to closely work for bringing the data storage module from EdgeXFoundry to LF Home Edge.

Collaboration initiated with the conference call(Dec-4)9.30PM MAST with JimWhite(EdgexFoundry CoFounder).

Build Failure

Describe the bug
Build Failure

To Reproduce
Run the build script on the terminal.
$ ./build.sh

Log

----------------------------------------
 Create Executable binary from GoMain
----------------------------------------
go build -a -ldflags '-extldflags "-static" -X main.version= -X main.commitID=cd0ed5b -X main.buildTime=20200706.1730 -X main.buildTags=' -o /home/t25kim/contribution/edge-home-orchestration-go/GoMain/bin/edge-orchestration main || exit 1
# controller/servicemgr/executor/containerexecutor
src/controller/servicemgr/executor/containerexecutor/ce_docker.go:49:10: undefined: docker
src/controller/servicemgr/executor/containerexecutor/ce_docker.go:53:17: undefined: docker
src/controller/servicemgr/executor/containerexecutor/ce_docker.go:78:2: not enough arguments to return
src/controller/servicemgr/executor/containerexecutor/ce_docker.go:88:2: not enough arguments to return
Makefile:65: recipe for target 'build-binary' failed
make: *** [build-binary] Error 1

Test environment configuration (please complete the following information):

  • Firmware version: Ubuntu 16.04
  • Hardware: x86-64
  • Edge Orchestration Release: Baobab

Role based Access Control (security feature)

Is your feature request related to a problem? Please describe.
it is necessary to impose restrictions on access to resources for different users

Describe the solution you'd like
It would be great to implement the Role based Access Control

Modify the `binPath` for elasticity

Originally posted by @t25kim in #135 (comment)

image

What do you think of using edgeDir as below?

	binPath                = edgeDir + "/bin"

If this is more reasonable, the YamlFileName and the ConfigFileName are should be changed as well.

in kvm machine , it fail to pass init

Describe the bug
AS title shows

To Reproduce
Steps to reproduce the behavior:

set up docker env on kvm based VM , and get key , user id ready
then launch it

Expected behavior
get cpu info step pass ,

Screenshots
2020/03/10 08:54:04 cpu.go:100: resourceutil cpu info getting fail : open /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq: no such file or director

Test environment configuration (please complete the following information):
root@ubuntu18:/var/edge-orchestration/log# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
root@ubuntu18:/var/edge-orchestration/log# uname -a
Linux ubuntu18 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu18:/var/edge-orchestration/log# virt-what
virtualbox
kvm

  • Firmware version: (OS type and version, e.g., Ubuntu 14.04, etc.)
  • Hardware: (e.g., x86-64, ARM, CPU type, RAM size, etc.)
  • Toolchain: (e.g., Docker and Go versions)
  • Edge Orchestration Release: (e.g., Alpha, Baobab, Coconut)

Additional context
Add any other context about the problem here.

[Copyright statement] Revoke the PR #157 for the copyright statement correction

Regarding PR: #157 suggested by @tdrozdovsky, accepted by me (@MoonkiHong) and @suresh-lc:

We have deeply validated the copyright statement policy from open source communities especially projects in LF Edge, and our final decision from the evaluation is to revoke the PR #157 to the original one to state the latest released year for the contributed source codes. So I would like to report as the required item to be fixed as soon as possible. I am going to present a regarding PR shortly.

Additional ScoringImpl options

Not being a expert Go programmer, what is the expectation to extend the GetScore functionality with additional implementations, other than the current:

func calculateScore(ID string) float64 {
        cpuUsage, err := resourceIns.GetResource(resourceutil.CPUUsage)
        if err != nil {
                return 0.0
        }
        cpuCount, err := resourceIns.GetResource(resourceutil.CPUCount)
        if err != nil {
                return 0.0
        }
        cpuFreq, err := resourceIns.GetResource(resourceutil.CPUFreq)
        if err != nil {
                return 0.0
        }
        cpuScore := cpuScore(cpuUsage, cpuCount, cpuFreq)

        netBandwidth, err := resourceIns.GetResource(resourceutil.NetBandwidth)
        if err != nil {
                return 0.0
        }
        netScore := netScore(netBandwidth)

        resourceIns.SetDeviceID(ID)
        rtt, err := resourceIns.GetResource(resourceutil.NetRTT)
        if err != nil {
                return 0.0
        }
        renderingScore := renderingScore(rtt)

        return float64(netScore + (cpuScore / 2) + renderingScore)
}

Resource Information Update Period

Is your feature request related to a problem? Please describe.
Currently, the resource information update interval is 5 seconds and the accuracy is too low.
The following is the log of when the Edge Orchestration consistently received service offload requests. (see time and cpuUsage)

2020/09/01 08:00:38 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:1.7456359102244388 netBandwidth:500 rtt:0.000816483]
2020/09/01 08:00:38 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:1.7456359102244388 netBandwidth:500 rtt:0.000816483]
2020/09/01 08:00:39 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:1.7456359102244388 netBandwidth:500 rtt:0.000816483]
2020/09/01 08:00:41 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:1.7456359102244388 netBandwidth:500 rtt:0.000816483]
2020/09/01 08:00:42 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:1.7456359102244388 netBandwidth:500 rtt:0.000816483]
2020/09/01 08:00:43 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:1.7456359102244388 netBandwidth:250 rtt:0.000425345]
2020/09/01 08:00:44 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:13.316582914572864 netBandwidth:250 rtt:0.000425345]
2020/09/01 08:00:44 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:13.316582914572864 netBandwidth:250 rtt:0.000425345]
2020/09/01 08:00:45 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:13.316582914572864 netBandwidth:250 rtt:0.000425345]
2020/09/01 08:00:46 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:13.316582914572864 netBandwidth:250 rtt:0.000425345]
2020/09/01 08:00:47 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:13.316582914572864 netBandwidth:250 rtt:0.000425345]
2020/09/01 08:00:47 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:13.316582914572864 netBandwidth:250 rtt:0.000425345]
2020/09/01 08:00:48 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:13.316582914572864 netBandwidth:250 rtt:0.000452635]
2020/09/01 08:00:49 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:13.316582914572864 netBandwidth:166 rtt:0.000452635]
2020/09/01 08:00:49 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:19.6405648267009 netBandwidth:166 rtt:0.000452635]
2020/09/01 08:00:50 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:19.6405648267009 netBandwidth:166 rtt:0.000452635]
2020/09/01 08:00:51 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:19.6405648267009 netBandwidth:166 rtt:0.000452635]
2020/09/01 08:00:52 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:19.6405648267009 netBandwidth:166 rtt:0.000452635]
2020/09/01 08:00:52 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:19.6405648267009 netBandwidth:166 rtt:0.000452635]
2020/09/01 08:00:53 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:19.6405648267009 netBandwidth:166 rtt:0]
2020/09/01 08:00:54 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:19.6405648267009 netBandwidth:166 rtt:0]
2020/09/01 08:00:54 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:19.6405648267009 netBandwidth:166 rtt:0]
2020/09/01 08:00:56 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:15.83011583011583 netBandwidth:333 rtt:0]
2020/09/01 08:00:57 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:15.83011583011583 netBandwidth:333 rtt:0]
2020/09/01 08:00:58 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:15.83011583011583 netBandwidth:333 rtt:0.000969224]
2020/09/01 08:00:59 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:15.83011583011583 netBandwidth:333 rtt:0.000969224]
2020/09/01 08:00:59 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:15.83011583011583 netBandwidth:333 rtt:0.000969224]
2020/09/01 08:01:00 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:15.83011583011583 netBandwidth:333 rtt:0.000969224]
2020/09/01 08:01:01 orchestration_api.go:421: candidate resource : map[cpuCount:8 cpuFreq:4200 cpuUsage:15.83011583011583 netBandwidth:333 rtt:0.000969224]

Describe the solution you'd like
How about updating resource information while the service(container) is executing?

[DataStorage] Two types of Logger

DataStorage uses the package log and the logger of edgex.
As a result, there are separate log files for edgex and for edge-orchestration in /var/edge-orchestration/log/logmgr.log with different style.

level=INFO ts=2020-11-11T07:43:11.040198344Z app=datastorage source=init.go:138 msg="Check Metadata service's status ..."
level=INFO ts=2020-11-11T07:43:11.040289585Z app=datastorage source=init.go:138 msg="Check Data service's status ..."
level=ERROR ts=2020-11-11T07:43:11.042473566Z app=datastorage source=init.go:149 msg="Error getting ping: Get \"http://localhost:48081/api/v1/ping\": dial tcp 127.0.0.1:48081: connect: connection refused "
level=ERROR ts=2020-11-11T07:43:11.043495198Z app=datastorage source=init.go:149 msg="Error getting ping: Get \"http://localhost:48080/api/v1/ping\": dial tcp 127.0.0.1:48080: connect: connection refused "
2020/11/11 07:43:11 discovery.go:562: [deviceDetectionRoutine] edge-orchestration-3125da9e-1e9a-41aa-ac83-004725eb2d1e
2020/11/11 07:43:11 discovery.go:563: [deviceDetectionRoutine] confInfo    : ExecType(container), Platform(docker)
2020/11/11 07:43:11 discovery.go:564: [deviceDetectionRoutine] netInfo     : IPv4([]), RTT(0)
2020/11/11 07:43:11 discovery.go:565: [deviceDetectionRoutine] serviceInfo : Services([])
2020/11/11 07:43:11 discovery.go:566:

I think we should have same style at least.

[MNEDC] Refactoring in terms of Architecture

I think that the mnedcmgr function should be under the discoverymgr since they are similar with respect to functionality.
How about moving the mnedcmgr code under the discoverymgr?
Below should be changed.

  • Change package name from mnedcmgr to mnedc or something.
  • The discoverymgr should call StartMNEDCServer() and StartMNEDCClient().

Any comments are welcome!

Configure secure and mnedc mode while running the image

Is your feature request related to a problem? Please describe.
Currently, the edge-orchestration sets the secure and mnedc options while building the project.

Describe the solution you'd like
Read the environment values if the edge-orchestration runs with SECURE and MNEDC options like below.

docker run -it -d --privileged ... -e SECURE=true -e MNEDC=server edge-orchestration:coconut

Edge Orchestrator container fails to shutdown gracefully

After Edge Orchestrator container is stopped by 'docker stop edge-orchestrator' command, Edge Orchestrator remains active, listening to port 56002; this causes any further attempt to restart container to fail. Current workaround is manually killing process and cleaning db folder.
log.txt

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.