lf-edge / adam Goto Github PK

Seems, that we need to implement apps endpoint to view logs of apps.

Recently, we have seen many issues where EVE didn't properly handle a change in the controller signing certificate. It would be therefore beneficial to prepare an eden test in which the signing certificate used by Adam is changed after device is onboarded. This can be combined with poor network connectivity (which can be modeled with eden), etc. (conditions that triggered aforementioned issues).

However, even though there is a POST method declared for /certs (to presumably change the controller certificate), it is not implemented. Handler apiv2.certs does not check the request method type and always behaves as GET method, returning the set of installed certificates.

Edit: As pointed out by @giggsoff, instead of using the mentioned /certs endpoint, which is actually used between EVE and Adam, we should add a new POST method under the Admin handler, that could be then used from Eden CLI to update the controller certificate in runtime.

CC @eriknordmark @rouming

EVE has transitioned to the V2 API by default. Adam knows only about V1 still. This means we have to force all instances of EVE that talk to Adam to have Force-API-V1 in the config folder -- otherwise EVE won't be able to talk to Adam at all.

There's work in EVE going on to publish V2 API spec updates -- once that is done we'll use the spec to update Adam to V2 API.

The drivers all are supposed to store json format, but the newlog endpoint is storing the wire format.

For the newlog marshaling, which should be changed, see here where it calls proto.Marshal.

For the other marshaling, e.g. logs, see here where it calls protojson.Marshal

I was having issues running Adam in Docker on a t4g instance in AWS. After running around in circles trying to figure out what I was doing wrong I realized after looking at the Docker tags that latest was released 5 months ago.

Looking at your GitHub Workflows I see you don't tag any image as latest. Would it make sense to add the tag under the publish workflow?

Alternatively, you could call out in the README one should explicitly define a tag. Currently all of the examples just use lfedge/adam.

I'd be happy to make a pull request either way.

Now we can see onboarded devices. Would like to extend to: list all devices and their states -
onboarded
trying to onboard
detected
failed to onboard - with a reason of failure

When I go to the page I'd like to see at least something like:

Hello. Welcome to ADAM. Here are the Endpoints:

The following are the admin endpoints:

GET /onboard - list all onboard certificates
GET /onboard/{cn} - get a specific onboard certificate
POST /onboard - upload a new onboarding certificate
DELETE /onboard - clear all onboarding certificates
DELETE /onboard/{cn} - delete a specific onboarding certificate
GET /device - list all devices
GET /device/{uuid} - get details of one device
GET /device/{uuid}/config - get config for one device
PUT /device/{uuid}/config - update config for one device
GET /device/{uuid}/logs - get all known logs for one device; set header X-Stream=true to stream all new logs instead
GET /device/{uuid}/info - get all known info messages for one device; set header X-Stream=true to stream all new info instead
POST /device - create a new device
DELETE /device - delete all devices
DELETE /device/{uuid} - delete one specific device

Even better if I see that I have some EVE connected to ADAM

With the new version (0.0.57) I see in logs:

2020/09/15 09:28:42 CN=onboard,O=Zededa\, Inc,L=Santa Clara,ST=California,C=US requested /api/v1/edgedevice/info
2020/09/15 09:28:42 http: panic serving 192.168.0.106:45378: runtime error: invalid memory address or nil pointer dereference
goroutine 188 [running]:
net/http.(*conn).serve.func1(0xc0001f7400)
        /usr/local/go/src/net/http/server.go:1769 +0x139
panic(0xa80180, 0x112afd0)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/lf-edge/adam/pkg/driver/redis.(*DeviceManager).WriteRequest(0xc00010f260, 0xbfd04032904986ed, 0x201adaf06c, 0x1144840, 0x7443d375cd2a7abc, 0xd9659d255950c3bc, 0xc00035f620, 0x13, 0x0, 0x0, ...)
        /adam/src/pkg/driver/redis/device_manager_redis.go:500 +0x35b
github.com/lf-edge/adam/pkg/server.(*apiHandler).recordClient(0xc00000de60, 0xc0003ed1a0, 0xc0002b3800)
        /adam/src/pkg/server/apiHandler.go:37 +0x20b
github.com/lf-edge/adam/pkg/server.(*apiHandler).info(0xc00000de60, 0xc2fcc0, 0xc0001d8620, 0xc0002b3800)
        /adam/src/pkg/server/apiHandler.go:208 +0x176
net/http.HandlerFunc.ServeHTTP(0xc0001dc750, 0xc2fcc0, 0xc0001d8620, 0xc0002b3800)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/lf-edge/adam/pkg/server.logRequest.func1(0xc2fcc0, 0xc0001d8620, 0xc0002b3800)
        /adam/src/pkg/server/server.go:144 +0x1db
net/http.HandlerFunc.ServeHTTP(0xc0005a4d60, 0xc2fcc0, 0xc0001d8620, 0xc0002b3800)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/lf-edge/adam/pkg/server.ensureMTLS.func1(0xc2fcc0, 0xc0001d8620, 0xc0002b3800)
        /adam/src/pkg/server/server.go:135 +0xb6
net/http.HandlerFunc.ServeHTTP(0xc0005a4d80, 0xc2fcc0, 0xc0001d8620, 0xc0002b3800)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0000f6900, 0xc2fcc0, 0xc0001d8620, 0xc0002b3600)
        /go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0xe3
net/http.serverHandler.ServeHTTP(0xc0001c6680, 0xc2fcc0, 0xc0001d8620, 0xc0002b3600)
        /usr/local/go/src/net/http/server.go:2774 +0xa8
net/http.(*conn).serve(0xc0001f7400, 0xc31880, 0xc0001d5ac0)
        /usr/local/go/src/net/http/server.go:1878 +0x851
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2884 +0x2f4

After ~ a week or so (or after ~ 500 MB of logs) - Adam + Redis + Eve stop working:
Certificate in Redis is lost. Rebooting of Eve / Eden / Redis/ Adam doesn't help.
This is due to the current redis implementation and logs filling redis memory.

It will be cool to be able to rebuild and clean all images and local builds.

Explicit import/export and improvement of separate certificate storage
Make an admin endpoint to be able to export/import node configs with certificates.
Thus it will allow to replace Adam without any changes on Eve. (if domain/ip is the same)

The logs currently are one file per logbundle, which causes 2 issues:

1. It creates a lot of small files very quickly.
2. The files can be sorted using tools, but the natural OS sort is not necessarily in order. The sheer number of files makes listing them all and sorting impractical.
3. It is hard to create a log rotation

The proposal is to change to a more modern log structure. Logfiles start on a given date and time. When a milestone is reached - either time since opening or file size - a new logfile is created.

When the entire log directory hits a given milestone - total size or time since first file - files are pruned, oldest first.

Add integration tests to Adam

Currently, when I try to post a config for a given UUID the config itself is still expected to have:

"id":{"uuid":"XXXXX","version":"YYYY"}

This is pretty annoying (since I now need to request the config first, make sure that I provide the correct XXXX value and also make sure to bump YYYY value). I suggest that we simply ignore that part of JSON on set and make Adam manage it internally.

The Adam server uses /run/adam for runtime files, rather than /adam/run/adam as described in the documentation. Elsewhere in the documentation a relative path ./run/adam without specifying what the PWD is inside the docker container, leading one to assume it is /adam rather than / which is currently is.

As per conversation in lf-edge/eve#1284 we need to update Adam ASAP

It will be great if a user can use the nice web-API via swagger-UI for the ADAM controller. Makes it much easier to use, especially remotely. Go-Swagger module https://github.com/go-swagger/go-swagger

Description

There is link to API-compliant Controller in README.md, but I can't access to API-compliant Controller
I get 404 Error - Page not found!

Allow a method that:

• Takes a config of an existing onboarded node

• Expands it to some other existing onboarded node, saving unique names and replacing workloads / networks /ports with the clone info

The docker-compose.yml file contains the option "--self-cert". If I issue "docker compose up" I get:

robert@arch ~/F/Z/r/r/adam> docker compose up                                                                                         
[+] Running 1/1                                                                                                                       
 ⠿ Container adam-adam-1  Recreated                                                                                              0.2s
Attaching to adam-adam-1                                                                                                              
adam-adam-1  | Error: unknown flag: --self-cert  
...

Also adam seems to return exit code 0 in this case which is probably not optimal:

...
adam-adam-1  |       --server-key string       path to server key (default "run/adam/server-key.pem")
adam-adam-1  | 
adam-adam-1 exited with code 0

Regards,
Robert

The command:
docker run lfedge/adam server
doesn't work correctly as the docker image does not have certificates. It also doesn't take them from the local build generated by adam generate server if I use make image-local

error loading server cert run/adam/server.pem and server key run/adam/server-key.pem: open run/adam/server.pem: no such file or directory