dyrector-io / dyrectorio Goto Github PK

As a user, I'd like to be able to schedule the deployment of a product when deployment time is inconvenient for manual initialization.

Our docker container builder has a small abstraction misfit, it does a bit more than it should.

A builder builds.

This is issue is a good one for someone who wants to see the Docker Client Golang SDK more closely.

TODO:

• Keep all the private, With, Without, Create method on builder,
• Move Start & StartWait, GetContainerID, GetNetworkID to a new type: DockerContainer
• copy/move all the necessary struct members into the new DockerContainer
• The container builder should have Create as its last function returning (DockerContainer,error)
• Consumer of this API will start the container using the methods on DockerContainer
• Modify the interface accordingly
• Modify other pieces of code using this checking error every it is used
• Fix the test you've just broken (:
• Add the same to tests with the builder (sufficient to add to one test):

// this is a forced interface implementation check
var _ containerbuilder.ExecBuilder = (*containerbuilder.DockerExecBuilder)(nil)

If in doubt don't hesitate to reach us on Discord.

As the title says.

Request discussion coming from: #559

In agent/internal/config/config.go we store our configuration defaults. In line 28 you can find NodeID which is an UUIDv4 string, but we don't validate it. Please create a validation for this particular data.

If you are ready with the task, please remove the // TODO line from the file.

More info:
A universally unique identifier (UUID) is an identifier standard used in software construction. The meaning of each bit is defined by any of several variants. It is a 128-bit value. Currently, UUID's are as specified in RFC4122. https://www.rfc-editor.org/rfc/rfc4122#section-4.1.7

The original task is split into two, this is the first half.
It might seem difficult, it covers a lot, but the issue is explicit.

You are going to need a cluster to develop this feature & test it, that can be done locally: kind or minikube.

Issue:
Right now crane (that is the in-house name of our k8s agent) deployment is depending on having a persistent volume attach to store its private key that is used for secret encryption. Attaching a PVC imposes heavy restrictions and possible configuration problems possibly leading to inconsistent behavior in different environments, while using kubernetes native objects can already save the day.

Folder of k8s manifests:
https://github.com/dyrector-io/dyrectorio/tree/develop/agent/manifest/kubernetes

Currently the private key that is generated is written to a file.

TODO:

1. Extend crane with a CLI command as an option to run crane init, if nothing is provided start the program normally
2. Add new makefile (agent/Makefile) target: go-crane-init

• it starts the init process, then exit

3. The init function can use the generateKey function to generate the key (internal/config/secret.go) - expose it
4. Save the output to a k8s opaque secret
5. Reorganize configurations

• Move the current secret behavior (type ConfigFromFile) into pkg/dagent/config/secret.go package and into dagent's own config type
• Create new type in a similar way for achieving similar effects using secrets into crane ConfigFromSecret pkg/crane/config/secret.go
• The new type setter in crane expects a string (namespace), from env variable: NAMESPACE - also new.
• In the setter
  • If namespace not provided, attempt to use the dyrectorio namespace
  • GetSecretClient function can be reused from k8s package - expose it
  • Load secret value

Hint:
Avoid panics, make sure to run lint, unit tests are welcome.

The main goal is to create an environment that facilitates easy testing of the application for newcomers. Currently, this process is quite cumbersome. Please utilize the CLI for this purpose, and it will require everyone to adjust their workflow.

first step // command: make test

Kubernetes pod status: CrashLoopBackOffice

Crux status: Dead

Not correct.

Create proper Kubernetes statuses for Kubernetes deloyments.

From meeting note: Need to supply the correct Kubernetes state to the user. We should provide tips and links based on container status.

I see that to run this I need a Go compiler, and to compile the source for now. But could you provide a docker container to get the server / web-ui up and running, then use that to orchestrate the other servers / nodes?

I think this reduces the barrier to entry a bit. I love the ideas and concepts shown. This looks really amazing.

Best! And thank you for making it open source!

At present we have Discord, Slack & Team integration with webhooks. Please implement Mattermost Webhooks next to the existing ones.

You have to work around the web/crux/src/app/notification for the backend and web/crux-ui/src/components/notification for the user interface. Furthermore, you have to extend the protobuff file with the new NotificationType.

Add regexp to check the validation of the webhook URL.

Mattermost Webhook Documentation: https://developers.mattermost.com/integrate/webhooks/incoming/

In the code base at present, we are using 3 versions of catching errors. Versions: catch (e), catch (err), catch (error), please refactor the code and use only one of them.

Affected folders:

• web/crux - NextJS
• web/crux-ui - NestJS

We need to annotate the controller with the possible HTTP statuses. If there are numerous identical errors, we should create a custom decorator that gathers them all together and use that instead.

We (work in progress #243) implemented templates in the platform so you can create products from predefined templates eg.: Strapi. We are using the Google Microservice Example project as a demo to show off the power of our platform. You can currently seed the example project via Prisma seeder, but you must add your Id and your team's Id. We would like to add this as a template.

The task is to add a similar file like web/crux/templates/strapi.json but with defined in the web/crux/prisma/seeds/data/demo folder, to create the ability for users to add a Google Microservice Example. You don't have to delete the seeder in this task, just implement these data points in a template way.

Related feature: #243

Idea: Webhook implementation

1. String(hook) generation in UI, API implementation. (latest)
2. Curl
3. Github Action
4. Dagger?

Event:

• rolling version deployment
• other events

Definition of done: works in Azure DevOps

At present we have Discord, Slack & Team integration with webhooks. Please implement Rocket.Chat Webhooks next to the existing ones.

You have to work around the web/crux/src/app/notification for the backend and web/crux-ui/src/components/notification for the user interface. Furthermore, you have to extend the protobuff file with the new NotificationType.

Add regexp to check the validation of the webhook URL.

Rocket.Chat Webhook Documentation: https://docs.rocket.chat/guides/administration/admin-panel/integrations

We are using grpc-node and they don't have a precompiled ARM binary. So if you are trying to install the crux-ui component from the code repository, you have to add a --target_arch=x64 flag. Please create a new install script eg.: install:x64 with the mentioned flag.

package.json location: web/crux-ui/package.json

Information about the issue and the solution: grpc/grpc-node#1405

Users want to add images from JFrog Container Registry, so we have to support the integration. Before we save the registries we always check the given credentials, so please do not miss it. You have to extend the Prisma schema RegistryTypeEnum with jfrog.

You can find existing validations in web/crux/src/app/registry/guards/registry.auth.validation.guard.ts - we are using Nest.js guards for this.

JFrog Container Registry REST API: https://www.jfrog.com/confluence/display/JCR6X/JFrog+Container+Registry+REST+API

Hi,

I managed to get dyrectorio working using the docker-compose file. However, the script generated when adding a node seems to have a typo:

  if [ -z ${CRI_EXECUTABLE:-} ]; then
    if [ -z "$(which docker)" ]; then
      if [ -z "$(which podman)" ]; then
        echo "Docker is required, make sure it is installed and available in PATH!"
        exit 1
      else
        CRI_EXECUTABLE="podman"
      fi
    fi
    CRI_EXECUTABLE="docker"
  fi

There is an else missing preventing from using podman. It should be:

    else
      CRI_EXECUTABLE="docker"
    fi

In agent/internal/config/config.go we store our configuration defaults. We use GRPC_TOKEN which is a JWT Token, but right now it's a simple string. Please create a validation for it to check if it's a JWT token.

Add intermediate pre-condition to make targets (root Makefile), as make dependencies to relevant code parts.

For golang code check go-dependencies, go-dev dependencies – separate.

For nodejs, the same.

These targets execute and if dependencies are missing or there is a version mismatch it bails with an exit code and an error.

Dependencies can be defined like so:

.PHONY target
target: dependency1 dependency2
  step1
  step2

i would like to improve the readme by adding contributors profile in readme
assign this task to me.i can do it flawlessly

Steps to reproduce the behavior:

1. Add a node to your workspace
2. Create a new Product (simple product type is good for now)
3. Click the 'Add deployment'
4. Select your Node
5. Fill up the input with - and you will see the error.

Showing an error in the attached picture.

At present we have Discord, Slack & Team integration with webhooks. Please implement HipChat Webhooks next to the existing ones.

You have to work around the web/crux/src/app/notification for the backend and web/crux-ui/src/components/notification for the user interface. Furthermore, you have to extend the protobuff file with the new NotificationType.

Add regexp to check the validation of the webhook URL.

HipChat Webhook Documentation: https://developer.atlassian.com/server/hipchat/webhooks/

Creative issue: make the linked function better.

We want ouput to go into: general logs or CLI output into user-facing terminal window - it has to be readable & understandable.

Issues:

• image layer is not useful information for a human
• printing each image layer's progress in percentage format without knowing how many layers are there in total is still not real information about progress

This is issue is open for suggestions.

Plus addressing is a widely used thing. It works in our product but in the Account verification screen, the + disappears from the email.

Reproduction:

• Register a user with an email like [email protected]
• In the next screen (Account verification) the + signal disappears from the input field - the email works but its still confusing for users

Or you can easily check the production with the following query parameter: app.dyrectorio.com/auth/[email protected]

Tested: Production 0.4.1

What is plus addressing in emails? Plus addressing and subdomain addressing

Users want to add images from ECR, so we have to support Amazon Elastic Container Registry integration. Before we save the registries we always check the given credentials, so please do not miss it. You have to extend the Prisma schema RegistryTypeEnum with amazon.

You can find existing validations in web/crux/src/app/registry/guards/registry.auth.validation.guard.ts - we are using Nest.js guards for this.

Amazon ECR API: https://docs.aws.amazon.com/AmazonECR/latest/APIReference/Welcome.html

This issue is caused by: #248

Currently when we create a container using DockerBuilder we can't define additional hosts, but we should be able to.
Implement a function named WithHosts where we expect a string slice/array as Docker SDK does, and we pass this to it. If we leave the function empty, it should use the current hardcoded value as default. You can find more information about this behaviour in docker cli as the --add-host parameter or in docker-compose the extra_hosts: directive in the yaml file.

If you have any question about this issue feel free to reach out!

Right now /agent service code coverage is around 8.67% - at events such as Hacktoberfest, it's a good first issue to write unit tests around the code base. You can add tests to /cli if you're a black belt unit test writer.

We know unit test writing isn't the most exciting task, so we'd like to motivate our contributors. So if you add more than 10% coverage in the golang side of the repository, we'll send you an awesome dyrectorio swag cap.

You can find codecov agent/ coverage: https://app.codecov.io/gh/dyrector-io/dyrectorio/blob/develop/agent/pkg/builder/container/types.go

For a while we had other solution working, but it went obsolete and then we are with stock go command now, but live reload is a nice comfort feature.

As a suggestion, I've found this tool to solve the same problem: https://github.com/cosmtrek/air

Modify agent's Makefile. The 2 build targets go-crane & go-dagent are to use the tool above. 👌

The issue is done if running the targets above and modifying code results in live-reloading.

#443 is blocked because templating lacks some features, that are necessary to make dyrector.io working correctly when deployed via templates.

Hey,
I am deploying dyrectorio with docker-compose, everything is going well, but when registering I do not receive an email.

I have put an address in the environment variables.

FROM_EMAIL=[email protected]
SMTP_URI=smtp://172.17.0.1:25/?skip_ssl_verify=true&legacy_ssl=false&smtp_ssl_enabled=false&disable_starttls=true

but in the logs, I repeatedly receive an error from kratos:
msg=Unable to send email using SMTP connection. audience=application error=map[message:gomail: could not send email 1: 554 5.7.1 [email protected]: Recipient address rejected: Access denied] message_from=[email protected] service_name=Ory Kratos service_version=v0 .11.0 smtp_server=172.17.0.1:25 smtp_ssl_enabled=false

I use postfix, do you have any idea?
Thanks a lot!!

I can make it more interactive , fix styling issues using tailwind .

CLI configures environments according to the input parameters.

As of right now the --disable-crux and the --disable-crux-ui flags are not handled correctly, in case of these scenarios those components are running on the host machine.

Expected behavior:

• only create the dynamic_conf.yml file if there is at least one disabled component
• if only one is disabled the other should still function normally & get proxied using labels

Hint: changes affect (mostly) the golang/pkg/cli package.

The original task is split into two, this is the second half.
The other part: #244

You are going to need a cluster to develop this feature & test it, that can be done locally: kind or minikube.

Folder of k8s manifests:
https://github.com/dyrector-io/dyrectorio/tree/develop/agent/manifest/kubernetes

TODO:

• Extend the deployment manifest with an init-container, using the same image, receiving the config map, but having an extra init argument
• Test

Using the cli, you can start the application fairly easy and can test if it is working.

Expected behavior:

• Setup a k8s node in dyrectorio
• apply your local version of the updated manifest using kubectl
• it should start, connect normally
• create a simple product
• creating a deployment targeting your node you can provide secrets

In the web/crux backend, we use the Prisma ORM and have created a schema with an optional updatedAt field annotated with the @updatedAt schema. This annotation automatically stores the time when a record was last updated, making the optional field unnecessary and somewhat illogical. We should remove the optional field from schema.prisma and investigate if it occurs anywhere else in the web/crux codebase.

For more information on the @updatedAt schema, see the Prisma documentation: https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference#updatedat."

Affected directory and scope: web/crux

In the first version, we would like to create a simple role. A user who can only view or Cloud Engineer who can only add nodes for the team.

Next step: We have to investigate the OAuthkeeper (by Ory) tool, which is doing this stuff.

At present, if you setup dyrector.io stack with CLI, you'll get a port binding error if you run services on one of these ports (5000, 5001, 3000, 5432, 4433, 4434, 5433, 4436)

For example: Error response from daemon: driver failed programming external connectivity on endpoint dyrectorio-stack_crux-ui (39694...): Bind for 0.0.0.0:3000 failed: port is already allocated

We have to check if the given ports are free, if not generate a new port number or ask the user to give another to run the application. Definition of done: The user can easily set up the whole stack with CLI without port binding errors.

Ports can be found in the SettingsFile.Options struct (int type), after initialization process
currently: https://github.com/dyrector-io/dyrectorio/blob/develop/cli/config_file.go#L158

Workdir: /cli

Instead of the default log package, there is a growing need for more advanced logging capabilities.

The library zerolog seems nice, fast & clean.
Note the features:

• multi-level
• stacktrace

TODO:

• add new library zerolog
• replace all log imports with zerolog imports
• configure it as https://github.com/rs/zerolog#pretty-logging
• everything can be logged as Info, as changing these will be a separate issue

We ignored this since it may not have a significant impact, but it would be nice to have and not too complicated to implement.

Compared to Kubernetes, resource requests don't really make sense for us because we don't need to instruct the scheduler since it doesn't exist. However, Docker does have CPU and memory limits.

Affected scope:

• golang: Add CPU and memory limits to the builder, dagent, and proto.
• crux: Display the options on the UI.

Please provide a test case in golang as well.

Additional information: Our platform backend is built with Node.js called crux, and the UI built with React is called crux-ui.

Based on our container builder

TODO: Add a new builder named ExecBuilder, into the same package into a file exec_builder.go.

The builder should be created new a NewExecBuilder(containerID string) function.
Use the builder to set any given parameter and .Start() should return the output / result of the executed command.

Use the API as a reference.
https://docs.docker.com/engine/api/v1.41/#tag/Exec/operation/ContainerExec

Golang package:
https://pkg.go.dev/github.com/docker/docker/client#Client.ContainerExecCreate

As the title says upgrade the Prisma package in web/crux service.

Users want to add images from ACR, so we have to support Azure Container Registry integration. Before we save the registries we always check the given credentials, so please do not miss it. You have to extend the Prisma schema RegistryTypeEnum with azure.

You can find existing validations in web/crux/src/app/registry/guards/registry.auth.validation.guard.ts - we are using Nest.js guards for this.

ACR API: https://learn.microsoft.com/en-us/rest/api/containerregistry

At present the Google Demo example seed not working. Please extend the seeder with the necessary code. You have to edit only the web/crux/prisma/seeds/demo-seeder.ts file. When you are ready please rename the file to microservice-seeder.ts.

Steps to reproduce:

1. Setup dyrector.io platform
2. Create a User and log in
3. After the first login create a Team
4. Check the UsersOnTeams table for userId and teamId
5. Run the Google Microservice example seeder in web/crux folder with the following command: npm run seed:microservice

PrismaClientValidationError: 
Invalid `prisma.image.create()` invocation in
/Users/orbanl/Work/dyrector.io/repos/dyrectorio/web/crux/prisma/seeds/demo-seeder.ts:55:40

You can install dyrectorio agent whether you are using Kubernetes or Docker. In a Docker way, we can't assume you are using Docker on Linux, because there are other container runtimes like Podman or other platforms like Windows.

At present you can pass this options in hostDockerSockPath at agent/pkg/dagent/config/config.go:16, but the parsing and checks are missing. If the env is missing please add a default with cleanenv which can be computed at runtime since we currently assume a static default, other hand If the user is provided some path please check the socket.

If the value is empty you can calculate it from DOCKER_HOST variable using URL parsing from the official go package, if the DOCKER_HOST variable is empty as well please use the Docker SDK's default (platform-specific) socket value.

Add transactable app to the Azure Marketplace.

https://github.com/dyrector-io/dyrectorio/tree/develop/agent/pkg/builder/container

We have a general purpose container builder.
One can already set CMD, ENTRYPOINT (and many other things), but SHELL is not on option list yet.

TODO: extend the contianer builder interface and the implementation with a new WithShell function, similar to the functions already present in the builder. Use the reference linked below.

https://docs.docker.com/engine/api/v1.41/#tag/Container/operation/ContainerCreate

The issue is complete if the WithShell function is there, and there are tests verifying its basic use case.
Note: for a unit test it is sufficient to check if the value is really set, for an integration test one has to inspect the started container if its shell is changed.

Current about is: "Open-source IDP for developers and non-specialists to manage multi-instance deployments, microservices and configurations in Kubernetes & Docker."
I'm a staff-level engineer, but don't know the resolution of "IDP". (best guess: "Integrated Deployment Platform"?) Google didn't help me, Wikipedia didn't help me, either.
If we strongly believe this is a common abbreviation, I suggest we add it to the Wikipedia page.
Otherwise, I suggest we clearly introduce the term at its first use on the page.

I'd create a PR for this, but:

1. I don't know the resolution of the term
2. I don't think I have permission to set the "About" part of the GitHub project, so this would need contribution from an admin, anyway.

Any thoughts?

We have a few updateAt columns in our database schema. To avoid boilerplate codes Prisma has a sophisticated solution for this particular problem. They made an @updateAt signature: automatically stores the time when a record was last updated. If you do not supply a time yourself, the Prisma Client will automatically set the value for fields with this attribute.

Prisma Reference documentation: https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference#updatedat

Hi :)

I tried to deploy dyrector-io on the following system:

podman 3.4.2
go version go1.19.5 linux/amd64
Debian 11

using the following command

DOCKER_HOST=unix:///var/run/podman/podman.sock go/bin/dyo --disable-podman-checks up

(without --disable-podman-checks it fails:
10:43AM FTL Podman command execution error error="exit status 125"
though even with --debug I can't get more verbose output than that)

The pods start but postgres fails with:

initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Read-only file system

An inspect gives the following:

        "Mounts": [
            {
                "Type": "volume",
                "Name": "dyrectorio-stack_kratos-postgres-data",
                "Source": "/var/lib/containers/storage/volumes/dyrectorio-stack_kratos-postgres-data/_data",
                "Destination": "/var/lib/postgresql/data",
                "Driver": "local",
                "Mode": "",
                "Options": [
                    "nosuid",
                    "nodev",
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            }
        ],

Could it occurs because of "RW": false ?