Giter Club home page Giter Club logo

horizon's Introduction

Horizon official website

This website is built using Docusaurus 2, a modern static website generator.

Installation

$ yarn

Local Development

$ yarn start

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

Build

$ yarn build

This command generates static content into the build directory and can be served using any static contents hosting service.

Deployment

Using SSH:

$ USE_SSH=true yarn deploy

Not using SSH:

$ GIT_USER=<Your GitHub username> yarn deploy

If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the gh-pages branch.

Translation

https://crowdin.com/project/horizon-cd/zh-CN

horizon's People

Contributors

arvintian avatar bysph avatar cubxxw avatar doing-cr7 avatar feizekai avatar iamyeka avatar kilosonc avatar machine3 avatar mouuii avatar tanoya avatar work-jlsun avatar xuzhu-591 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

horizon's Issues

gitlab status error

One click installation by helm,gitlab can not start for long time:
horizoncd horizon-gitlab-0 0/1 ContainerCreating 0 28m
my k8s version v1.21.14+k3s1
root@horizon-gitlab-0:/# gitlab-ctl status
run: gitaly: (pid 3916) 195s; run: log: (pid 963) 1567s
run: logrotate: (pid 3941) 194s; run: log: (pid 817) 1592s
run: postgresql: (pid 3972) 194s; run: log: (pid 1178) 1535s
run: redis: (pid 3983) 193s; run: log: (pid 878) 1579s
run: sshd: (pid 3992) 193s; run: log: (pid 108) 1669s

helm install error?

helm install horizon horizon/horizon -n horizoncd --version 2.0.1 --create-namespace
coalesce.go:223: warning: destination for horizon.tektonci-resources.auth.dockerConfigJson is a table. Ignoring non-table value ({})
coalesce.go:223: warning: destination for horizon.tektonci-resources.auth.dockerConfigJson is a table. Ignoring non-table value ({})
Error: INSTALLATION FAILED: failed to install CRD crds/clustertriggerbindings-crd.yaml: resource mapping not found for name: "clustertriggerbindings.triggers.tekton.dev" namespace: "" from "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first

my k8s version is v1.25.6+k3s1

Add a trigger for CD step

Event Description Status
Trigger Add triggers to support other systems calling for builds, such as Github Actions Pending

[INSTALLATION]

Installation Type

script (minimal)

Installation Result

success

Installation Duration

less than 2min

Environment

- OS: macOS  Monterey
- CPU: Intel Core i7
- Memory: 32GB
- Docker: colima start --cpu 4 --memory 8

Suggestions & Bugs

~~~~~>curl -sfL https://raw.githubusercontent.com/horizoncd/horizon/main/scripts/install.sh | bash -s -- -k  -cn
The binary docker is installed
The binary kind is installed
The binary kubectl is installed
The binary helm is installed
Creating kind cluster
Creating cluster "horizon" ...
 ✓ Ensuring node image (kindest/node:v1.19.16) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-horizon"
You can now use your cluster with:

kubectl cluster-info --context kind-horizon

Have a nice day! 👋
Switched to context "kind-horizon".
Installing ingress-nginx
"ingress-nginx" already exists with the same configuration, skipping
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/tomsun/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/tomsun/.kube/config
Waiting for ingress-nginx to be ready
^[pod/my-ingress-nginx-controller-5bc646f87d-chxr4 condition met
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/tomsun/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/tomsun/.kube/config
"horizon" already exists with the same configuration, skipping
Update helm repo
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/tomsun/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/tomsun/.kube/config
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "horizon-template" chart repository (https://harbor.yf-online.service.163.org/chartrepo/horizon-template):
	Get "https://harbor.yf-online.service.163.org/chartrepo/horizon-template/index.yaml": dial tcp: lookup harbor.yf-online.service.163.org on 10.246.101.33:53: no such host
...Unable to get an update from the "cloudnative" chart repository (https://harbor.yf-online.service.163.org/chartrepo/cloudnative):
	Get "https://harbor.yf-online.service.163.org/chartrepo/cloudnative/index.yaml": dial tcp: lookup harbor.yf-online.service.163.org on 10.246.101.33:53: no such host
...Successfully got an update from the "horizon" chart repository
...Successfully got an update from the "r" chart repository
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "eks" chart repository
...Successfully got an update from the "kubevela" chart repository
...Successfully got an update from the "argo" chart repository
...Successfully got an update from the "nginx-stable" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "bitnami" chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "goharbor" chart repository
...Successfully got an update from the "harbor" chart repository
Update Complete. ⎈Happy Helming!⎈
Installing horizon
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/tomsun/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/tomsun/.kube/config
coalesce.go:223: warning: destination for horizon.tektonci-resources.auth.dockerConfigJson is a table. Ignoring non-table value ({})
coalesce.go:223: warning: destination for horizon.tektonci-resources.auth.dockerConfigJson is a table. Ignoring non-table value ({})
coalesce.go:223: warning: destination for horizon.tektonci-resources.auth.dockerConfigJson is a table. Ignoring non-table value ({})
Applying init job to k8s
configmap/horizon-init created
job.batch/horizon-init created
job.batch/horizon-init condition met
Horizon is installed successfully!
Please access the Horizon UI at http://horizon.h8r.site:80
~~~ ~>
  1. [suggestions]: print out the initial login username and password
  2. [bug]: invalid schema when create application
  3. [bug]: Unkown column 'is_favorite' in 'field list' ClusterInDB intert DB
  4. [bug?]: api.github.com 403
  5. [suggestions]: doc local_install and production_install (include all kinds of cloud)
  6. [suggestions]: if in china(can not access docker hub efficiency ) please give an obvious single cmd
  7. [suggestions]: and if u use minimal install, print to show use their gitops cd repo are store in github.com/horizoncd

curl -sfL https://raw.githubusercontent.com/horizoncd/horizon/main/scripts/install.sh | bash -s -- -k -cn

Local debugging

Intro

The main development of Horizon is concentrated on horizon-core and horizon-web. The fallowing method is proposed to facilitate our coding and debugging:

Requirements

Debug process

Horizon-core

Horizon-core is a web server built with gin framework. Due to its complex interaction with other components, we could use nocalhost tool to connect to kind locally for debugging.

  1. Download nocalhost extension

  2. Open nocalhost's panel, click the "+" button, add the kubeconfig location, and select context and namespace.
    image

  3. Start DevMode: select kind -> horizoncd -> horizon ->Deployments -> horizon-core. Start DevMode in the right-click menu.
    image

  4. Remote debugging: click Remote Debug in the right-click menu, and wait for the service to start.

  5. Debug your code!

Horizon-web

  1. change proxy to forward requests to horizon-core deployed on your k8s.

    // config/proxy.ts
    export default {
      dev: {
        '/apis/': {
          // target: 'http://localhost:8080',
          target: 'http://horizon.h8r.site'
          changeOrigin: true,
          pathRewrite: { '^': '' },
        },
      },
    };
  2. start the project and visit http://localhost:8000 for debugging.

    # prepare
    yarn
    # start
    yarn start

no owners file?

What would you like to be added:
是否有owners 文件,方便知道谁对哪个模块比较清楚
Why is this needed:

Put gitOpsRepoDefaultBranch flag into config file

It is inconvenient for development using nocalhost to treat option gitOpsRepoDefaultBranch as a flag, because developers need to additionally modify the startup and debugging script when the default branch of the gitops repo he specifies changes.

#! /bin/sh

# debug.sh
dlv --headless --log --listen :9009 --api-version 2 --accept-multiclient debug core/main.go -- -config=/home/appops/config -roles=/home/appops/roles -environment=production -scopes=/home/appops/scopes -buildjsonschema=/home/appops/buildjsonschema -builduischema=/home/appops/builduischema -gitOpsRepoDefaultBranch=main

What would you like to be added:
It could be a good proposal to put it in config file, such as:

// GitopsRepoConfig gitops repo config
type GitopsRepoConfig struct {
	URL           string `yaml:"url"`
	Token         string `yaml:"token"`
	RootGroupPath string `yaml:"rootGroupPath"`
	DefaultBranch string `yaml:"defaultBranch"`
}

Can't log in with nodeport

Installation Type

script (full)

Installation Result

success

Installation Duration

2 ~ 5min

Environment

- OS:Centos8
- CPU:vm
- Memory:8G

Suggestions & Bugs

I used helm install horizon horizon/horizon -n horizoncd --version 2.1.4 --create-namespace to successfully install the application, but my cluster doesn't have ingress, so I tried to put horizon-web service type to nodePort, the page jumps to /dashboard/clusters after accessing it and I can't log in, it doesn't work
image
image
image

Roadmap

Roadmap

If you would like to join the development of Horizon, please join the Horizon Slack, where we discuss Horizon development-related matters.

If you are a new member of Horizon, you can find an issue that suits you in good first issue.

If you decide to work on a specific feature, please let us know in Slack, and we will change the status to WIP to prevent two people from working on the same feature.

New Feature

Feature Description Status
Activity Recording the user's operations under Group, Application, and Cluster makes it easier for administrators to audit. Help Wanted
Collection Users may find it difficult to locate the applications or clusters they want. The bookmarking feature allows users to bookmark the applications or clusters they frequently use, making it easier for them to access them next time. WIP
Temporary member If a user's cluster has a problem and they need other people to help troubleshoot, they need to add temporary member permissions to their collaborators. Help Wanted
Process control When releasing in a production environment, it is recommended to have a third party perform an audit before the release. Help Wanted
File upload and download During debugging, users may want to upload a script to a Pod for debugging purposes or download logs to their local machine for easy viewing. Help Wanted

Enhancement

Enhancement Description Status
CI Enhancement Horizon's current CI capabilities are still very weak, and in the future, it should be able to support more flexible CI, such as allowing users to customize their own CI. Help Wanted
Delivery Enhancement Currently, Horizon only supports building and releasing through Git repositories, and does not support users directly specifying an image for release. In the future, it may also support other methods of release, such as through charts or Kubernetes manifests. Help Wanted
Resources Display Enhancement Currently, the build and release page in Horizon only displays a list of Pods, but Horizon should support more comprehensive display, such as a resource tree like in ArgoCD. Help Wanted

收藏功能优化

1、应用的集群列表,支持收藏
2、搜索历史记录,改成搜索记录
3、优化集群标签和搜索条件一致,比如性能环境,现在搜索框过滤条件是perf
4、输入条件,点击enter 搜索后,下拉框要隐藏一下
5、搜索输入框,用户输入的内容下拉框不能匹配的,就把下拉框隐藏了

Roadmap - May

Event Description Status Link
Decoupling of CI/CD Support deploying directly through images without building Pending #145
Trigger Add triggers to support other systems calling for builds, such as Github Actions Pending #146
Generic event mechanism Horizon listens to events on Kubernetes and supports sending related events through Webhooks Designing #147
Unit test optimization Horizon has integrated with the Code Climate platform, which shows the test coverage of all files in Horizon. It is possible to selectively modify some of the files with low coverage. For external system dependencies, mocks should be used to solve the problem. Pending #148

CD Refactor

Interface defination

CD in horizon shows below, NextPromotePauseResume is not related to CD, should be removed.

type CD interface {
	CreateCluster(ctx context.Context, params *CreateClusterParams) error
	DeployCluster(ctx context.Context, params *DeployClusterParams) error
	DeleteCluster(ctx context.Context, params *DeleteClusterParams) error
	Next(ctx context.Context, params *ClusterNextParams) error
	Promote(ctx context.Context, params *ClusterPromoteParams) error
	Pause(ctx context.Context, params *ClusterPauseParams) error
	Resume(ctx context.Context, params *ClusterResumeParams) error
	// Deprecated: GetClusterState get cluster state in cd system
	// replaced by GetClusterStatusV2
	GetClusterState(ctx context.Context, params *GetClusterStateParams) (*ClusterState, error)
	GetClusterStateV2(ctx context.Context, params *GetClusterStateV2Params) (*ClusterStateV2, error)
	GetResourceTree(ctx context.Context, params *GetResourceTreeParams) ([]ResourceNode, error)
	GetStep(ctx context.Context, params *GetStepParams) (*Step, error)
	GetContainerLog(ctx context.Context, params *GetContainerLogParams) (<-chan string, error)
	GetPod(ctx context.Context, params *GetPodParams) (*corev1.Pod, error)
	GetPodContainers(ctx context.Context, params *GetPodParams) ([]ContainerDetail, error)
	GetPodEvents(ctx context.Context, params *GetPodEventsParams) ([]Event, error)
	Exec(ctx context.Context, params *ExecParams) (map[string]ExecResp, error)
	// Deprecated
	Online(ctx context.Context, params *ExecParams) (map[string]ExecResp, error)
	// Deprecated
	Offline(ctx context.Context, params *ExecParams) (map[string]ExecResp, error)
	DeletePods(ctx context.Context, params *DeletePodsParams) (map[string]OperationResult, error)
}

Workload

By Workload interface, Horizon describes all kinds of kubernetes workload(such as deployment、pod, etc...)

type Workload interface {
	MatchGK(gk string) bool
}

type GreyscaleReleaser interface {
	Workload

	GetSteps(node *v1alpha1.ResourceNode, client *kube.Client) (*Step, error)
}

type PodsLister interface {
	Workload

	ListPods(node *v1alpha1.ResourceNode, client *kube.Client) ([]corev1.Pod, error)
}

type HealthStatusGetter interface {
	Workload

	IsHealthy(node *v1alpha1.ResourceNode, client *kube.Client) (bool, error)
}

Actions

add Action Method to workload

type Workload interface {
	MatchGK(gk string) bool
	Action(aName string, un *unstructured.Unstructured) (*unstructured.Unstructured, error)
}

Add RunAction method to CD interface, it will call Action in workload

RunAction(clusterName string, actionName string, resourceName string, gvk *meta.GroupVersionKind) error

The returned Unstructured data is the next status of workload in kubernetes, CD should update workload on kubernetes based the returned Unstructured data.

Example

Example for Action of Rollout

func (r *rollout) Action(actionName string, un *unstructured.Unstructured) (*unstructured.Unstructured, error) {
   switch actionName {
   case "resume":
       ...
   case "next":
       ...
   case "stop":
       ...
   case "promote":
       ...
   }
   return newUn, nil
}

[INSTALLATION] specify the horizon helm chart when execute 'helm repo update' command

Installation Type

script (minimal)

Installation Result

success

Installation Duration

more than 10min

Environment

docker info show that there is no limit on docker.
I use kind to install, no cpu, mem limit on kind

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
        protocol: TCP

Suggestions & Bugs

specify the horizon helm chart when execute 'helm repo update' command

【RFC】Design of Horizon Log Package Zap

Design of Horizon Log Package Zap

Introduction

Add an X-Request-ID field to all logs for a given HTTP request.

  • Fatal handlers: logrus allows one or more handlers to be registered, which are called when a fatal level log occurs. This feature is useful when our program needs to be gracefully shut down.

Disadvantages

  • Low execution efficiency, especially in high-concurrency scenarios
  • Does not support log rotation

Based on the above advantages and disadvantages, I propose the following solution:

  • Use zap as the log package for Horizon
  • zap is a high-performance, low-latency log package that supports log rotation and multiple output formats
  • zap is similar to logrus, easy to use and learn, and provides multiple log levels

By using zap, Horizon will obtain better performance and more complete logging functions.

Many excellent open source projects use zap as their log package, such as Kubernetes, etcd, and CockroachDB, as well as the k8s-iam project I am preparing in the future. Therefore, using zap as the log package for Horizon is also a good choice.

Why choose zap

Zap log package can be well compatible with glog. The encapsulation background is that when developing Horizon, we found that glog was used extensively in the Kubernetes source code, and the log package needed to be compatible with glog.

Why choose /pkg/log directory

Migrate the log package to the pkg/log directory instead of pkg/util/log.

  • The first one, the log package belongs to the Horizon project, and there are customization development contents;
  • The second one, the log package has complete and mature functions, and external projects can also use it.

Basic encapsulated log package support functions

  • Basic functions: support basic log information, support different log levels, support custom configuration, support output to standard output and files.
  • Advanced functions: support multiple log formats, can output by level, support structured logs, support log rotation, and have Hook capability.
  • Optional functions: support color output, compatible with the standard library log package, and support output to different locations.

EventHandler panic

What happened:

We catched a panic from event handler. For event handler running in a single goroutine, this goroutine exited, and we could receive any event.

What you expected to happen:

there's should have no panic.

How to reproduce it (as minimally and precisely as possible):

Unfortunately, we don't know what happened exactly.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Others:

request helm install support kubernetes 1.25+

hello, we are using kubernetes 1.25.4 and want to try horizon, but we're blocked at first step: helm templates currently only support 1.19.3 ~ v1.24.7. Please help build a helm template for kubernetes v1.25+, thanks!

Pods page keep spinning

If not region related to group, pods page keeps spinning. Because we checked whether the return value of "region" is empty, if the return value of "region" is empty, the relevant page should still be displayed.

image

Proposal: support creation of more types application

Why is this needed:

Now,

  1. The build method is single and does not support non-git build methods.
  2. The build method depends on Git and build configuration.
  3. The cluster template inherits the application's and cannot be modified.

What would you like to be added:

Detailed design: HEP: support creation of more types application

  1. navigation page for application/cluster creation:
    image
  2. more types of application/cluster creation page:
  • import from git:
    image

  • deploy without artifact (directly deploy a horizon template):
    image

  • deploy image (yet to be designed):
    image

  1. template support type field
field type description note
type varchar support ’normal’, ’nonArtifact’ and so on normal is for git build and image deploy

nonArtifact is for direct deploy

Generic event mechanism

Event Description Status
Generic event mechanism Horizon listens to events on Kubernetes and supports sending related events through Webhooks Designing

Restart operation does not follow GitOps now

Now, only restart operation does not follow GitOps mechanism. Restart time will be committed directly to the main branch, which will introduce other troublesome problems.

Possible solutions

  1. Only rely on argo or k8s to realize this function.
  2. Rely on some git operations on commit or branch.

is tekton deploy task ambiguity?

What would you like to be added:
in horizon system ci task deploy,we tell user "deploy success fully", is it better to be "notify cd success, start to deploy"?
Why is this needed:

Unit test optimization

Event Description Status
Unit test optimization Horizon has integrated with the Code Climate platform, which shows the test coverage of all files in Horizon. It is possible to selectively modify some of the files with low coverage. For external system dependencies, mocks should be used to solve the problem. Pending

my install problem from colima

Installation Type

script (minimal)

Installation Result

success

Installation Duration

less than 2min

Environment

- OS: MacOs Colima
- CPU: 4 core
- Memory: 4GB

Suggestions & Bugs

there are some problems when I install horizon.

  1. horizon-core module restarted when install.
  2. when I use default colima configuration, 2core2GB, I deploy failed. because cpu && memory are too small.
  3. When I create Cluster, the suggested cluster name has Syntax error.

Gitops based operations should not depend on database

Now, basic information of cluster depends on database, such as region, template and release. So, performing a gitops operation requires synchronizing database information.

what to do:

  • Get cluster configs only from git repo.
  • Caching cluster configs should be considered.

[INSTALLATION] Installation feedback

Installation Type

script (minimal)

Installation Result

success

Installation Duration

less than 2min

Environment

- OS: macOS  Monterey
- CPU: Intel Core i7
- Memory: 32GB
- Docker: colima start --cpu 4 --memory 8

Suggestions & Bugs

  1. If Without Promethes configured, give a prompt

image

  1. in CN install mode
    registry.k8s.io/ingress-nginx

  2. rate limit of the access to github api: use can enter the branch just edit by themself (not Strong dependence the git API)

  3. install sample: Speed up install by usge simple example code (less dependence package)

  4. gitRepo: use public repo. and print in the console after installed

  5. install simplize: horizon clean \ horizon --help \ horizon install

failed to deploy pipeline

Pipeline horizoncd/horizon-pipeline can't be Run; it has an invalid spec: invalid value: : finally.params[application].type, finally.params[clusterID].type, finally.params[cluster].type, finally.params[environment].type, finally.params[gitBranch].type, finally.params[gitCommit].type, finally.params[gitRepositoryUrl].type, finally.params[gitSubfolder].type, finally.params[gitTag].type, finally.params[imageURL].type, finally.params[pipelineJSONBlob].type, finally.params[pipelinerunID].type, finally.params[requestID].type, finally.params[token].type, tasks.params[application].type, tasks.params[clusterID].type, tasks.params[cluster].type, tasks.params[environment].type, tasks.params[gitBranch].type, tasks.params[gitCommit].type, tasks.params[gitRepositoryUrl].type, tasks.params[gitSubfolder].type, tasks.params[gitTag].type, tasks.params[imageURL].type, tasks.params[pipelineJSONBlob].type, tasks.params[pipelinerunID].type, tasks.params[requestID].type, tasks.params[token].type

Document improvement

  • The installation documentation in Horizon should be divided into two parts: local installation and production installation.
  • In the documentation, Docker containers should be given fixed names to avoid using IDs.
  • Trying Horizon without config it.

Feature Request: Local Installation with k3s Support for HorizonCD

Background

HorizonCD is a platform for continuous delivery and deployment that supports multiple deployment targets. Currently, HorizonCD only supports Kubernetes(minikube、kind) as a deployment target, but there is a growing demand for support of local installations.

Objective

The objective of this feature request is to add support for local installations using k3s, a lightweight Kubernetes distribution that is optimized for production workloads in resource-constrained environments.

Design

The following table outlines the design considerations for this feature request:

Design Consideration Description
Compatibility Investigate if k3s meets the business requirements of HorizonCD
HorizonCD Version Choose the most appropriate version of HorizonCD to support local installations with k3s
Container Runtime Choose the appropriate CRI (Docker or Containerd) for the local installation
Helm Controller Determine if the Helm controller is required for installation or if it can be installed without the Helm CLI

Implementation Steps

The following steps will be taken to implement this feature request:

Implementation Steps Description
k3s Installation Script Develop an installation script to deploy k3s
HorizonCD Directory Structure Choose an appropriate directory structure for HorizonCD on the local installation
Docker Installation Determine if Docker is required for the local installation
Makefile Deployment Develop a Makefile for one-click deployment of the local installation

Tasks

The current horizonci has a lot to improve in testing and CI, count the PR

Type Improvements Issue PR Status
Makefile Give the help information of Makefile to make it more readable, give some necessary lab #100 #65
k3s-install Suitable for developer local lightweight installation #81 #101

⚠️ Please note that this feature request is currently being discussed on the HorizonCD GitHub Discussions forum.

Who is using Horizon?

Who's using Horizon/谁在使用 Horizon?
The growth of the Horizon community is contingent upon the collaborative endeavors of each and every user and developer.
Horizon 社区的发展离不开每一位用户和开发者的共同努力。

The Horizon community appreciates every user and developer who uses and cares about Horizon. We strive to listen to each one of you to make the community more prosperous.
Horizon 社区感激每一位使用和关注 Horizon 的用户和开发者,我们努力倾听每一个人的声音,让社区变得更加繁荣。

So, the starting point for this topic is to encourage you, as a user or developer, to leave your real-world scenarios so that we can subsequently explore new and better solutions, schedule the releases, and solve real-world problems based on these scenarios!
所以,这个议题的出发点是希望作为用户或者开发者的您留下您的实际使用场景,以便我们后续基于此探索更好的解决方案,制定版本计划,解决实际问题。

We look forward to your comments in the comments space, including but not limited to the following areas.
我们期待您能在评论区留言,其内容包括但不限于以下方面。

Organization / 组织: e.g. NetEase Corp.
Region / 地区: e.g. Hangzhou, China
Contact / 联系方式: e.g. [email protected]
Usage Scenario / 使用场景: e.g., we use Horizon to automatically optimize Iceberg tables./ we use Horizon to build batch-and-stream-fused data pipelines.
Bests / 致敬,

Horizon Community

[INSTALLATION] <title>123-test

Installation Type

script (minimal)

Installation Result

success

Installation Duration

less than 2min

Environment

  • OS:
  • CPU:
  • Memory:

Suggestions & Bugs

12

Decoupling of CI/CD

Event Description Status
Decoupling of CI/CD Support deploying directly through images without building Pending

helm install --set host

What would you like to be added:
helm 在腾讯云安装时候,需要改变hosts,文档告知 helm --set 对用户比较好,同时一些外部以来如gitlab和registry可选安装
Why is this needed:

Build status should not relay on s3 service

What happened:
if s3 service is down, could not get building status

What you expected to happen:
Build status should not relay on s3 service

How to reproduce it (as minimally and precisely as possible):
Stop s3, then go to cluster pods page, and you can see the error

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Others:

Installation Improvement

  1. support creating pvc whose storage greater than 20Gi #84
  2. distributing horizon with k3s embeded #81
  3. upgrade horizon helm chart to suppport k8s v1.25 & v1.26
  4. we need the minimized intallation of horizon, which means we can install horizon without harbor, redis, minio, gitlab, etc.

To complete the minimal installation, we have listed the following tasks:

Task Description Assignees Status
Merge Core with Job After merging core and job, where a job can only have one instance, we need to rely on Kubernetes to implement the corresponding leader election logic so that the main instance can run the job logic on a goroutine. @kilosonc Done
Simplify Registry Harbor requires a lot of running resources. We plan to replace Harbor with a more lightweight image repository to facilitate launching Horizon locally. @kilosonc Done
Extrenal Gitlab Horizon should support users to use an external GitLab, such as a public GitLab service. @kilosonc Done
Horizon configuration automaticlly Now, after pulling up the Horizon system, we need configure kubernetes, environment, registry manually. We need an automated script to do the configuration instead of having the user configure it manually @kilosonc Done
Remove Redis Now redis is used to store session, we can just use memory to store session @fdiskbrain Done
Remove Minio Minio is used to store log of the PipelineRun, actually we don't need it if we didn't @xuzhu-591 Done

【RFC】Fix and improve the contributor documentation

Problems that arise

image

why?

Contributing to horizon

Contributing to horizon

So, you want to hack on horizon? Yay!

First of all, thank you for considering contributing to our project! We appreciate your time and effort, and we value any contribution, whether it's reporting a bug, suggesting a new feature, or submitting a pull request.

This document provides guidelines and best practices to help you contribute effectively.

📇Topics

What we expect of you

We hope that anyone can join horizon , even if you are a student, writer, translator

Please meet the minimum version of the Go language published in go.mod. If you want to manage the Go language version, we provide tools to install gvm in our Makefile

You'd better use Linux as the development environment, Linux with Makefile can help you quickly build and test horizon project.

If you are familiar with Makefile , you can easily see the clever design of the horizon Makefile. Storing the necessary tools such as golangci in the /tools directory can avoid some tool version issues.

The Makefile is for every developer, even if you don't know how to use the Makefile tool, don't worry, we provide two great commands to get you up to speed with the Makefile architecture, make help and make help-all, it can reduce problems of the developing environment.

Code of ConductCode of Conduct

We divide the problem into security and general problems:

Reporting security issues

Security issues are always treated seriously. As our usual principle, we discourage anyone to spread security issues. If you find a security issue of horizon, please do not discuss it in public and even do not open a public issue.

Instead we encourage you to send us a private email to [email protected]@gmail.com to report this.

Reporting general issues

To be honest, we regard every user of horizonas a very kind contributor. After experiencing horizon, you may have some feedback for the project. Then feel free to open an issue via NEW ISSUE.

Since we collaborate project horizon in a distributed way, we appreciate WELL-WRITTEN, DETAILED, EXPLICIT issue reports. To make the communication more efficient, we wish everyone could search if your issue is an existing one in the searching list. If you find it existing, please add your details in comments under the existing issue instead of opening a brand new one.

To make the issue details as standard as possible, we setup an ISSUE TEMPLATE for issue reporters. You can find three kinds of issue templates there: question, bug report and feature request. Please BE SURE to follow the instructions to fill fields in template.

There are a lot of cases when you could open an issue:

  • bug report
  • feature request
  • horizon performance issues
  • feature proposal
  • feature design
  • help wanted
  • doc incomplete
  • test improvement
  • any questions on horizon project
  • and so on

Also, we must be reminded when submitting a new question about horizon, please remember to remove the sensitive data from your post. Sensitive data could be password, secret key, network locations, private business data and so on.

Code and doc contribution

Every action to make project horizon better is encouraged. On GitHub, every improvement for horizon could be via a PR (short for pull request).

  • If you find a typo, try to fix it!
  • If you find a bug, try to fix it!
  • If you find some redundant codes, try to remove them!
  • If you find some test cases missing, try to add them!
  • If you could enhance a feature, please DO NOT hesitate!
  • If you find code implicit, try to add comments to make it clear!
  • If you find code ugly, try to refactor that!
  • If you can help to improve documents, it could not be better!
  • If you find document incorrect, just do it and fix that!
  • ...

Where should I start?

  • If you are new to the project, don't know how to contribute horizon, please check out the good first issue label.
  • You should be good at filtering the horizon issue tags and finding the ones you like, such as RFC for big initiatives, features for feature proposals, and bug fixes.
  • If you are looking for something to work on, check out our open issues.
  • If you have an idea for a new feature, please open an issue, and we can discuss it.

Design documents

For any substantial design, there should be a well-crafted design document. This document is not just a simple record, but also a detailed description and manifestation, which can help team members better understand the design thinking and grasp the design direction. In the process of writing the design document, we can choose to use tools such as Google Docs or Notion, and even mark RFC in issues or discussions for better collaboration. Of course, after completing the design document, we should also add it to our Shared Drive and notify the appropriate working group to let everyone know of its existence. Only by doing so can we maximize the effectiveness of the design document and provide strong support for the smooth progress of the project.

Anybody can access the shared Drive for reading. To get access to comment. Once you've done that, head to the shared Drive and behold all the docs.

In addition to that, we'd love to invite you to join our Slack where you can play with your imagination, tell us what you're working on, and get a quick response.

When documenting a new design, we recommend a 2-step approach:

  1. Use the short-form RFC template to outline your ideas and get early feedback.
  2. Once you have received sufficient feedback and consensus, you may use the longer-form design doc template to specify and discuss your design in more details.

In order to contribute a feature to horizon you'll need to go through the following steps:

  • Discuss your idea with the appropriate working groups on the working group's Slack channel.
  • Once there is general agreement that the feature is useful, create a GitHub issue to track the discussion. The issue should include information about the requirements and use cases that it is trying to address.
  • Include a discussion of the proposed design and technical details of the implementation in the issue.

But keep in mind that there is no guarantee of it being accepted and so it is usually best to get agreement on the idea/design before time is spent coding it. However, sometimes seeing the exact code change can help focus discussions, so the choice is up to you.

Getting Started

To propose PR for the horizon item, we assume you have registered a GitHub ID. Then you could finish the preparation in the following steps:

  1. Fork the repository(horizon)

  2. CLONE your own repository to master locally. Use git clone https://github.com/<your-username>/horizon.git to clone repository to your local machine. Then you can create new branches to finish the change you wish to make.

  3. Set Remote upstream to be https://github.com/horizoncd/horizon.git using the following two commands:

    ❯ git remote add upstream https://github.com/horizoncd/horizon.git
    ❯ git remote set-url --push upstream no-pushing

    With this remote setting, you can check your git remote configuration like this:

    git remote -v
    origin     https://github.com/<your-username>/horizon.git (fetch)
    origin     https://github.com/<your-username>/horizon.git (push)
    upstream   https://github.com/horizoncd/horizon.git (fetch)
    upstream   no-pushing (push)

    Adding this, we can easily synchronize local branches with upstream branches.

  4. Create a new branch for your changes (use a descriptive name, such as fix-bug-123 or add-new-feature).

    cd horizon
    ❯ git fetch upstream
    ❯ git checkout main

    Create a new branch:

    ❯ git checkout -b <new-branch>

    Make any change on the new-branch then use Makefile build and test your codes.

  5. Commit your changes to your local branch, lint before committing and commit with sign-off

    ❯ git rebase upstream/main
    ❯ make link	  # golangci-lint run -c .golangci.yml
    ❯ git add -A  # add changes to staging
    ❯ git commit -a -s -m "message for your changes" # -s adds a Signed-off-by trailer
  6. Push your branch to your forked repository, it is recommended to have only one commit for a PR.

    # sync up with upstream
    ❯ git fetch upstream main
    ❯ git rebase upstream/main
    ❯ 
    ❯ git rebase -i	<commit-id> # rebase with interactive mode to squash your commits into a single one
    ❯ git push # push to the remote repository, if it's a first time push, run git push --set-upstream origin <new-branch># sync up with upstream
    ❯ git fetch upstream main
    git rebase upstream/main
    
    ❯ git rebase -i	<commit-id> # rebase with interactive mode to squash your commits into a single one
    ❯ git push # push to the remote repository, if it's a first time push, run git push --set-upstream origin <new-branch>

    You can also use git commit -s --amend && git push -f to update modifications on the previous commit.

    If you have developed multiple features in the same branch, you should create PR separately by rebasing to the main branch between each push:

    # create new branch, for example git checkout -b feature/infra
    ❯ git checkout -b <new branch>
    # update some code, feature1
    ❯ git add -A
    ❯ git commit -m -s "feat: feature one"
    ❯ git push # if it's first time push, run git push --set-upstream origin <new-branch>
    # then create pull request, and merge
    # update some new feature, feature2, rebase main branch first.
    ❯ git rebase upstream/main # rebase the current branch to upstream/main branch
    ❯ git add -A
    ❯ git commit -m -s "feat: feature two"
    # then create pull request, and merge
  7. File a pull request to horizoncd/horizon:main

    It is recommended to review your changes before filing a pull request. Check if your code doesn't conflict with the main branch and no redundant code is included.

Style and Specification

Commit Rules

Actually in horizon, we take two rules serious when committing:

🥇 Commit Message:

Commit message could help reviewers better understand what the purpose of submitted PR is. It could help accelerate the code review procedure as well. We encourage contributors to use EXPLICIT commit message rather than ambiguous message. In general, we advocate the following commit message type:

We use Semantic Commits to make it easier to understand what a commit does and to build pretty changelogs. Please use the following prefixes for your commits:

  • docs: xxxx. For example, "docs: add docs about storage installation".
  • feature: xxxx.For example, "feature: make result show in sorted order".
  • bugfix: xxxx. For example, "bugfix: fix panic when input nil parameter".
  • style: xxxx. For example, "style: format the code style of Constants.java".
  • refactor: xxxx. For example, "refactor: simplify to make codes more readable".
  • test: xxx. For example, "test: add unit test case for func InsertIntoArray".
  • chore: xxx. For example, "chore: integrate travis-ci". It's the type of mantainance change.
  • other readable and explicit expression ways.

On the other side, we discourage contributors from committing message like the following ways:

  • fix bug
  • update
  • add doc

🥈 Commit Content:

Commit content represents all content changes included in one commit. We had better include things in one single commit which could support reviewer's complete review without any other commits' help.

In another word, contents in one single commit can pass the CI to avoid code mess. In brief, there are two minor rules for us to keep in mind:

  1. avoid very large change in a commit.
  2. complete and reviewable for each commit.
  3. words are written in lowercase English, not uppercase English or other languages such as Chinese.

No matter what the commit message, or commit content is, we do take more emphasis on code review.

An example for this could be:

❯ git commit -a -s -m "docs: add a new section to the README"

PR Description

PR is the only way to make change to horizon project files. To help reviewers better get your purpose, PR description could not be too detailed. We encourage contributors to follow the PR template to finish the pull request.

You can find some very formal PR in RFC issues and learn about them.

📖 Opening PRs:

  • As long as you are working on your PR, please mark it as a draft
  • Please make sure that your PR is up-to-date with the latest changes in main
  • Mention the issue that your PR is addressing (Fix: #{ID_1}, #{ID_2})
  • Make sure that your PR passes all checks

🈴 Reviewing PRs:

  • Be respectful and constructive
  • Assign yourself to the PR
  • Check if all checks are passing
  • Suggest changes instead of simply commenting on found issues
  • If you are unsure about something, ask the author
  • If you are not sure if the changes work, try them out
  • Reach out to other reviewers if you are unsure about something
  • If you are happy with the changes, approve the PR
  • Merge the PR once it has all approvals and the checks are passing

⚠️ DCO check:

We have a DCO check which runs on every PR to verify that the commit has been signed off.

To sign off the last commit you made, you can use:

❯ git commit --amend --signoffgit commit --amend --signoff

You can also automate signing off your commits by adding the following to your .zshrc or .bashrc:

git() {
  if [ $# -gt 0 ] && [[ "$1" == "commit" ]] ; then
     shift
     command git commit --signoff "$@"
  else
     command git "$@"
  fi
}

Docs Contribution

The documentation for horizon includes:

  • README.md: This file includes the basic information and instructions for getting started with horizon.
  • CONTRIBUTING.md: This file contains guidelines for contributing to horizon's codebase, such as how to submit issues, pull requests, and code reviews.
  • DEVELOPGUIDE.md: This file provides a more in-depth guide to developing horizon, including information on the project's architecture, coding conventions, and testing practices.
  • Official Documentation: This is the official documentation for horizon, which includes comprehensive information on all of its features, configuration options, and troubleshooting tips.

Please obey the following rules to better format the docs, which would greatly improve the reading experience.

  1. Please do not use Chinese punctuations in English docs, and vice versa.
  2. Please use upper case letters where applicable, like the first letter of sentences / headings, etc.
  3. Please specify a language for each Markdown code blocks, unless there's no associated languages.
  4. Please insert a whitespace between Chinese and English words.
  5. Please use the correct case for technical terms, such as using HTTP instead of http, MySQL rather than mysql, Kubernetes instead of kubernetes, etc.
  6. Please check if there's any typos in the docs before submitting PRs.

Engage to help anything

We choose GitHub as the primary place for horizon to collaborate. So the latest updates of horizon are always here. Although contributions via PR is an explicit way to help, we still call for any other ways.

  • reply to other's issues if you could;
  • help solve other user's problems;
  • help review other's PR design;
  • discuss about horizon to make things clearer;
  • advocate horizon technology beyond GitHub;
  • write blogs on horizon and so on.

In a word, ANY HELP IS CONTRIBUTION.

Release version

Releases of horizon are done using Release Please and GoReleaser. The workflow looks like this:

🎯 A PR is merged to the main branch:

  • Release please is triggered, creates or updates a new release PR
  • This is done with every merge to main, the current release PR is updated every time

🎯 Merging the 'release please' PR to main:

  • Release please is triggered, creates a new release and updates the changelog based on the commit messages
  • GoReleaser is triggered, builds the binaries and attaches them to the release
  • Containers are created and pushed to the container registry

With the next relevant merge, a new release PR will be created and the process starts again

👀 Manually setting the version:

If you want to manually set the version, you can create a PR with an empty commit message that contains the version number in the commit message. For example:

Such a commit can get produced as follows:

❯ git commit --allow-empty -m "chore: release 0.0.3" -m "Release-As: 0.0.3

Contact Us

We value close connections with our users, developers, and contributors here at horizon. With a large community and maintainer team, we're always here to help and support you. Whether you're looking to join our community or have any questions or suggestions, we welcome you to get in touch with us.

Our most recommended way to get in touch is through Slack. Even if you're in China, Slack is usually not blocked by firewalls, making it an easy way to connect with us. Our Slack community is the ideal place to discuss and share ideas and suggestions with other users and developers of horizon. You can ask technical questions, seek help, or share your experiences with other users of horizon.

In addition to Slack, we also offer the following ways to get in touch:

  • : We also have Slack channels for you to communicate and discuss. To join, visit https://slack.com/ and join our 👀 horizon slack team channel.
  • : Get in touch with us on Gmail. If you have any questions or issues that need resolving, or any suggestions and feedback for our open source projects, please feel free to contact us via email.
  • : Read our blog. Our blog is a great place to stay up-to-date with horizon projects and trends. On the blog, we share our latest developments, tech trends, and other interesting information.
  • : Add Wechat and indicate that you are a user or developer of horizon. We will process your request as soon as possible.

Whether you're looking to join our community or have any questions or suggestions, we welcome you to get in touch with us.

horizon-mysql start error

my k8s version: v1.19.3+k3s3

k get pod -n horizoncd|grep mysql
horizon-mysql-0 0/1 Running 1 4m8s

mysql 07:53:30.24 INFO ==> ** Starting MySQL setup **
mysql 07:53:30.88 INFO ==> Validating settings in MYSQL_/MARIADB_ env vars
mysql 07:53:31.06 INFO ==> Initializing mysql database
mysql 07:53:31.75 WARN ==> The mysql configuration file '/opt/bitnami/mysql/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mysql 07:53:31.77 INFO ==> Using persisted data
mysql 07:53:32.64 INFO ==> Running mysql_upgrade
mysql 07:53:32.85 INFO ==> Starting mysql in background
mysql 07:56:02.74 INFO ==> Loading user's custom files from /docker-entrypoint-initdb.d
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1061 (42000) at line 67 in file: '/docker-entrypoint-initdb.d/init.sql': Duplicate key name 'idx_application_id'
mysql 07:56:07.85 ERROR ==> Failed executing /docker-entrypoint-initdb.d/init.sql
mysql 07:56:08.34 INFO ==> Stopping mysql

k get pod -n horizoncd|egrep -v "1/1|3/3|2/2"
NAME READY STATUS RESTARTS AGE
horizon-mysql-0 0/1 Running 1 4m31s
horizon-job-57b588ffc6-8zpmf 0/1 CrashLoopBackOff 4 117s
horizon-core-86f8f4c76f-xphr8 0/1 CrashLoopBackOff 4 95s

k logs horizon-core-86f8f4c76f-xphr8 -n horizoncd
2023/02/17 11:35:12 /horizon/lib/orm/orm.go:44
[error] failed to initialize database, got error dial tcp 10.43.164.121:3306: connect: connection refused

2023/02/17 11:35:12 /horizon/lib/orm/metrics.go:49 dial tcp 10.43.164.121:3306: connect: connection refused
[2.632ms] [rows:-] SHOW STATUS

2023/02/17 11:35:12 /horizon/lib/orm/metrics.go:52
[error] gorm:prometheus query error: dial tcp 10.43.164.121:3306: connect: connection refused
panic: dial tcp 10.43.164.121:3306: connect: connection refused

goroutine 1 [running]:
github.com/horizoncd/horizon/core/cmd.Run(0xc000974ab0)
/horizon/core/cmd/cmd.go:267 +0x6525
main.main()
/horizon/core/main.go:26 +0x2a

k logs horizon-job-57b588ffc6-8zpmf -n horizoncd

2023/02/17 11:34:56 /horizon/lib/orm/orm.go:44
[error] failed to initialize database, got error dial tcp 10.43.164.121:3306: connect: connection refused

2023/02/17 11:34:56 /horizon/lib/orm/metrics.go:49 dial tcp 10.43.164.121:3306: connect: connection refused
[1.812ms] [rows:-] SHOW STATUS

2023/02/17 11:34:56 /horizon/lib/orm/metrics.go:52
[error] gorm:prometheus query error: dial tcp 10.43.164.121:3306: connect: connection refused
panic: dial tcp 10.43.164.121:3306: connect: connection refused

goroutine 1 [running]:
github.com/horizoncd/horizon/job/cmd.Run(0xc000b86d40)
/horizon/job/cmd/cmd.go:105 +0x15e5
main.main()
/horizon/job/main.go:24 +0x2a

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.