caicloud / cyclone Goto Github PK
View Code? Open in Web Editor NEWPowerful workflow engine and end-to-end pipeline solutions implemented with native Kubernetes resources. https://cyclone.dev
License: Apache License 2.0
Powerful workflow engine and end-to-end pipeline solutions implemented with native Kubernetes resources. https://cyclone.dev
License: Apache License 2.0
Now, Cyclone can only rolling update deployment for k8s. It need to support create new deployment. Firstly, we can read deployment config from files (maybe k8s config map) to create new deployment. Secondly, we need to config new deployment from web.
Pipeline record is the running record of the pipeline.
What happened
What you expected to happen
Reproduce steps (as minimally and precisely as possible)
Let Cyclone to be a open source platform.
We should support badge in cyclone, just like travis.
Need to provide the REST APIs to operate the pipeline record.
When I run e2e-k8s.sh
, the command-line interface shows me that "All system worker are busy, wait for 10 seconds" and then it gets failed.
Now, the CI/CD workflow is prebuild -> build -> integration -> post build -> deploy. It's Inflexible. We need to let user can compose the workflow freely.
Environment
What happened
What you expected to happen
Reproduce steps (as minimally and precisely as possible)
Anything else
We should use the circle, not caicloud to store images.
project
was used as a union building in last year. Now, we use this concept to group pipelines. We need to clean the code relating about the old project
There are some mistakes in setup docs:
scripts/build-image.sh
not exist use scripts/build-server-image.sh
insteaddocker-compose
is cyclone_clair_1
not clair_clair
.(docker-compose version 1.9.0)We need a document for anyone wanting to either write code which directly accesses the Cyclone API, or to contribute directly to the Cyclone project. It should include:
Now the docker image is not found at cargo.caicloud.io, we should fix it immediately.
Due: 11.9
Now we only support config file named "caicloud.yml", we need to support customized file name.
The job has done, waiting for a PR.
Now we check the docker version in worker node to make sure that the version is same as WORKER_NODE_DOCKER_VERSION defined in circle server.
I think we should research the changelog of docker and reduce the claim about docker version :) Maybe above 1.10
is OK.
Describe:
When i restarted the cyclone server, the worker can't send heart beat and log info to server through websocket.
Now cyclone has some unit test cases but we don't have a tool to get the test coverage analysis.
There are some tools, such as gocover.io and goveralls.
gocover.io is ease of use but doesn't support multi-package test now(ref https://github.com/vieux/gocover.io/issues/15), so maybe we could use goveralls instead.
As we discussed on 11/15th, we need to modify the operation durning the integration stage. Cyclone uses the image which waits to be published to run a container, and execute integration test in this container. Now, Cyclone needs to run this container as a service, and execute integration test in another container whose image specialed in the integration section of yaml file.
./scripts/build-image.sh
+++++ Start building cyclone server
can't load package: package github.com/caicloud/cyclone: open /go/src/github.com/caicloud/cyclone: permission denied
Now cyclone's godep has some mannual work, it is not encouraged. So we should update godep and make sure that there is no human intervention when update the denpendency.
Environment
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Tue Mar 28 00:40:02 2017
OS/Arch: darwin/amd64
Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Fri Mar 24 00:00:50 2017
OS/Arch: linux/amd64
Experimental: true
What happened
Successful collection GET operation MUST return metadata with 'total' and 'items' array, i.e.
{
"metadata": {
"total": 100,
"itemsLength": 10
},
"items": [ {}, {}, {} ]
}
Collection GET (A.K.A. List APIs)
SHOULD support pagination query: GET /api/v0/items?start=&queryName2=
Cyclone need to count sending time for every workflow steps.
The project is used to manager a group of related CICD workflows, just like Jenkins folder to manage pipelines. It can be a real project, which has several or many applications or micro services.
Todo Lists:
Now we pull the images every time we run the integration step. Maybe we could mount a volume to /var/lib/docker, or something else.
Now code coverage of cyclone is a little low 😕 , so we need more test cases.
Environment
What happened
What you expected to happen
Reproduce steps (as minimally and precisely as possible)
Anything else
Need to provide the REST APIs to operate the pipeline.
ref https://travis-ci.org/caicloud/cyclone/builds/176273667
It seems that there is an error raised by circle-worker:
json: cannot unmarshal string into Go value of type api.VersionLogCreateResponse
------------------------------
• Failure [14.006 seconds]
Yaml
/home/travis/gopath/src/github.com/caicloud/cyclone/tests/yaml/yaml_test.go:270
with right version information
/home/travis/gopath/src/github.com/caicloud/cyclone/tests/yaml/yaml_test.go:269
with the integration operation
/home/travis/gopath/src/github.com/caicloud/cyclone/tests/yaml/yaml_test.go:219
should be able to get version via HTTP GET method. [It]
/home/travis/gopath/src/github.com/caicloud/cyclone/tests/yaml/yaml_test.go:213
Expected
<api.VersionStatus>: failed
to equal
<api.VersionStatus>: healthy
/home/travis/gopath/src/github.com/caicloud/cyclone/tests/yaml/yaml_test.go:212
------------------------------
Describe:
The worker container was shut down because of timeout or be cancelled, this cause that mongo do not have the log about CI/CD.
Solution:
Server after the closure of the worker container put the log in kafka into the mongo
ERROR: for cyclone_etcd_1 Cannot start service etcd: driver failed programming external connectivity on endpoint cyclone_etcd_1 (24b767714249941982970cfd2e43141ff31cdcdef92b44c73545fc4c6120a8f9): Error starting userland proxy: listen tcp 0.0.0.0:2380: bind: address already in use
ERROR: for etcd Cannot start service etcd: driver failed programming external connectivity on endpoint cyclone_etcd_1 (24b767714249941982970cfd2e43141ff31cdcdef92b44c73545fc4c6120a8f9): Error starting userland proxy: listen tcp 0.0.0.0:2380: bind: address already in use
ERROR: Encountered errors while bringing up the project
INFO[0182] watch unfinshed events set: "{\"event_id\":\"e8986325-ec7e-440b-873c-d4f1329e2d69\",\"operation\":\"create-version\",\"service\":{\"_id\":\"230773ca-5515-4e69-9975-8ed157defd65\",\"user_id\":\"fake-user-id\",\"name\":\"test-service\",\"username\":\"gaocegege\",\"description\":\"This is a test-service\",\"version_fails\":[\"36526ffd-d07f-4f18-bf3d-d79ef09e929b\"],\"repository\":{\"url\":\"https://github.com/gaocegege/cyclone\",\"vcs\":\"git\",\"status\":\"healthy\"},\"jconfig\":{},\"profile\":{\"profiles\":[{\"email\":\"[email protected]\",\"cellphone\":\"useless\"}],\"setting\":\"sendwhenfinished\"},\"last_createtime\":\"2016-11-15T10:04:50.449+08:00\",\"last_versionname\":\"v0.1.3\"},\"version\":{\"_id\":\"e8986325-ec7e-440b-873c-d4f1329e2d69\",\"service_id\":\"230773ca-5515-4e69-9975-8ed157defd65\",\"name\":\"v0.1.0\",\"description\":\"just for test\",\"create_time\":\"2016-11-15T10:05:47.711198846+08:00\",\"url\":\"https://github.com/gaocegege/cyclone\",\"status\":\"pending\",\"yaml_deploy_status\":\"norun\",\"operation\":\"integration\",\"operator\":\"api\",\"build_resource\":{\"memory\":5.36870912e+08,\"cpu\":512}},\"project\":{\"create_time\":\"0001-01-01T00:00:00Z\"},\"project_version\":{\"create_time\":\"0001-01-01T00:00:00Z\"},\"data\":{\"Token\":\"\",\"service-name\":\"test-service\",\"username\":\"gaocegege\",\"version-name\":\"v0.1.0\"},\"worker_info\":{\"docker_host\":\"0.0.0.0:2375\",\"container_id\":\"46f49a6d4d92267413637e3a4fd2b90201a5132f960e3c237c887b03da976935\",\"due_time\":\"2016-11-15T12:05:49.293378316+08:00\",\"used_resource\":{}},\"status\":\"running\"}"
_file_=manager.go:103
INFO[0192] check token: e8986325-ec7e-440b-873c-d4f1329e2d69 _file_=event.go:169
INFO[0192] receive heart_beat packet _file_=application.go:113
ERRO[0192] packet data unmarshal recover: %sruntime error: invalid memory address or nil pointer dereference _file_=application.go:56
ERRO[0192] set tcp [::1]:8000: use of closed network connection _file_=server.go:56
INFO[0231] check token: e8986325-ec7e-440b-873c-d4f1329e2d69 _file_=event.go:169
INFO[0231] set etcd: {"event_id":"e8986325-ec7e-440b-873c-d4f1329e2d69","operation":"create-version","service":{"_id":"230773ca-5515-4e69-9975-8ed157defd65","user_id":"fake-user-id","name":"test-service","username":"gaocegege","description":"This is a test-service","version_fails":["36526ffd-d07f-4f18-bf3d-d79ef09e929b"],"repository":{"url":"https://github.com/gaocegege/cyclone","vcs":"git","status":"healthy"},"jconfig":{},"profile":{"profiles":[{"email":"[email protected]","cellphone":"useless"}],"setting":"sendwhenfinished"},"last_createtime":"2016-11-15T10:04:50.449+08:00","last_versionname":"v0.1.3"},"version":{"_id":"e8986325-ec7e-440b-873c-d4f1329e2d69","service_id":"230773ca-5515-4e69-9975-8ed157defd65","name":"v0.1.0","description":"just for test","commit":"a179d6c8446e32cd43165e426204b2005d96cefa","create_time":"2016-11-15T10:05:47.711198846+08:00","url":"https://github.com/gaocegege/cyclone","status":"pending","yaml_deploy_status":"norun","operation":"integration","operator":"api","build_resource":{"memory":5.36870912e+08,"cpu":512}},"project":{"create_time":"0001-01-01T00:00:00Z"},"project_version":{"create_time":"0001-01-01T00:00:00Z"},"data":{"Token":"","service-name":"test-service","username":"gaocegege","version-name":"v0.1.0"},"worker_info":{"docker_host":"0.0.0.0:2375","container_id":"46f49a6d4d92267413637e3a4fd2b90201a5132f960e3c237c887b03da976935","due_time":"2016-11-15T12:05:49.293378316+08:00","used_resource":{}},"status":"fail","error_msg":"container meets error"} _file_=event.go:152
INFO[0231] watch unfinshed events set: "{\"event_id\":\"e8986325-ec7e-440b-873c-d4f1329e2d69\",\"operation\":\"create-version\",\"service\":{\"_id\":\"230773ca-5515-4e69-9975-8ed157defd65\",\"user_id\":\"fake-user-id\",\"name\":\"test-service\",\"username\":\"gaocegege\",\"description\":\"This is a test-service\",\"version_fails\":[\"36526ffd-d07f-4f18-bf3d-d79ef09e929b\"],\"repository\":{\"url\":\"https://github.com/gaocegege/cyclone\",\"vcs\":\"git\",\"status\":\"healthy\"},\"jconfig\":{},\"profile\":{\"profiles\":[{\"email\":\"[email protected]\",\"cellphone\":\"useless\"}],\"setting\":\"sendwhenfinished\"},\"last_createtime\":\"2016-11-15T10:04:50.449+08:00\",\"last_versionname\":\"v0.1.3\"},\"version\":{\"_id\":\"e8986325-ec7e-440b-873c-d4f1329e2d69\",\"service_id\":\"230773ca-5515-4e69-9975-8ed157defd65\",\"name\":\"v0.1.0\",\"description\":\"just for test\",\"commit\":\"a179d6c8446e32cd43165e426204b2005d96cefa\",\"create_time\":\"2016-11-15T10:05:47.711198846+08:00\",\"url\":\"https://github.com/gaocegege/cyclone\",\"status\":\"pending\",\"yaml_deploy_status\":\"norun\",\"operation\":\"integration\",\"operator\":\"api\",\"build_resource\":{\"memory\":5.36870912e+08,\"cpu\":512}},\"project\":{\"create_time\":\"0001-01-01T00:00:00Z\"},\"project_version\":{\"create_time\":\"0001-01-01T00:00:00Z\"},\"data\":{\"Token\":\"\",\"service-name\":\"test-service\",\"username\":\"gaocegege\",\"version-name\":\"v0.1.0\"},\"worker_info\":{\"docker_host\":\"0.0.0.0:2375\",\"container_id\":\"46f49a6d4d92267413637e3a4fd2b90201a5132f960e3c237c887b03da976935\",\"due_time\":\"2016-11-15T12:05:49.293378316+08:00\",\"used_resource\":{}},\"status\":\"fail\",\"error_msg\":\"container meets error\"}"
_file_=manager.go:103
INFO[0231] Afer release, the userid fake-user-id's left resource memory 4294967296.000000 cpu 4096.000000 _file_=resource.go:146
INFO[0231] create version post hook _file_=event.go:167
ERRO[0231] Unable to get remote according coderepository _file_=event.go:193 user_id=fake-user-id
INFO[0231] Env variant SUCCESSTEMPLATE found, using env value: ./notify/provider/success.html _file_=osutil.go:44
INFO[0231] Env variant ERRORTEMPLATE found, using env value: ./notify/provider/error.html _file_=osutil.go:44
INFO[0237] Env variant SUCCESSTEMPLATE found, using env value: ./notify/provider/success.html _file_=osutil.go:44
INFO[0237] Env variant ERRORTEMPLATE found, using env value: ./notify/provider/error.html _file_=osutil.go:44
INFO[0243] watch finshed events delete: "{Key: /events/unfinished/e8986325-ec7e-440b-873c-d4f1329e2d69, CreatedIndex: 18, ModifiedIndex: 18, TTL: 0}"
_file_=manager.go:122
I'm focusing on #21 , and meets this error. I don't know the cause now. But it doesn't impact the CI and CD step, just about log.
FYI
We need to redefine out union build function. Not only union CD, but also union CI.
Pipeline is used to describe the config and workflow of CI/CD.
Cyclone only supports to running service, as a workaround to run pipeline, need to convert it to service.
Let Cyclone more simple.
Our API doc uses Swagger UI with go-restful, and now it needs to input the http://<host>:7099/apidocs.json
into the input box, we should optimize the API doc by removing the input step 😉
Cyclone need to rebuild or build special commit .
Describe:
Log file operations should only exist in the worker part, so the "Output" field does not need to exist in the event struct , and modify the worker code.
We need to add function doc to introduce what Cyclone can do.
Now there is no script to register nodes easily, there is a scripts/tools/api.py
but it is complicated, we need to refactor it, or rewrite it in bash.
optimize code by goreport
We could use cyclone to test and deploy cyclone, just like other CI tools.
Need to provide the REST APIs to operate the project.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.