Giter Club home page Giter Club logo

lighthouse's Introduction

Lighthouse

Lighthouse is a lightweight ChatOps based webhook handler which can trigger Jenkins X Pipelines, Tekton Pipelines or Jenkins Jobs based on webhooks from multiple git providers such as GitHub, GitHub Enterprise, BitBucket Server and GitLab.

Installing

Lighthouse is bundled and released as Helm Chart. You find the install instruction in the Chart's README.

Depending on the pipeline engine you want to use, you can find more detailed instructions in one of the following documents:

Background

Lighthouse derived originally from Prow and started with a copy of its essential code.

Currently, Lighthouse supports the standard Prow plugins and handles push webhooks to branches to then trigger a pipeline execution on the agent of your choice.

Lighthouse uses the same config.yaml and plugins.yaml for configuration than Prow.

Comparisons to Prow

Lighthouse reuses the Prow plugin source code and a bunch of plugins from Prow

Its got a few differences though:

  • rather than being GitHub specific Lighthouse uses jenkins-x/go-scm so it can support any Git provider
  • Lighthouse does not use a ProwJob CRD; instead, it has its own LighthouseJob CRD.

Porting Prow commands

If there are any prow commands you want which we've not yet ported over, it is relatively easy to port Prow plugins.

We've reused the prow plugin code and configuration code; so it is mostly a case of switching imports of k8s.io/test-infra/prow to github.com/jenkins-x/lighthouse/pkg/prow, then modifying the GitHub client structs from, say, github.PullRequest to scm.PullRequest.

Most of the GitHub structs map 1-1 to the jenkins-x/go-scm equivalents (e.g. Issue, Commit, PullRequest). However, the go-scm API does tend to return slices to pointers to resources by default. There are some naming differences in different parts of the API as well. For example, compare the githubClient API for Prow lgtm versus the Lighthouse lgtm.

Development

Building

To build the code, fork and clone this git repository, then type:

make build

make build will build all relevant Lighthouse binaries natively for your OS which you then can run locally. For example, to run the webhook controller, you would type:

./bin/webhooks

To see which other Make rules are available, run:

make help

Environment variables

While Prow only supports GitHub as SCM provider, Lighthouse supports several Git SCM providers. Lighthouse achieves the abstraction over the SCM provider using the go-scm library. To configure your SCM, go-scm uses the following environment variables :

Name Description
GIT_KIND the kind of git server: github, gitlab, bitbucket, gitea, stash
GIT_SERVER the URL of the server if not using the public hosted git providers: https://github.com, https://bitbucket.org or https://gitlab.com
GIT_USER the git user (bot name) to use on git operations
GIT_TOKEN the git token to perform operations on git (add comments, labels etc.)
HMAC_TOKEN the token sent from the git provider in webhooks

Testing

To run the unit tests, type:

make test

For development purposes, it is also nice to start an instance of the binary you want to work. Provided you have a connection to a cluster with Lighthouse installed, the locally started controller will join the cluster, and you can test your development changes directly in-cluster.

For example to run the webhook controller locally:

make build
GIT_TOKEN=<git-token> ./bin/webhooks -namespace <namespace> -bot-name <git-bot-user>

In the case of the webhook controller, you can also test webhook deliveries locally using a ngrok tunnel. Install ngrok and start a new tunnel:

$ ngrok http 8080
ngrok by @inconshreveable                                                                                                                                                                     (Ctrl+C to quit)

Session Status                online
Account                       ***
Version                       2.3.35
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://e289dd1e1245.ngrok.io -> http://localhost:8080
Forwarding                    https://e289dd1e1245.ngrok.io -> http://localhost:8080

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

Now you can use your ngrok URL to register a webhook handler with your Git provider.

NOTE Remember to append /hook to the generated ngrok URL. In the case of the above example http://e289dd1e1245.ngrok.io/hook

Any events that happen on your Git provider are now sent to your local webhook instance.

Debugging Lighthouse

You can setup a remote debugger for Lighthouse using delve via:

dlv --listen=:2345 --headless=true --api-version=2 exec ./bin/lighthouse -- $*  

You can then debug from your Go-based IDE (e.g. GoLand / IDEA / VS Code).

Debugging webhooks

If you want to debug lighthouse locally from webhooks in your cluster there are a couple of tools that could help:

Localizer

If you install localizer (see the blog for more detail you can easily debug webhooks on your cluster.

  • first run localizer:
sudo localizer

Then run/debug lighthouse locally.

e.g. in your IDE run the cmd/webhooks/main.go (passing --namespace jx as program arguments)

Then to get the webhooks to trigger your local process:

localizer expose jx/hook --map 80:8080

when you have finished debugging, return things to normal via:

localizer expose jx/hook --stop

Telepresence

You can replace the running version in your cluster with the one running locally using telepresence.
First install the telepresence cli on your device then the traffic-manager into your cluster For webhooks, just run:

telepresence intercept hook --namespace=jx --port 80 --env-file=/tmp/webhooks-env

in another terminal:

export $(cat /tmp/webhooks-env | xargs)
dlv --listen=:2345 --headless=true --api-version=2 exec ./bin/webhooks -- --namespace=jx

You can do the same for any other deployment (keeper, foghorn...), just make sur to check the command args used for it an set them instead of --namespace=jx.

to stop intercepting:

telepresence leave hook-jx # hook-jx is the name of the intercept

Using a local go-scm

If you are hacking on support for a specific Git provider, you may find yourself working on the Lighthouse code and the jenkins-x/go-scm code together. Go modules lets you easily swap out the version of a dependency with a local copy of the code; so you can edit code in Lighthouse and jenkins-x/go-scm at the same time.

Just add this line to the end of your go.mod file:

replace github.com/jenkins-x/go-scm => /workspace/go/src/github.com/jenkins-x/go-scm

Using the exact path to where you cloned jenkins-x/go-scm

lighthouse's People

Contributors

abayer avatar ankitm123 avatar cameronbraid avatar ccojocar avatar chrissena avatar dependabot[bot] avatar dippynark avatar garethjevans avatar hferentschik avatar jenkins-x-bot avatar jenkins-x-bot-test avatar jordangoasdoue avatar joshuasimon-taulia avatar jstrachan avatar juneezee avatar linuxsuren avatar msvticket avatar osamamagdy avatar patrickleet avatar pow-devops2020 avatar rajatgupta24 avatar randomnoise avatar rawlingsj avatar skisocks avatar slimm609 avatar soulseen avatar tomhobson avatar vbehar avatar wbrefvem avatar yelhouti avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lighthouse's Issues

Allow to request additional approval

In some cases, you are not comfortable with merging/approving a change. You want to merge it, but would like another pair of eyes on it.

Adding \lgtm would merge the PR directly. Instead, you would like to do something like:

\lgtm
\request-second-approval

Create the skeleton of the projects

Define the initial skeleton of the project which should be imported in jenkins-x production cluster and should be built.

This is essentially a minimalistic HTTP server which will expose some endpoint for Git provider to post events into it.

Cleanup the prototype

Cleanup the prototype and review the changes done to get the prow's hook working with git-scm. This does not include any refactoring related to jx interface. This is going to be address by #61.

Add BDD tests to lighthouse repo

We are currently only running BDD tests against lighthouse in the jenkins-x-versions repo. Running them on PR builds in this repo will allow us to capture errors before they worm their way into the release process.

Define an interface for meta-pipeline client

It would be nice to have a client with a clean interface to start/stop the meta-pipeline. This can be defined for now in jx.

The interface will define the required data struct and the actions which can be performed.

@hferentschik you can assign this to you if you want to work on it.

Load the plugin configuration from the Schedule CRD

we need to load the Schedule configuration for the current SourceRepository when a webhook comes in to know what plugins/configuration to use. Right now in the PoC we're using an environment variable ;)

Quiet period before triggering jobs

For example, the https://github.com/jenkins-x/jx repo triggers four different tests, some create clusters, these are all triggered as soon as the PR is created. We also have "Hound" which is a hosted service to check style guide violations. It's fairly common to raise a PR, have all the jobs triggered and then Hound report you are missing a comment or something, this is a quick change and folks will fix and push the update which triggers another set of builds.

It would be nicer if we could configure a quiet period so that we avoid the first round of jobs being triggered so we can catch any hound (or other checks) violations before they are triggered.

Job has not succeeded message

At the moment, it's possible to receive the message "Job has not succeeded with "wrong" contexts in the "config" ConfigMap (e.g. capital case paths).
It happened to me that my organization in Github is "ABCD", config's contexts are registering the right path (with capital case), but Lighthouse/Tide are spitting the "Not succeeded" message even if the Tekton TaskRun was successful (probably not able to identify it).
Not sure if this is a bug or configuration issue, but it might happen pretty easily.

lighthouse image: gcr.io/jenkinsxio/lighthouse:0.0.208

Github status events failing

{"level":"error","msg":"failed to parse webhook: Unknown webhook event: status.","time":"2019-11-13T17:59:00Z"}
{"level":"info","msg":"500 Internal Server Error: Failed to parse webhook: Unknown webhook event: status.","response":"500 Internal Server Error: Failed to parse webhook: Unknown webhook event: status.","status-code":500,"time":"2019-11-13T17:59:00Z"}

Move status updates from jx build controller to Lighthouse

This is a bit of a thorny issue - there are some concerns I can't go into detail on here. But the goal here is to stop doing the status updates at the end of a pipeline's execution in jx's build controller and instead do it in LH. This would be much cleaner, and I think would be better overall as a result of colocating all the GitHub/BitBucket/Gitlab/etc API interactions here.

I'm not sure yet what the implementation would entail - if we want to go in the direction of making Lighthouse able to handle triggering things besides just jx pipelines, we'd definitely need a ProwJob equivalent here, but my instinct is to lean into the jx-specificness of LH and instead add a field to PipelineActivity named something like Reported, and then watch for PipelineActivitys in LH that have a non-nil CompletedTimestamp and Reported as false, reporting status for such PipelineActivitys and updating their Reported field.

Run/require certain contexts based on files changed in PR

Right now, we've got Prow's behavior of having contexts either be required or not required, with no granularity. A really, really nice feature would be to say "the frontend context is only required and should only be run if files in the ui directory have changed in this PR", or "if the only file(s) changed in this PR are in the examples subdirectory, don't require/run any contexts".

Implement a clean integration with jx meta-pipeline

Integrate the jx meta-pipeline client from #61 in Ligthouse.

Use meta-pipeline client into the Plumber instead of calling the jx command options.

func (b *PipelineBuilder) Create(request *PipelineOptions) (*PipelineOptions, error) {

The meta-client interface can be found here pkg/tekton/metapipeline/client.go and see the PR which implemented the client jenkins-x/jx#5051. An example is already available in jx in pipelinerunner https://github.com/jenkins-x/jx/pull/5051/files#diff-cfadf679d93b353852ef9097113f3e53

Webhook tests should use faked k8s client

Currently we're using factories to create the various clients, which means the tests will connect to whatever k8s context is configured. Instead, they should use a faked client provided by client-go.

Lighthouse should report to SCM provider commit status for errors creating metapipeline

Prow creates a "pending" commit status as soon as it triggers a pipeline, and if that pipeline never actually runs, or fails to kick off for whatever reason, that status will be marked as failed. Right now, as best as I can tell, Lighthouse does not do that. Since all status reporting is handled in jx's build controller, nothing is reported until there's a pod.

Now, this gap is much less important with metapipeline than it was previously - nearly every pipeline gets kicked off, even if it fails in the metapipeline (due to merge conflicts, etc). But it's still theoretically possible for something to go wrong creating the metapipeline's Tekton CRDs and/or PipelineActivity, resulting in no commit status ever getting created and no feedback for the user, let alone useful feedback. At a minimum, we should be creating a failure status if we're unable to create the pipeline.

Can't build with Go 1.13

Following the README.md instructions to build causes the following error:

make build

generates this error after a fresh a clone:

GO111MODULE=on go build -i -ldflags "-X github.com/jenkins-x/lighthouse/pkg/version.Version='-dev+'" -o bin/lighthouse pkg/main/main.go 
go: k8s.io/[email protected]+incompatible: invalid pseudo-version: preceding tag (v2.0.0-alpha.0) not found
make: *** [build] Error 1

Enable test coverage and identify which areas needs more testing

Enable test coverage and identify which area needs more unit testing after that we should create an issues with more details which needs more testing.

Enable codeconv into the pipeline file and add a secret for its token that the test reports can be uploaded.

webhook handler is firing 2x per event

I have vague recollections of this happening once before. Basically a single webhook from GitHub is getting processed twice. This causes the bdd tests to fail, since, e.g. in the golang http test, the expectation in at least one place is that only a single build exists.

Describe relationship to Tide

https://github.com/jenkins-x/lighthouse/#comparisons-to-prow does not mention Tide, one of the most significant Prow plugins. In particular, its feature of batch-merging PRs whose head commit has already been vetted is a major advantage. It is not obvious what, if any, equivalent Lighthouse offers. (As I recall, Tide runs as a background service rather than as a Hook plugin, so it would not fit into the Lighthouse model unless it were redesigned to react to events.) This should be discussed in the README.

jx does not support gitlab v4 REST API

so we should maybe look at using creating a gitProvider in jx using go-scm which already supports gitlab v4.

If this works we could then consider switching folks over to a go-scm based implementation of the jx pkg/gits providers to simplify the codebase and trim the dependences in jx

BDD/E2E tests for ChatOps

our current BDD tests test that we can create projects, promote them and create PRs against them and have previews created via the different git providers.

However we've no BDD tests that verify the various ChatOps commands like /lgtm /approve /test all and the like work on the different git providers.

We should add some BDD tests for those things on the created PR in the BDD test

e2e tests for LGTM plugin

Expanding on #64, we should have tests that verify that LGTM is working as expected. This would involve creating a PR, then adding an LGTM command as a comment, then verifying that the expected behavior occurred depending on configuration (i.e., the PR is labeled and put in the merge pool, or not depending on the test user's access level).

Test PR builds on go-scm

Currently, we're only testing releases. At a minimum, we should run the existing unit tests on each PR build.

Define contexts for BDD tests

In order to prevent bloat in jenkins-x.yaml, we should define separate prow contexts for running BDD tests against GitHub, GHE, etc.

implement bitbucket server in the go-scm library

the bitbucket server support is lacking in the jenkins-x/go-scm library we are using so lets add a new provider.

there's lots of great examples of tests of all the different APIs we need in the other providers (e.g. github).

in terms of priorities obviously parsing push webhooks is first; then pushing of issue/pr comment webhooks then probably APIs for things like listing labels / adding/removing labels, listing PR comments / changes etc for prow-like commands

Errors in the LightHouse and Tide Pods when creating a Pull/Merge Request on Gitlab. Serverless Jenkins X Pipeline for Preview Environment is not created.

Hi,

Apologies if this isn't the right repository to create this issue.

I've been using Serverless Jenkins X with Gitlab, as a result I've also been using LightHouse and Tide. I've tried creating a pipeline by creating a Pull/Merge Requests, but after following these steps

  1. Import my project with jx import
  2. Create a new branch (using the GitLab GUI)
  3. Make a change
  4. Create a merge request, to merge the new branch into master (using the Gitlab GUI)

I run into the following errors in the LightHouse and Tide Pods.

Errors in the LightHouse Pod

The errors in question:

  • "error":"could not get .generated_files: 404 Commit Not Found"
  • "error":"could not check membership: error in IsCollaborator: "
  • "error","msg":"Failed to unmarshal aliases from \"/tmp/git895832845/OWNERS_ALIASES\". Using empty alias map."
  • "error","msg":"Failed to list collaborators while loading RepoOwners. Skipping collaborator filtering.
  • "error":"failed to get issue comments for USER/serviceA#5: 404 Not found"

The logs are below, all the errors are in the bottom half

$ kubectl logs LIGHTHOUSE-POD -n jx
{"level":"info","msg":"about to parse webhook","time":"2019-10-11T15:39:24Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","level":"info","msg":"invoking Push handler","time":"2019-10-11T15:39:24Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","head":"","level":"info","msg":"Push event.","org":"USER","ref":"refs/heads/PR-3","repo":"serviceA","time":"2019-10-11T15:39:24Z"}
{"level":"info","msg":"found plugins approve, assign, blunderbuss, help, hold, lgtm, lifecycle, override, size, trigger, wip, heart, cat, dog, pony\n","time":"2019-10-11T15:39:24Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","count":"1","head":"","level":"info","msg":"number of push handlers","org":"USER","ref":"refs/heads/PR-3","repo":"serviceA","time":"2019-10-11T15:39:24Z"}
{"level":"info","msg":"about to parse webhook","time":"2019-10-11T15:39:40Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","level":"info","msg":"invoking Push handler","time":"2019-10-11T15:39:40Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","head":"","level":"info","msg":"Push event.","org":"USER","ref":"refs/heads/PR-3","repo":"serviceA","time":"2019-10-11T15:39:40Z"}
{"level":"info","msg":"found plugins approve, assign, blunderbuss, help, hold, lgtm, lifecycle, override, size, trigger, wip, heart, cat, dog, pony\n","time":"2019-10-11T15:39:40Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","count":"1","head":"","level":"info","msg":"number of push handlers","org":"USER","ref":"refs/heads/PR-3","repo":"serviceA","time":"2019-10-11T15:39:40Z"}

{"level":"info","msg":"about to parse webhook","time":"2019-10-11T15:40:31Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","level":"info","msg":"invoking PR handler","time":"2019-10-11T15:40:31Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","author":"USER","level":"info","msg":"Pull request opened.","org":"USER","pr":5,"repo":"serviceA","time":"2019-10-11T15:40:31Z","url":"https://GITLAB_URL/USER/serviceA/merge_requests/5"}
{"level":"info","msg":"found plugins approve, assign, blunderbuss, help, hold, lgtm, lifecycle, override, size, trigger, wip, heart, cat, dog, pony\n","time":"2019-10-11T15:40:31Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","author":"USER","count":"5","level":"info","msg":"number of PR handlers","org":"USER","pr":5,"repo":"serviceA","time":"2019-10-11T15:40:31Z","url":"https://GITLAB_URL/USER/serviceA/merge_requests/5"}
{"level":"info","msg":"found plugins approve, assign, blunderbuss, help, hold, lgtm, lifecycle, override, size, trigger, wip, heart, cat, dog, pony\n","time":"2019-10-11T15:40:31Z"}
{"client":"git","level":"info","msg":"Cloning USER/serviceA for the first time.","time":"2019-10-11T15:40:46Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","author":"USER","error":"could not get .generated_files: 404 Commit Not Found","level":"error","msg":"Error handling PullRequestEvent.","org":"USER","plugin":"size","pr":5,"repo":"serviceA","time":"2019-10-11T15:40:48Z","url":"https://GITLAB_URL/USER/serviceA/merge_requests/5"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","author":"USER","error":"could not check membership: error in IsCollaborator: ","level":"error","msg":"Error handling PullRequestEvent.","org":"USER","plugin":"trigger","pr":5,"repo":"serviceA","time":"2019-10-11T15:40:49Z","url":"https://GITLAB_URL/USER/serviceA/merge_requests/5"}
{"client":"git","level":"info","msg":"Checkout master.","time":"2019-10-11T15:42:12Z"}
{"base":"master","client":"repoowners","error":"error unmarshaling JSON: while decoding JSON: json: cannot unmarshal array into Go struct field .aliases of type map[string][]string","level":"error","msg":"Failed to unmarshal aliases from \"/tmp/git895832845/OWNERS_ALIASES\". Using empty alias map.","org":"USER","repo":"serviceA","time":"2019-10-11T15:42:12Z"}
{"base":"master","client":"repoowners","error":"","level":"error","msg":"Failed to list collaborators while loading RepoOwners. Skipping collaborator filtering.","org":"USER","repo":"serviceA","time":"2019-10-11T15:42:12Z"}
{"Branch":"master","Clone":"https://GITLAB_URL/USER/serviceA.git","CloneSSH":"git@GITLAB_URL:USER/serviceA.git","ID":"0","Link":"https://GITLAB_URL/USER/serviceA","Name":"serviceA","Namespace":"USER","author":"USER","error":"failed to get issue comments for USER/serviceA#5: 404 Not found","level":"error","msg":"Error handling PullRequestEvent.","org":"USER","plugin":"approve","pr":5,"repo":"serviceA","time":"2019-10-11T15:42:13Z","url":"https://GITLAB_URL/USER/serviceA/merge_requests/5"}

Errors in the Tide Pod

The error in question: no GraphQL graphql supported for git provider gitlab. The logs are below.

$ kubectl logs TIDE-POD -n jx

{"component":"tide","file":"/go/src/github.com/jenkins-x/lighthouse/pkg/prow/gitprovider/issues.go:48","func":"github.com/jenkins-x/lighthouse/pkg/prow/gitprovider.(*Client).Query","level":"warning","msg":"no GraphQL graphql supported for git provider gitlab","time":"2019-10-11T16:15:06Z"}
{"component":"tide","controller":"sync","duration":"340.393µs","file":"/go/src/github.com/jenkins-x/lighthouse/pkg/tide/tide.go:282","func":"github.com/jenkins-x/lighthouse/pkg/tide.(*Controller).Sync.func1","level":"info","msg":"Synced","time":"2019-10-11T16:15:06Z"}
{"component":"tide","file":"/go/src/github.com/jenkins-x/lighthouse/pkg/prow/gitprovider/issues.go:48","func":"github.com/jenkins-x/lighthouse/pkg/prow/gitprovider.(*Client).Query","level":"warning","msg":"no GraphQL graphql supported for git provider gitlab","time":"2019-10-11T16:15:06Z"}
{"component":"tide","controller":"status-update","duration":"160.365µs","file":"/go/src/github.com/jenkins-x/lighthouse/pkg/tide/status.go:452","func":"github.com/jenkins-x/lighthouse/pkg/tide.(*statusController).sync.func1","level":"info","msg":"Statuses synced.","time":"2019-10-11T16:15:06Z"}

Expected Behaviour

I expect a pipeline to be created that will deploy the service to a Preview environment.

My Configurations

These are my configurations (the relevant files), should I expect Pull Requests on Gitlab to create a pipeline and deploy to a Preview environment with these configurations?

jenkins-x-boot-config/jx-requirements.yml file

# jx-requirements.yml
autoUpdate:
  enabled: false
  schedule: ""
cluster:
  clusterName: test-cluster
  environmentGitOwner: owner
  gitKind: gitlab
  gitName: test
  gitServer: GITLAB_URL
  namespace: jx
  project: serverless-jx-tests
  provider: gke
  zone: europe-west2-b
environments:
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: dev
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: staging
- ingress:
    domain: ""
    externalDNS: false
    namespaceSubDomain: ""
    tls:
      email: ""
      enabled: false
      production: false
  key: production
gitops: true
ingress:
  cloud_dns_secret_name: external-dns-gcp-sa
  domain: DOMAIN
  externalDNS: true
  namespaceSubDomain: -jx.
  tls:
    email: ""
    enabled: false
    production: false
kaniko: true
secretStorage: vault
storage:
  backup:
    enabled: false
    url: ""
  logs:
    enabled: false
    url: ""
  reports:
    enabled: false
    url: ""
  repository:
    enabled: false
    url: ""
vault: {}
velero: {}
versionStream:
  ref: master
  url: https://github.com/jenkins-x/jenkins-x-versions.git
webhook: lighthouse

MyProject/jenkins-x.yml file

# jenkins-x.yml

pipelineConfig:
  agent:
    image: nodejs
    label: jenkins-nodejs
  pipelines:
    pullRequest:
      pipeline:
        agent:
          image: nodejs
        stages:
        - name: build-project
          steps: 
          - command: mkdir temp -p
            name: make-temp-dir
            dir: /workspace
          - command: curl -Lo hugo.tar.gz https://github.com/gohugoio/hugo/releases/download/v0.32/hugo_0.32_Linux-64bit.tar.gz
            dir: /workspace/temp
            name: curl-hugo-release
          - command: tar xvzf hugo.tar.gz
            dir: /workspace/temp
            name: unzip-hugo-tar-gz
          - command: ./hugo version
            dir: /workspace/temp
            name: output-hugo-version                
          - command: ./hugo -s /workspace/source
            name: generate-website
            dir: /workspace/temp
        - name: everything-else
          steps:
          - args:
            - --cache=true
            - --cache-dir=/workspace
            - --context=/workspace/source
            - --dockerfile=/workspace/source/Dockerfile
            - --destination=DOCKER_REGISTRY/REPLACE_ME_DOCKER_REGISTRY_ORG/serviceA:${inputs.params.version}
            - --cache-repo=DOCKER_REGISTRY/REPLACE_ME_DOCKER_REGISTRY_ORG/cache
            command: /kaniko/executor
            dir: /workspace/source
            image: gcr.io/kaniko-project/executor:9912ccbf8d22bbafbf971124600fbb0b13b9cbd6
            name: build-container-build
          - command: jx step post build --image DOCKER_REGISTRY/REPLACE_ME_DOCKER_REGISTRY_ORG/serviceA:${VERSION}
            name: postbuild-post-build
          - command: make preview
            dir: /workspace/source/charts/preview
            name: promote-make-preview
          - command: jx preview --app $APP_NAME --dir ../..
            dir: /workspace/source/charts/preview
            name: promote-jx-preview
    release:
      pipeline:
        agent: 
          image: nodejs
        stages:
        - name: build-project
          steps: 
          - command: ls -l 
            name: ls-source-dir
            dir: /workspace/source
          - command: mkdir temp -p
            name: make-temp-dir
            dir: /workspace
          - command: curl -Lo hugo.tar.gz https://github.com/gohugoio/hugo/releases/download/v0.32/hugo_0.32_Linux-64bit.tar.gz
            dir: /workspace/temp
            name: curl-hugo-release
          - command: tar xvzf hugo.tar.gz
            dir: /workspace/temp
            name: unzip-hugo-tar-gz
          - command: ./hugo version
            dir: /workspace/temp
            name: output-hugo-version                
          - command: ./hugo -s /workspace/source
            name: generate-website
            dir: /workspace/temp
        - name: everything-else
          steps:
          - args:
            - --cache=true
            - --cache-dir=/workspace
            - --context=/workspace/source
            - --dockerfile=/workspace/source/Dockerfile
            - --destination=DOCKER_REGISTRY/REPLACE_ME_DOCKER_REGISTRY_ORG/serviceA:${inputs.params.version}
            - --cache-repo=DOCKER_REGISTRY/cache
            command: /kaniko/executor
            dir: /workspace/source
            image: gcr.io/kaniko-project/executor:9912ccbf8d22bbafbf971124600fbb0b13b9cbd6
            name: build-container-build
          - command: jx step post build --image DOCKER_REGISTRY/REPLACE_ME_DOCKER_REGISTRY_ORG/serviceA:${VERSION}
            name: build-post-build
          - command: jx step changelog --version v${VERSION}
            name: promote-changelog
          - command: jx step helm release
            dir: /workspace/source/charts/serviceA
            name: promote-helm-release
          - command: jx promote -b --all-auto --timeout 2m --no-wait  --version ${VERSION}
            dir: /workspace/source/charts/serviceA
            name: promote-jx-promote

MyProject/OWNERS file

approvers:
- USER
reviewers:
- USER

MyProject/OWNERS_ALIASES file

aliases:
- USER
best-approvers:
- USER
best-reviewers:
- USER

JX Version

NAME               VERSION
jx                 2.0.837
Kubernetes cluster v1.13.7-gke.24
kubectl            v1.16.1
helm client        Client: v2.13.1+g618447c
git                2.17.1
Operating System   Ubuntu 18.04.3 LTS

Validate the GitHub webhook and Chatops commands

Validate the GitHub webhook and Chatops (plugins) works as expected.

Create a Go project and try the following use case:

  • push a change in the master and check that the release pipeline triggers
  • create a PR and execute /test this and check that the pullRequest pipeline triggers

Add webhook tests

There are a number of methods in the webhook package that should have unit tests. Some refactoring may be necessary to expose certain methods, unless we can get away with putting the tests in the same package (i.e., not a separate webhook_test package).

work-in-progress label chatops test fails for github.com and GHE providers

This slipped through cracks, but I verified that with GitHub and GHE providers, when we run the quickstart BDD test and therefore invoke the chatops test code, the test trying to apply the do-not-merge/work-in-progress label fails. The label tests before it work, but not that one. See #452 for failing builds.

We don't even get to that point with Gitlab or Bitbucket Server - BBS fails 'cos we don't currently support creating an issue there in the first place, and Gitlab seems to barf on the creation of the issue.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.