jenkinsci / ci.jenkins.io-runner Goto Github PK

An emulator of ci.jenkins.io instance, packaged as a single-shot Jenkinsfile Runner

License: MIT License

Dockerfile 43.52% Makefile 18.64% Groovy 37.83%

ci.jenkins.io-runner's Introduction

ci.jenkins.io-runner

This project offers environment for running Jenkinsfile instances from ci.jenkins.io locally. It is powered by Jenkinsfile Runner and the experimental JFR Maven packaging flow introduced in 1.0-beta-16. If you want a classic runtime Jenkins master with agents, checkout my Jenkins Configuration-as-code demo.

The runner can execute buildPlugin() builds and some other commands from the Jenkins Pipeline Library. In particular, it is possible to run builds against multiple JDK and Jenkins core version combinations.

See the Limitations section below for some of known limitations.

Quickstart

Checkout this repo
Run make docker to build the base image
Run make run to run a simple demo
Run make demo-plugin to run a demo of the plugin build

Usage

The runner can be invoked against a workspace which contains a Jenkinsfile and, if needed, the project's sourcecode.

	docker run --rm -v maven-repo:/root/.m2 \
	    -v $(pwd)/demo/locale-plugin/:/workspace/ \
	    onenashev/ci.jenkins.io-runner

Developing Jenkins Pipeline Library

Jenkins Pipeline library may be passed from a volume so that it is possible to test a local snapshot.

	docker run --rm -v maven-repo:/root/.m2 \
	    -v ${MY_PIPELINE_LIBRARY_DIR}:/var/jenkins_home/pipeline-library \
	    -v $(pwd)/demo/locale-plugin/:/workspace/ \
	    onenashev/ci.jenkins.io-runner

Developer notes

Upgrade management

This repository uses Dependabot to track dependencies and to propose updates. Many plugin and library dependencies actually come from Bills of Materials supplied by the JFR packaging parent POM: Jenkins Core BOM and Jenkins Plugin BOM. It reduces the number of moving parts by consuming the cross-verified plugin versions.

Debugging Jenkinsfile Runner

To debug the execution, you can pass the JFR_LOCAL_WORKSPACE=true environment variable to the image. It will make the builder to execute Pipeline directly. It is also possible to debug Jenkinsfile Runner and Groovy init hooks by passing the remote debug options and exposing the debug port:

	docker run --rm -v maven-repo:/root/.m2 \
	    -v $(pwd)/demo/locale-plugin/:/workspace/ \
	    -p 5005:5005 -e JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=y" \
	    onenashev/ci.jenkins.io-runner

Profiling Jenkinsfile Runner

This repository supports profiling of Jenkinsfile Runner with Java Flight Recorder. Due to performance reasons, profiling happens on a local machine instead of the Docker containers.

To run profiling on a Unix machine:

Build ci.jenkins.io-runner locally via mvn clean package
Run make jfr-profile and wait till completion
Retrieve the demo/locale-plugin/work/recording.jfr file with Java Flight Recorder dump
Use performance analysis tools to analyze the Java Flight Recorder dump (e.g. IntelligIDEA, JDK Mission Control in AdoptOpenJDK). CPU and memory usage analysis can be done with the existing tools.

Limitations

This project has just started, so it has some downsides being compared to the runtime Pipeline Development instance here. All of the limitations below can be improved in the future.

A custom fork of Jenkins Pipeline Library is needed to run it
- Follow jenkins-infra/pipeline-library#78
ci.jenkins.io-runner is a single-container package with only 1 executor
Only JDK8 and JDK11 are provided in the image
Windows steps are not supported
Docker image is pretty big, because it bundles two versions of JDK

ci.jenkins.io-runner's People

Contributors

Stargazers

Watchers

Forkers

markjacksonfishing kouzouigh fcojfernandez jvz madchap oleg-nenashev lili2311 henri-fang xaseron v1v makrgroup amitrintzler adre9 lemeurherve

ci.jenkins.io-runner's Issues

Add support of building everything with a single Dockerfile

The current build flow uses local Custom WAR Packager, so it is hard to automate packaging of images. It would be great to just put everything into a single Docker flow and to automate the build with DockerHub or other service like Jenkins or Jenkins X.

Enable continuous delivery of version tags on ci.jenkins.io or DockerHub

Currently the tag releases are not being automatically built and deployed. For better usability, we should enable it somehow

make clean build doesn't work out of the box

Hi @oleg-nenashev

I want to try this project, because it seem to be very helpful

I follow the guide, and run make clean build but I have a first error

$ make clean build
rm -rf tmp
java -jar .build/cwp-cli-1.5.jar \
	     -configPath packager-config.yml -version 256.0-test ""
 --batch-mode          : Enables the batch mode for the build (default: false)
 --bomPath FILE        : Path to the BOM file. If defined, it will override
                         settings in Config YAML
 --environment VAL     : Environment to be used
 --installArtifacts    : If set, the final artifacts will be automatically
                         installed to the local repository (current version -
                         only WAR) (default: false)
 -configPath FILE      : Path to the configuration YAML. See the tool's README
                         for format (default: packager-config.yml)
 -demo                 : Enables demo mode with predefined config file
                         (default: false)
 -mvnSettingsFile FILE : Path to a custom Maven settings file to be used within
                         the build
 -tmpDir FILE          : Temporary directory for generated files and the output
                         WAR. Defaults to 'tmp'
 -version VAL          : Version of WAR to be set. Defaults to '1.0-SNAPSHOT'
                         (default: 256.0-test)
Exception in thread "main" java.io.IOException: Failed to read command-line arguments
	at io.jenkins.tools.warpackager.cli.Main.main(Main.java:26)
Caused by: org.kohsuke.args4j.CmdLineException: No argument is allowed: 
	at org.kohsuke.args4j.CmdLineParser.parseArgument(CmdLineParser.java:509)
	at io.jenkins.tools.warpackager.cli.Main.main(Main.java:23)
make: *** [build] Error 1

To fix this I have delete ${CWP_OPTS} part of the build section in Makefile

So next I have the following error, but this time I can't fix it :

Exception in thread "main" java.io.IOException: CasC section is declared, but CasC plugin is not declared in the plugins list
	at io.jenkins.tools.warpackager.lib.impl.Builder.verifyConfig(Builder.java:66)
	at io.jenkins.tools.warpackager.lib.impl.Builder.build(Builder.java:97)
	at io.jenkins.tools.warpackager.cli.Main.main(Main.java:56)
make: *** [build] Error 1

I put the full trace if needed
build.txt

Do you know why ?

Thx you

What error do I get?

 make run
docker run --rm -v maven-repo:/root/.m2 \
	    -v /Users/vmartinez/work/src/github.com/v1v/ci.jenkins.io-runner/demo/simple/:/workspace/ \
	    jenkins4eval/ci.jenkins.io-runner:local-test 
   3.806 [id=29]	SEVERE	jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Docker Commons Plugin v1.15 (docker-commons)
java.io.IOException: Docker Commons Plugin version 1.15 failed to load.
 - Pipeline: Declarative Extension Points API version 1.3.2 is older than required. To fix, install version 1.3.8 or later.
	...
java.io.IOException: Docker Pipeline version 1.14 failed to load.
 - Docker Commons Plugin version 1.15 failed to load. Fix this plugin first.
	...
java.io.IOException: Pipeline: Declarative version 1.3.2 failed to load.
 - Docker Pipeline version 1.14 failed to load. Fix this plugin first.
	...
java.io.IOException: Pipeline version 2.6 failed to load.
 - Pipeline: Declarative version 1.3.2 failed to load. Fix this plugin first.
	...
java.io.IOException: Yet Another Docker Plugin version 0.1.0-rc52 failed to load.
 - Docker Commons Plugin version 1.15 failed to load. Fix this plugin first.
	...
== Configuring the Jenkins Pipeline library
===== Using the Pipeline library from https://github.com/oleg-nenashev/pipeline-library 
...
java.lang.NoSuchMethodError: No such DSL method 'pipeline' found among steps

#127
#127 # How to reproduce?

make docker
make clean build

Edit demo/simple/Jenkinsfile with the below snippet:

#!/usr/bin/env groovy
pipeline {
    agent {
        label "java"
    }
    tools {
        maven 'mvn'
        jdk 'jdk8'
    }
    options {
        timestamps()
    }
    stages {
        stage('Checkout') {
            steps {
                deleteDir()
                git url: 'https://github.com/v1v/pipeline-library', branch: 'master'                    
            }
        }
        stage('Test') {
            steps {
                sh 'mvn clean test'
            }
            post {
                always {
                    junit(keepLongStdio: true, testResults: "target/surefire-reports/junit-*.xml,target/surefire-reports/TEST-*.xml")
                }
            }
        }
    }
}

Then make run

I was able to get the above issue using my fork repo: https://github.com/v1v/ci.jenkins.io-runner
which it's up to date with the commit f43c078

Please let me know if you need further details.

Unable to use a local library

When using the docker image if I pass in my local lib it fails to load because my library doesn't have a vars directory.

If I add a vars directory then it still fails.

== Configuring the Jenkins Pipeline library
===== Adding local Pipeline libs
2021-03-10 08:19:07.021+0000 [id=26]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class jenkins.plugins.git.GitSCMSource#owner: type is abstract but not Describable.
Started
Resume disabled by user, switching to high-performance, low-durability mode.
ERROR: Could not find any definition of libraries [pipeline-library]
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: Loading libraries failed

1 error

        at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
        at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
        at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
        at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
        at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
        at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:337)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:428)
Finished: FAILURE