Giter Club home page Giter Club logo

ci's Introduction

Concourse: the continuous thing-doer.

Discord Build Contributors Help Wanted

Concourse is an automation system written in Go. It is most commonly used for CI/CD, and is built to scale to any kind of automation pipeline, from simple to complex.

booklit pipeline

Concourse is very opinionated about a few things: idempotency, immutability, declarative config, stateless workers, and reproducible builds.

The road to Concourse v10

Concourse v10 is the code name for a set of features which, when used in combination, will have a massive impact on Concourse's capabilities as a generic continuous thing-doer. These features, and how they interact, are described in detail in the Core roadmap: towards v10 and Re-inventing resource types blog posts. (These posts are slightly out of date, but they get the idea across.)

Notably, v10 will make Concourse not suck for multi-branch and/or pull-request driven workflows - examples of spatial change, where the set of things to automate grows and shrinks over time.

Because v10 is really an alias for a ton of separate features, there's a lot to keep track of - here's an overview:

Feature RFC Status
set_pipeline step #31 ✔ v5.8.0 (experimental)
Var sources for creds #39 ✔ v5.8.0 (experimental), TODO: #5813
Archiving pipelines #33 ✔ v6.5.0
Instanced pipelines #34 ✔ v7.0.0 (experimental)
Static across step 🚧 #29 ✔ v6.5.0 (experimental)
Dynamic across step 🚧 #29 ✔ v7.4.0 (experimental, not released yet)
Projects 🚧 #32 🙏 RFC needs feedback!
load_var step #27 ✔ v6.0.0 (experimental)
get_var step #27 🚧 #5815 in progress!
Prototypes #37 ⚠ Pending first use of protocol (any of the below)
run step 🚧 #37 ⚠ Pending its own RFC, but feel free to experiment
Resource prototypes #38 🙏 #5870 looking for volunteers!
Var source prototypes 🚧 #6275 planned, may lead to RFC
Notifier prototypes 🚧 #28 ⚠ RFC not ready

The Concourse team at VMware will be working on these features, however in the interest of growing a healthy community of contributors we would really appreciate any volunteers. This roadmap is very easy to parallelize, as it is comprised of many orthogonal features, so the faster we can power through it, the faster we can all benefit. We want these for our own pipelines too! 😆

If you'd like to get involved, hop in Discord or leave a comment on any of the issues linked above so we can coordinate. We're more than happy to help figure things out or pick up any work that you don't feel comfortable doing (e.g. UI, unfamiliar parts, etc.).

Thanks to everyone who has contributed so far, whether in code or in the community, and thanks to everyone for their patience while we figure out how to support such common functionality the "Concoursey way!" 🙏

Installation

Concourse is distributed as a single concourse binary, available on the Releases page.

If you want to just kick the tires, jump ahead to the Quick Start.

In addition to the concourse binary, there are a few other supported formats. Consult their GitHub repos for more information:

Quick Start

$ wget https://concourse-ci.org/docker-compose.yml
$ docker-compose up
Creating docs_concourse-db_1 ... done
Creating docs_concourse_1    ... done

Concourse will be running at 127.0.0.1:8080. You can log in with the username/password as test/test.

⚠️ If you are using an M1 mac: M1 macs are incompatible with the containerd runtime. After downloading the docker-compose file, change CONCOURSE_WORKER_RUNTIME: "containerd" to CONCOURSE_WORKER_RUNTIME: "houdini". This feature is experimental

Next, install fly by downloading it from the web UI and target your local Concourse as the test user:

$ fly -t ci login -c http://127.0.0.1:8080 -u test -p test
logging in to team 'main'

target saved

Configuring a Pipeline

There is no GUI for configuring Concourse. Instead, pipelines are configured as declarative YAML files:

resources:
- name: booklit
  type: git
  source: {uri: "https://github.com/vito/booklit"}

jobs:
- name: unit
  plan:
  - get: booklit
    trigger: true
  - task: test
    file: booklit/ci/test.yml

Most operations are done via the accompanying fly CLI. If you've got Concourse installed, try saving the above example as booklit.yml, target your Concourse instance, and then run:

fly -t ci set-pipeline -p booklit -c booklit.yml

These pipeline files are self-contained, maximizing portability from one Concourse instance to the next.

Learn More

Contributing

Our user base is basically everyone that develops software (and wants it to work).

It's a lot of work, and we need your help! If you're interested, check out our contributing docs.

ci's People

Contributors

aoldershaw avatar chenbh avatar clarafu avatar ddadlani avatar deniseyu avatar dtimm avatar estebanfs avatar infra-red avatar kcmannem avatar markstokan avatar muntac avatar nader-ziada avatar notrepo05 avatar pivotal-bin-ju avatar pvaramballypivot avatar scottietremendous avatar spimtav avatar staylor14 avatar taylorsilva avatar vito avatar wayneadams avatar xtreme-sameer-vohra avatar xtremerui avatar youssb avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ci's Issues

Move concourse-charts pipeline to prod and add failure notification

Background:

  • We have a pipeline running under hush-house that updates the merged branch in concourse/charts repo to be a merging of the helm/charts master branch and branches that we specify in this file so then we can use this branch as an RC for the Concourse chart when we have unmerged PRs.
  • This pipeline has been failing silently for some time now resulting in k8s-topgun failures.

Request:

  • This pipeline should be moved to our prod environment, so it can be easily monitored.
  • Also, would be handy to add a slack notification for that whenever it fails (or maybe adding it to the concourse pipeline itself 🤔)

k8s-topgun Failing

k8s-topgun has been failing since Friday, September 27th. No concourse code changes occurred between the last passing build and the builds that followed it, all of them failing.

GKE cluster topgun run's on: cluster-1

There are two types of errors we're seeing.

1. Port forwarding fails

For a variety of tests we keep seeing port-forwarding fail.

One example, where pod not found:

STEP: Creating the web proxy
STEP: [1569964314.490616322] running: kubectl port-forward --namespace=topgun-swi-80011428 service/topgun-swi-80011428-web :8080
error: error upgrading connection: unable to upgrade connection: pod not found ("topgun-swi-80011428-web-7d4c8fb55b-6kk56_topgun-swi-80011428")
STEP: [1569964314.650369883] running: helm delete --purge topgun-swi-80011428
release "topgun-swi-80011428" deleted
STEP: [1569964315.270037413] running: kubectl delete namespace topgun-swi-80011428 --wait=false 
namespace "topgun-swi-80011428" deleted

• Failure [75.169 seconds]
Scaling web instances
/tmp/build/7f688b19/concourse/topgun/k8s/web_scaling_test.go:12
  succeeds [It]
  /tmp/build/7f688b19/concourse/topgun/k8s/web_scaling_test.go:22

  No future change is possible.  Bailing out early after 0.155s.
  Got stuck at:

  Waiting for:
      Forwarding

second example, broken pipe:

running wget -O- topgun-dp-10084543-web:8080/api/v1/info
Handling connection for 45551
initializing
failed
Handling connection for 45551
E1001 21:07:06.913259    7232 portforward.go:372] error copying from remote stream to local connection: readfrom tcp4 127.0.0.1:45551->127.0.0.1:51636: write tcp4 127.0.0.1:45551->127.0.0.1:51636: write: broken pipe
STEP: [1569964026.936306000] running: helm delete --purge topgun-dp-10084543
release "topgun-dp-10084543" deleted
STEP: [1569964027.552978992] running: kubectl delete namespace topgun-dp-10084543 --wait=false
namespace "topgun-dp-10084543" deleted



• Failure [75.527 seconds]
DNS Resolution
/tmp/build/7f688b19/concourse/topgun/k8s/dns_proxy_test.go:14
  different proxy settings
  /tmp/build/7f688b19/gopath/pkg/mod/github.com/onsi/[email protected]/extensions/table/table.go:92
    Proxy Disabled, with short service name [It]
    /tmp/build/7f688b19/gopath/pkg/mod/github.com/onsi/[email protected]/extensions/table/table_entry.go:46

    Expected
        <int>: 1
    to be zero-valued

third example, where port-forwarding breaks down in the middle of the test:

STEP: [1569963914.924532890] running: /tmp/gexec_artifacts279253227/g904034638/fly -t concourse-topgun-k8s-4 execute -c tasks/dns-proxy-task.yml -v url=topgun-dp-11517743-web.topgun-dp-11517743.svc.cluster.local:8080/api/v1/info
Handling connection for 38901
uploading topgun done.8KiB/s))
executing build 1 at http://127.0.0.1:38901/builds/1 
initializing
fetching busybox@sha256:dd97a3fe6d721c5cf03abac0f50e2848dc583f7c4e41bf39102ceb42edfd1808
7c9d20b9b6cd [======================================] 742.9KiB/742.9KiB
running wget -O- topgun-dp-11517743-web.topgun-dp-11517743.svc.cluster.local:8080/api/v1/info
Connecting to topgun-dp-11517743-web.topgun-dp-11517743.svc.cluster.local:8080 (10.11.241.215:8080)
wget: can't connect to remote host (10.11.241.215): Connection refused
failed
Handling connection for 38901
STEP: [1569963925.802273273] running: helm delete --purge topgun-dp-11517743
release "topgun-dp-11517743" deleted
STEP: [1569963926.385312796] running: kubectl delete namespace topgun-dp-11517743 --wait=false
namespace "topgun-dp-11517743" deleted

• Failure [61.513 seconds]
DNS Resolution
/tmp/build/7f688b19/concourse/topgun/k8s/dns_proxy_test.go:14
  different proxy settings
  /tmp/build/7f688b19/gopath/pkg/mod/github.com/onsi/[email protected]/extensions/table/table.go:92
    Proxy Enabled, with full service name [It]
    /tmp/build/7f688b19/gopath/pkg/mod/github.com/onsi/[email protected]/extensions/table/table_entry.go:46

    Expected
        <int>: 1
    to be zero-valued

    /tmp/build/7f688b19/concourse/topgun/k8s/dns_proxy_test.go:85

Fourth, failing in the middle of a test, example:

STEP: [1569920401.788959742] running: /tmp/gexec_artifacts116775232/g015069087/fly -t concourse-topgun-k8s-4 login -c http://127.0.0.1:39607 -u test -p test

logging in to team 'main'

Handling connection for 39607
Handling connection for 39607
target saved
STEP: [1569920401.954967737] running: /tmp/gexec_artifacts116775232/g015069087/fly -t concourse-topgun-k8s-4 workers --json
Handling connection for 39607
[]
STEP: [1569920411.992353678] running: /tmp/gexec_artifacts116775232/g015069087/fly -t concourse-topgun-k8s-4 workers --json
Handling connection for 39607
E1001 09:00:12.060585   10831 portforward.go:400] an error occurred forwarding 39607 -> 8080: error forwarding port 8080 to pod 2102f8dd698b355c7f0cc17193d227187229e3af46245ee35dae98bd3a77c0f4, uid : exit status 1: 2019/10/01 09:00:12 socat[174757] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused
could not reach the Concourse server called concourse-topgun-k8s-4:

    Get http://127.0.0.1:39607/api/v1/info: EOF

is the targeted Concourse running? better go catch it lol

STEP: [1569920412.065531969] running: helm delete --purge topgun-bd-overlay-55649115
release "topgun-bd-overlay-55649115" deleted
STEP: [1569920412.527161837] running: kubectl delete namespace topgun-bd-overlay-55649115 --wait=false
namespace "topgun-bd-overlay-55649115" deleted

• Failure [42.312 seconds]
baggageclaim drivers
/tmp/build/7f688b19/concourse/topgun/k8s/baggageclaim_drivers_test.go:11
  GKE
  /tmp/build/7f688b19/concourse/topgun/k8s/k8s_suite_test.go:302
    ubuntu image
    /tmp/build/7f688b19/concourse/topgun/k8s/baggageclaim_drivers_test.go:36
      overlay
      /tmp/build/7f688b19/concourse/topgun/k8s/baggageclaim_drivers_test.go:46
        works [It]
        /tmp/build/7f688b19/concourse/topgun/k8s/baggageclaim_drivers_test.go:47

        Expected
            <int>: 1
        to be zero-valued

        /tmp/build/7f688b19/concourse/topgun/fly.go:100

2. Baggageclaim driver test fails

This one has been happening for most failures but it's not 100% either. When this specific test in topgun fails, it's always with this error, never the port-forwarding error. It happens for both the cos and ubuntu node selectors.
Doesn't happen in:

The test is

• Failure [302.169 seconds]

baggageclaim drivers

/tmp/build/7f688b19/concourse/topgun/k8s/baggageclaim_drivers_test.go:11
  GKE
  /tmp/build/7f688b19/concourse/topgun/k8s/k8s_suite_test.go:302
    cos image
    /tmp/build/7f688b19/concourse/topgun/k8s/baggageclaim_drivers_test.go:30
      btrfs
      /tmp/build/7f688b19/concourse/topgun/k8s/baggageclaim_drivers_test.go:72
        fails [It]
        /tmp/build/7f688b19/concourse/topgun/k8s/baggageclaim_drivers_test.go:73

        Expected
            <int>: 1
        to equal
            <int>: 0

        /tmp/build/7f688b19/concourse/topgun/k8s/k8s_suite_test.go:171

You can reproduce the error with this helm install

helm upgrade --install --force --wait --namespace topgun-btrfs-fails --set=web.livenessProbe.failureThreshold=3 --set=web.livenessProbe.initialDelaySeconds=3 --set=web.livenessProbe.periodSeconds=3 --set=web.livenessProbe.timeoutSeconds=3 --set=concourse.web.kubernetes.keepNamespaces=false --set=postgresql.persistence.enabled=false --set=image=concourse/concourse-rc --set=imageTag=latest --set=imageDigest=sha256:b61fbd930eefab9300744ca83e05a2f7bcb954ec341c996a7009014a50c8a374 --set=concourse.web.kubernetes.enabled=false --set=concourse.worker.baggageclaim.driver=btrfs --set=worker.replicas=1 --set=worker.nodeSelector.nodeImage=cos topgun-btrfs-fails /Users/pivotal/workspace/charts/stable/concourse

The worker pod fails to come up and goes into a crash loop. It's failing to create the btrfs volume. kubectl logs shows the following:

{"timestamp":"2019-10-01T21:15:25.924431822Z","level":"error","source":"baggageclaim","message":"baggageclaim.fs.run-command.failed","data":{"args":["bash","-e","-x","-c","\n\t\tif [ ! -e $IMAGE_PATH ] || [ \"$(stat --printf=\"%s\" $IMAGE_PATH)\" != \"$SIZE_IN_BYTES\" ]; then\n\t\t\ttouch $IMAGE_PATH\n\t\t\ttruncate -s ${SIZE_IN_BYTES} $IMAGE_PATH\n\t\tfi\n\n\t\tlo=\"$(losetup -j $IMAGE_PATH | cut -d':' -f1)\"\n\t\tif [ -z \"$lo\" ]; then\n\t\t\tlo=\"$(losetup -f --show $IMAGE_PATH)\"\n\t\tfi\n\n\t\tif ! file $IMAGE_PATH | grep BTRFS; then\n\t\t\tmkfs.btrfs --nodiscard $IMAGE_PATH\n\t\tfi\n\n\t\tmkdir -p $MOUNT_PATH\n\n\t\tif ! mountpoint -q $MOUNT_PATH; then\n\t\t\tmount -t btrfs $lo $MOUNT_PATH\n\t\tfi\n\t"],"command":"/bin/bash","env":["PATH=/usr/local/concourse/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","MOUNT_PATH=/concourse-work-dir/volumes","IMAGE_PATH=/concourse-work-dir/volumes.img","SIZE_IN_BYTES=10266165248"],"error":"exit status 32","session":"3.1","stderr":"+ '[' '!' -e /concourse-work-dir/volumes.img ']'\n++ stat --printf=%s /concourse-work-dir/volumes.img\n+ '[' 10266165248 '!=' 10266165248 ']'\n++ losetup -j /concourse-work-dir/volumes.img\n++ cut -d: -f1\n+ lo=/dev/loop4\n+ '[' -z /dev/loop4 ']'\n+ file /concourse-work-dir/volumes.img\n+ grep BTRFS\n+ mkdir -p /concourse-work-dir/volumes\n+ mountpoint -q /concourse-work-dir/volumes\n+ mount -t btrfs /dev/loop4 /concourse-work-dir/volumes\nmount: /concourse-work-dir/volumes: unknown filesystem type 'btrfs'.\n","stdout":"/concourse-work-dir/volumes.img: BTRFS Filesystem sectorsize 4096, nodesize 16384, leafsize 16384, UUID=226829f7-4ca4-41ed-8377-794b6095ef94, 114688/10266165248 bytes used, 1 devices\n"}}
{"timestamp":"2019-10-01T21:15:25.925165479Z","level":"error","source":"baggageclaim","message":"baggageclaim.failed-to-set-up-driver","data":{"error":"failed to create btrfs filesystem: exit status 32"}}

author README(s)

Hey all - Thanks for your great work on Concourse and thanks for publishing Concourse's ci automation to GitHub. As a user, enthusiast, and potential contributor, I'd love to learn more about what's happening throughout this repo. A README -- or even multiple READMEs, if needed -- could help. Thanks!

Move away from statically building `fly` with `glibc`

Hey,

In the current build process for fly (see

if [ "$platform" = "linux" ]; then
ldflags+=' -extldflags "-static"'
if [ "$USE_EXTERNAL_LINKER" = "true" ]; then
ldflags+=' -linkmode external'
), we end up statically building a binary that contains glibc in it.

I'm not a lawyer, but the arguments are that you just can't do it.

On the legal side, the LGPL license of GNU libc requires you to permit your users to enhance their libc; so if you sell a proprietary software which is statically linked (a technically bad idea), you need to give the means to your user to relink it to a better version of libc; concretely you'll probably need to also give the *.o object of your software.

(see https://stackoverflow.com/a/11693411)

If you're wondering why I say just fly and not concourse, that because we're not passing any extra flags to the building of concourse:

pushd concourse
go install github.com/gobuffalo/packr/packr
packr build -o concourse -ldflags "$ldflags" ./cmd/concourse

which means that we're already generating a dynamically linked binary (you can confirm with ldd):

ldd ./concourse
	linux-vdso.so.1 (0x00007ffcf446c000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0c14626000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0c14422000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0c14031000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0c14845000)
ldd ./fly
	not a dynamic executable

Moving forward, I can see that we could either:

  • build static fly binaries with musl - which is MIT licensed (rather than the LPGLed glibc)
  • just go dynamic (I think we had problems w/ this before)

I remember seeing a paper that compared glibc and musl performance-wise, and there was a small difference (although in most cases, that's not significant). While I can see that that could be a problem for concourse, I imagine that's not a big deal for fly.

Wdyt?

thanks!

add CODEOWNERS to concourse/ci

so that folks are not forced to request review every time. Might also be
worthwhile to create a 'Release Engineering' team in the Concourse org.

fix topgun after the big split

just an issue to track progress!

so the way the previous topgun was setup:
topgun folder containing many tests (individual go files) plus a test suite file.

we then moved a bunch of tests into 4 separate folders, pcf, core, runtime, both, but ginkgo doesnt allow the sharing of that test suite file.

so i made a pcf test suite file, a common package, and pulled out lots of stuff from the old test suite file into the common package (renaming and moving what i had to), and imported that common into the test suite / test file.

i don't know if this is the proper thing to do, but it still took a while!

fix docs PR job

to get concourse/docs#235 unblocked.

  • add a GITHUB_TOKEN param to jobs that build the docs
  • ensure that concourse_github_dummy is at the appropriate scope in Vault

include prod backups into the master pipeline

I think this would be a good means of dogfooding the Postgres release in a more realistic "production-like" setup. Might give a good example for curious community members, and hopefully gives us good empathy/experience to help bosh operators who use BBR.

improve release notes flow

  • store release notes in concourse/concourse
  • write release notes in markdown instead of booklit
  • remove any CI pipeline references to concourse/release-notes
  • destroy the concourse/release-notes repo

trigger release jobs on changes to this repo

it's possible that we make backwards-incompatible changes to this repo that
will hamper our ability to patch old versions when time is of the essence.
Even though it results in a lot of builds, perhaps it's worth the investment to
re-run all jobs on any change to ci so we can get notified of our breaking
changes earlier?

Drills pipeline on K8S

Background:

  • With the current Drills pipeline, it is very easy to create a pipeline that runs tests and deploy long-living branches.
  • The only problem is that the pipeline is using Bosh. This is making the deployment slow, especially for people trying to do quick experiments in a production-like environment.
  • The idea is to have a simple pipeline that resembles the drills pipeline, but deploys to k8s and only runs unit tests, testflight tests, and k8s-topgun test.
  • This pipeline should also be emitting metrics to Datadog.
  • Also, creating the subdomain for accessing the deployment.
  • Finally, it would be great if there is a destroy job that would delete the k8s deployment, delete the metrics dashboard, if possible, and remove the subdomain.

Research failure modes for bosh-related jobs

Observation

Lots of failures in bosh-related testing jobs, and most of them seem to concern failed deploys. Honestly I don't understand them very well, and I'm a bit worried that countermeasures like concourse/concourse#3945 won't help.

Hypothesis

I don't have a lot of confidence here, and am happy for folks to weigh in. @vito's suggestion was that the bosh director is getting overwhelmed by parallel tasks. The supporting argument is that we have more deployments than ever before, since we are running deploys for multiple release and feature branches as well as wings.

Experiment

This is admittedly a heavyweight suggestion (as pointed out by @cirocosta), but it involves the least working knowledge of bosh: we could modify our pipeline to distribute the deploys across multiple directors, and compare

  • frequency of deploy failures
  • frequency of test flakes

over time. This approach does carry a lot of unknowns, (though @taylorsilva's suggestion to use toolsmiths could offload some of those), so hopefully there's a less-costly experiment that we can run? Slightly lighter-weight is the below suggestion to scale the bosh director vertically. Can we do even better than that? Maybe we can run some query directly against the bosh director in its current state, or maybe just investigate the failure logs we already see, or else set up some kind of proactive monitoring and wait a while?

Run PKS tests in the Main Pipeline

[@cirocosta here]:

Summary

In order to continuously run [topgun tests][pks] for PKS, we will
need to target PKS when running our end-to-end tests in the main pipeline.

This will include:

  • Having a task that tests properly authenticating and targeting PKS
    • note.: the vSphere environment is not something that is available everywhere
      • only Pivotal networks can reach it.
  • Have the credentials available to the Concourse installation

Acceptance Criteria

  • k8s-pks-topgun job is green

Extra context

To know more about the PKS environment that the team has available to it, please
check out PKS on Toolsmiths vSphere.

`concourse-release` versions on main pipeline re-ordering on every check

On the prod database, first I ran

atc=> select v.version, v.check_order from resource_config_versions v, resources r where r.id = 1738 and r.resource_config_scope_id = v.resource_config_scope_id and version @> '{}' order by v.check_order desc limit 20;
                         version                          | check_order
----------------------------------------------------------+-------------
 {"version": "5.6.1-dev.20191002T133445Z.commit.511a255"} |      337900
 {"version": "5.6.1-dev.20191002T152647Z.commit.564c384"} |      337899
 {"version": "5.6.1-dev.20191002T163626Z.commit.7ee7637"} |      337898
 {"version": "5.6.1-dev.20191002T195320Z.commit.bc7f3db"} |      337897
 {"version": "5.6.1-dev.20191002T213956Z.commit.6166ef2"} |      337896
 {"version": "5.6.1-dev.20191003T011318Z.commit.68bef3c"} |      337895
 {"version": "5.6.1-dev.20191003T151219Z.commit.c619c91"} |      337894
 {"version": "5.6.1-dev.20191003T210445Z.commit.d04684a"} |      337893
 {"version": "5.5.1-dev.20190906T194845Z.commit.aca4553"} |      326396
 {"version": "5.5.1-dev.20190906T195037Z.commit.4056e00"} |      326395
 {"version": "5.5.1-dev.20190909T174636Z.commit.7873ba1"} |      326394
 {"version": "5.5.1-dev.20190909T174827Z.commit.ea9889b"} |      326393
 {"version": "5.5.1-dev.20190909T182242Z.commit.e0ab701"} |      326392
 {"version": "5.5.1-dev.20190909T182400Z.commit.eea8d51"} |      326391
 {"version": "5.5.1-dev.20190909T210958Z.commit.cc4b570"} |      326390
 {"version": "5.5.1-dev.20190910T165841Z.commit.aebae5e"} |      326389
 {"version": "5.5.1-dev.20190910T180821Z.commit.380ba85"} |      326388
 {"version": "5.5.1-dev.20190910T200449Z.commit.174a580"} |      326387
 {"version": "5.5.2-dev.20190910T200759Z.commit.a5f5e1f"} |      326386
 {"version": "5.5.2-dev.20190910T212119Z.commit.e4df302"} |      326385
(20 rows)


Let's boil this down to

511a255
564c384
7ee7637
bc7f3db
6166ef2
68bef3c
c619c91
d04684a
--- above this line is the latest 8 commits in reverse order
aca4553
4056e00
--- above this line is two random commits far earlier in the history (also
--- in reverse order.
7873ba1
ea9889b
e0ab701
eea8d51
cc4b570
aebae5e
380ba85
174a580
a5f5e1f
e4df302


Then clicked the 'check' button on the resource page. Then I ran

atc=> select v.version, v.check_order from resource_config_versions v, resources r where r.id = 1738 and r.resource_config_scope_id = v.resource_config_scope_id and version @> '{}' order by v.check_order desc limit 20;
                         version                          | check_order
----------------------------------------------------------+-------------
 {"version": "5.6.1-dev.20191002T152647Z.commit.564c384"} |      337907
 {"version": "5.6.1-dev.20191002T163626Z.commit.7ee7637"} |      337906
 {"version": "5.6.1-dev.20191002T195320Z.commit.bc7f3db"} |      337905
 {"version": "5.6.1-dev.20191002T213956Z.commit.6166ef2"} |      337904
 {"version": "5.6.1-dev.20191003T011318Z.commit.68bef3c"} |      337903
 {"version": "5.6.1-dev.20191003T151219Z.commit.c619c91"} |      337902
 {"version": "5.6.1-dev.20191003T210445Z.commit.d04684a"} |      337901
 {"version": "5.6.1-dev.20191002T133445Z.commit.511a255"} |      337900
 {"version": "5.5.1-dev.20190906T194845Z.commit.aca4553"} |      326396
 {"version": "5.5.1-dev.20190906T195037Z.commit.4056e00"} |      326395
 {"version": "5.5.1-dev.20190909T174636Z.commit.7873ba1"} |      326394
 {"version": "5.5.1-dev.20190909T174827Z.commit.ea9889b"} |      326393
 {"version": "5.5.1-dev.20190909T182242Z.commit.e0ab701"} |      326392
 {"version": "5.5.1-dev.20190909T182400Z.commit.eea8d51"} |      326391
 {"version": "5.5.1-dev.20190909T210958Z.commit.cc4b570"} |      326390
 {"version": "5.5.1-dev.20190910T165841Z.commit.aebae5e"} |      326389
 {"version": "5.5.1-dev.20190910T180821Z.commit.380ba85"} |      326388
 {"version": "5.5.1-dev.20190910T200449Z.commit.174a580"} |      326387
 {"version": "5.5.2-dev.20190910T200759Z.commit.a5f5e1f"} |      326386
 {"version": "5.5.2-dev.20190910T212119Z.commit.e4df302"} |      326385
(20 rows)


Let's boil this down, too.

564c384
7ee7637
bc7f3db
6166ef2
68bef3c
c619c91
d04684a
--- above is the last 7 commits in reverse order
511a255
---
aca4553
4056e00
7873ba1
ea9889b
e0ab701
eea8d51
cc4b570
aebae5e
380ba85
174a580
a5f5e1f
e4df302

delete zstd drills env

obviously this involves removing it from reconfigure-pipelines, but it's also a chance to think about cleaning up the related resources on GCP

automatically assign PRs

In the interest of ensuring that PRs get the attention they deserve, let's think about proactively assigning them as soon as they appear. There may be some concerns about who should be assigned what PR, or even whether an automated solution is overkill.

@nader-ziada could comment on how this has been done on other projects - i imagine via a github bot. If we're OK taking some bandwidth, it might be worth implementing something like this with a concourse pipeline, because dogfood.

Release pipelines to update concourse-bosh-release repo

  • Noticed that the current release pipelines don't update the concourse-bosh-deployment repo, that would make a discrepancy in the versions.yml file in case we create a patch version for a release.
  • Also, we don't have any mechanism in place that creates the release branch in that repo.

Thanks 🙏

figure out how to get 'releases' page to show the latest release by semver as 'latest'

We managed to put v5.4.1 back at the top of the list by following this advice on StackOverflow:

  • delete the v5.4.1 tag locally and remotely (this converts an existing release back to a draft)
  • re-tag locally with a date newer than any other release (might not actually be necessary)
  • push the tag (now the tag shows up as a separate entry on the Releases page)
  • copy the description and and assets from the draft into a new release based on the new tag
  • release again
  • delete the draft release

Maybe we should automate this in the release pipelines for patching older versions.

incorrect tagging of final `concourse/concourse` images

When working on the refactoring of the pipeline to support the publishing of concourse/concourse based on not only alpine resource types, but ubuntu as well, we end up not updating the way that tags are created.

echo $version >> tags/tags
echo $minor_version >> tags/tags
# if the version we're publishing is for the latest minor version *of our major
# version*, bump the major tag
if [ "$minor_version" = "$latest_minor_of_same_major_version" ]; then
echo $major_version >> tags/tags
fi
# if the version we're publishing is for the latest minor version, *overall*,
# bump the latest tag
if [ "$minor_version" = "$latest_minor_version" ]; then
echo latest >> tags/tags
fi

ci/pipelines/concourse.yml

Lines 1208 to 1222 in 6507e36

- task: docker-semver-tags
file: ci/tasks/docker-semver-tags.yml
input_mapping:
latest-of-same-major-version: latest-version
- in_parallel:
- put: concourse-image-alpine
inputs: [concourse-rc-image-alpine, tags]
params:
image: concourse-rc-image-alpine/image.tar
additional_tags: tags/tags
- put: concourse-image-ubuntu
inputs: [concourse-rc-image-ubuntu, tags]
params:
image: concourse-rc-image-ubuntu/image.tar
additional_tags: tags/tags

This means that we end up with a race condition on the put of our images.

Instead, we should have different tags for each image that we publish, preferably, following the patterns used by official docker images (e.g., see https://hub.docker.com/_/nginx).

alpine tags:

  • latest
  • 5.2.1
  • 5.2
  • 5

ubuntu tags:

  • ubuntu
  • 5.2.1-ubuntu
  • 5.2-ubuntu
  • 5-ubuntu

Wdyt?

thanks!

k8s: Add a job check chart's dependencies

Hey,

It is not easy for us to monitor our Concourse chart being up to date with its dependencies (currently only postgresql).

suggesting to add a job that checks the current dependencies versions against the latest versions. It should be non-blocking though, as we won't be obliged to bump the dependencies until tested.

Address flakiness in k8s-smoke

Hey,

k8s-smoke, the job which deploys Concourse on Kubernetes (using
the Helm chart) and then validates some basic functionality of it, has
been flaky for quite a while:

Screen Shot 2019-07-04 at 12 07 48 PM

We improved it a bit some time ago (concourse/concourse#3947)
but, still, we've been seeing some weirdness when trying to connect in the
k8s-smoke task:

A suggestion:

  • look at what process is exiting with a non 0 exit code

Acceptance Criteria

  • Have k8s-smoke running flawlessly for an entire day
    • seeing some 20 consecutive successful runs could be enough

Thanks!

Integrate drills into the branch pipeline flow

  • environment creation should be trivial - an easy way to skip any testing
    • see about deploying a dev release rather than committing blob references on the bosh-release repo
    • such a job should live alongside the default testing flow, deploying to the same bosh environment but always using the latest code from the experimental branch with no passed constraints.
  • metrics should be automatic
  • upgrade tests should be easy to run?

split topgun into separate track-owned jobs

ideas so far - we got interrupted several times with other things today and so haven't finished reading through all the test files.

@concourse/core-api:
Multiple ATCs Login Session Test
AWS SSM
Container scope
Credhub
Database secrets encryption

@concourse/runtime:
Rebalancing workers
Passing artifacts between build steps
An ATC with default resource limits set
Ephemeral Workers
A worker with a proxy configured

both:
ATC Shutting down
Garbage collecting build containers
Hijacked containers

@concourse/concourse-for-pcf:
BBR

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.