We have the following ignore paths in our pipeline.

ignore_paths:
  - releases/bosh-openstack-cpi/**
  - .final_builds/**
  - docs/**
  - README.md

We have tested the glob patterns locally with the following command which resembles what the check does in the git-resource:

$ git log --format='%H' -- . ':!.final_builds/**' ':!releases/**' ':!docs/**'

Unfortunately it does not filter expected commits:
https://main.bosh-ci.cf-app.com/pipelines/bosh-openstack-cpi/resources/bosh-cpi-release
(Concours apparently runs in version 0.71.1)

Commits that should not be part of the list are the following:
1ff876e53c2addd4cc2a4b54eaf20e30b7e2fb19
774ec6f200df0a93b3f669cf9bdb4b6d36f4429f

Strange thing is, in our internal concourse (currently 0.71.0) we have the same pipeline set and it works.
(We also have a pipeline where it partially worked, but here we are not fully aware what pipeline-configs have been applied)

Any ideas?

1. Checking a private key into a repo is sinful in any case.
2. Github API tokens can have permissions limited to pull-only. SSH keys effectively have "root" on everything a user account can push/pull to.

I'm running Concourse 2.1.0 and occasionally see git resources lock up. In happens across all pipelines without any error message. Attempting a check-resource via fly also hangs with no output. This seems similar to #68, although I'm not using btrfs.

I've tried aborting and restarting jobs without any luck. I've also left a job running overnight but came back to it still waiting on the git resource. Deleting and recreating the pipeline does allow it to immediately succeed.

commits with [ci skip] are now showing up inside our git resources and triggering builds looking at the repo in question is public should allow reproduction of the issue.

Local git version: git version 2.5.0
Version of git inside container which made commit: git version 1.9.1

Git resources using https auth don't seem to function at all.

Using this setup:

resources:
- name: reponame
  type: git
  source:
    uri: https://github.com/companyname/reponame.git
    branch: master
    username: githubusername
    password: githubpassword

The check script fails with the following error message:

resource script '/opt/resource/check []' failed: exit status 128

stderr:
Cloning into '/tmp/git-resource-repo-cache'...
fatal: could not read Username for 'https://github.com': No such device or address

HTTPS auth is important because you can't authenticate submodules using private keys. Thanks!

I have read Concourse CI documentation and couldn't find the git resource type settings in case of ssh private key which set a passphrase.

Could you support ssh private key which set a passphrase? Or is Concourse CI team responsible for this problem?

Hi,
first steps with Concourse CI for me, first of all thanks for that great tool with interesting different concepts. :)

I've tried to connect a git-resource to my private git repository (not hosted on github). Is it the case that only the scp-like url scheme git@...:/path/to/repo.git/ is supported? I've tried several attempts similar to the following ones, but only the last one seems to work (all the others fail with fatal: Could not read from remote repository.). I've specified the correct private_key, btw.

ssh://example.com:1234/path/to/repo.git/
git://example.com:1234/path/to/repo.git/
[email protected]:/path/to/repo.git/

The first schemes are needed if you want to specify a different port other than 22.

Also local directory repositories don't seem to work, neither with /path/to/repo.git/ nor file:///path/to/repo.git/.

Could you please confirm or clarify in the documentation what is supported?

Hi,

I just wonder about the following scenario:

GitRepoA*
+--- SubmoduleAA*
:    +--- SubmoduleAAA
     |    +--- fileAAAA
     |    +--- fileAAAB
     +--- SubmoduleAAB*
     :    :

SubSubModule AAB has changed, therefore SubmoduleAA and GitRepoA were changed, too (new submodule commits). SubmoduleAAA has no changes.

If GitRepoA is my source-uri, how do I express that I get triggered only if I had a change on fileAAAB? Just setting "paths: [SubmoduleAA/SubmoduleAAA/fileAAAB]" ???

Best regards, Stephan

Firstly, i should say that i am still trying to get my head round how this resource works, so i might have the wrong end of the stick here.

This is an idea for an enhancement.

The 'check' script does this:

destination=$TMPDIR/git-resource-repo-cache

if [ -d $destination ]; then
  cd $destination
  git fetch
  git reset --hard FETCH_HEAD

Which means that if the resource container is reused, it will do incremental fetches from the origin into $TMPDIR/git-resource-repo-cache to get new commits.

The 'in' script does this:

cat > $payload <&0
# ...
uri=$(jq -r '.source.uri // ""' < $payload)
# ...
git clone --single-branch $depthflag $uri $branchflag $destination

Which means that it's always pulling a fresh (albeit possibly shallow) clone from the origin.

Would it be possible for the 'in' script to clone from $TMPDIR/git-resource-repo-cache instead? That means the clone would be purely local, and would not need to go out to the network again.

You'd need to make sure it was a standalone clone, not using hardlinks or sharing or whatever, but there are flags for that.

I appreciate that you probably want the clone's remote to be the real origin, not the local cache, but that can be arranged. Perhaps do a clone from the origin with --depth 0 (is that allowed?) then do a pull from the local cache. Or else clone, then use git remote to rewrite the remote. You'd need to do something about tracking branches too.

You'd need to allow for the case where 'in' is asked for a version of the repository which isn't in the cache (say because the resource container was deleted in between the 'check' and the 'in'). You could deal with that by checking to see if the version is there, and if not, calling into the 'check' machinery to create and update the cache.

Git fetches all branches by default, and this resource is following this schema. I had small master branch (~1MB) with few other big branches (~0.5GB), simple getting the master took like 2-3 minutes. git clone done manually in shell did the same - and downloaded 0.5GB of data. Deleting big remote branches helped for me, this resource started getting master in few seconds instead.

git clone -b mybranch --single-branch git://sub.domain.com/repo.git

I would like to try out concourse, but almost all of my repos are encrypted with git-crypt.

So I was thinking about adding git-crypt support and sending a PR, is this something you are interested in?

If you have a paths option, then the check script will fail to trigger for merge commits even if the commits which they merged in contained changes to files which are included in paths.

Presumably this is because of the shallow clone, so there's NO files included in the merge commit.

Hi,

Is there a way to pass the git reference SHA to other tasks in the pipeline?

/cc @Jonty

Opening a separate issue to augment tag_filter. It would be nice if it could support the full resource interface, part of which defines that it should be possible to discover old versions, and ranges of versions, not just the latest.

Something like this might work pretty well (replacing the git describe that we currently do):

git tag --list <pattern> --sort=creatordate [--contains <cursor>]

Example, run from buildroot which has date-formatted tags and RCs:

~/w/buildroot (master) $ git tag --list "*.*" --sort=creatordate --contains 2015.08
2015.08
2015.08.1
2015.11-rc1
2015.11-rc2
2015.11-rc3
2015.11
2015.11.1
2016.02-rc1
2016.02-rc2
2016.02-rc3
2016.02
2016.05-rc1
2016.05-rc2
2016.05-rc3
2016.05

(...which brings up another point: how to omit RCs?)

And another example, in concourse/concourse:

~/w/concourse (develop) $ git tag --list "v*" --sort=creatordate
v0.14.0
v0.15.0
v0.16.0
v0.17.0
v0.18.0
v0.19.0
v0.20.0
v0.21.0
v0.22.0
v0.23.0
v0.24.0
v0.25.0
v0.26.0
v0.27.0
v0.28.0
v0.29.0
v0.30.0
v0.31.0
v0.32.0
v0.33.0
v0.34.0
v0.35.0
v0.36.0
v0.37.0
v0.38.0
v0.39.0
v0.40.0
v0.41.0
v0.42.0
v0.43.0
v0.44.0
v0.45.0
v0.46.0
v0.47.0
v0.48.0
v0.49.0
v0.50.0
v0.51.0
v0.52.0
v0.53.0
v0.54.0
v0.55.0
v0.56.0
v0.57.0
v0.58.0
v0.59.0
v0.59.1
v0.60.0
v0.60.1
v0.61.0
v0.62.0
v0.62.1
v0.63.0
v0.64.0
v0.64.1
v0.65.0
v0.65.1
v0.66.0
v0.66.1
v0.67.0
v0.67.1
v0.68.0
v0.69.0
v0.69.1
v0.70.0
v0.71.0
v0.71.1
v0.72.0
v0.72.1
v0.73.0
v0.74.0
v0.75.0
v0.76.0
v1.0.0
v1.0.1
v1.1.0
v1.2.0
~/w/concourse (develop) $ git tag --list "v*" --sort=creatordate --contains v1.0.1
v1.0.1
v1.1.0
v1.2.0
~/w/concourse (develop) $

I'd like to be able to git checkout my cf-release repo to a specific tag so I can access the correct templates folder.

I just wasted a non-trivial amount of time figuring out why git-resource wouldn't work for checking out a repo via ssh.

Turns out I was using private-key: instead of private_key: in my pipeline.yml

In both cases, fly set-pipeline happily accepted my inputs, letting me down only when trying to check the resource, with this error

$ fly -t local cr -r test/resource-tutorial
error: check failed with exit status '128':
Cloning into '/tmp/git-resource-repo-cache'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights

(see also #66 (comment))

At least git-resource should both provide proper diagnostics and check that it actually has a private key specified when trying to check out an ssh repo. A warning statement about missing settings in the output returned by fly would be good enough already.

Bettery yet, fly set-pipeline should validate the yml file, though that may be a lot more difficult

I have just started learning Concourse CI from the tutorial https://github.com/starkandwayne/concourse-tutorial On Step 12 got into troubles with git-resource, the process fails on SSH host key verification.

This is the link to the original issue so Dr Williams would be in the loop.

There is no reason this needs to be done serially. You could add something like this:

git submodule status | awk '{print $2}' | xargs -P5 -n1 git submodule update --init $depthflag --recursive

here:

I'll do a PR if you want.

It's worth noting that the filename for the 'tag:' parameter needs to be expressed relative to the base of the container (e.g.: version/number), whereas the filename for the 'annotate:' parameter needs to be expressed relative to the base of the git repository (e.g.: ../version/number)

Not too difficult to figure that out by reading the source, but it's a surprising piece of information that should probably be called out explicitly in the documentation.

Some environments only allow HTTP(S) traffic into the Git repository; and so need to use https:// protocol with basic auth credentials; rather than private key

We'd really like to only trigger some pipelines when we push a new tag marking a release, however we also push other tags so we would like to be able to filter the tags used by the check script.

Obviously the checked-out version would match the latest matching tag that has been pushed.

Opening this here as we've expressed interest in working on it in Slack and were told it would be accepted upstream.

Our git-resource for cf-deployment is having a few problems.

First, it seems to fetch the wrong branch. The desired branch is develop, which is at SHA 210655a. Instead, it seems to be fetching fa43ce from a 5-month-old WIP branch that we've since deleted. There hasn't been any activity on this branch, but for some reason, the three commits on this branch are the three topmost versions of git resource in the concourse UI.

Second, even without the inclusion of this WIP branch, the remainder of the history is missing the most recent commits on develop.

As you can see, ref 3e48b is the most recent commit in the history of develop that the git resource his picked up (it doesn't seem to have found refs 2782123, 9f545d2, or 210655a, even though they are all in the history of develop).

We're not sure what's causing this, or how to find out the root cause. (We were trying to delete a bunch of data from the build_events table. We're not sure how this would affect resource versions, but it could be that something was going wrong wit hthe db.) We're going to work around it by re-building the pipeline.

It seems that it does not use ssh key to do submodule init. I can clone my repo and its submodules separately (using the same ssh key) and each succeeds, but unless I say { submodules: none } when doing the get on the parent, the parent fails to clone with:

Identity added: /tmp/git-resource-private-key (/tmp/git-resource-private-key)
Cloning into '/tmp/build/get'...
Fetching HEAD
843d29e Rename some concourse stuff
Submodule 'src/main/resources/[redacted]-secrets' (git@[redacted].git) registered for path 'src/main/resources/[redacted]-secrets'
Cloning into '/tmp/build/get/src/main/resources/[redacted]-secrets'...
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@[redacted].git' into submodule path '/tmp/build/get/src/main/resources/[redacted]-secrets' failed

It'd be handy if there was a file containing the sha of the git repo checked out. Currently, the way to get this is via git rev-parse HEAD.

see #69

right now it'll always try to push the new tag, and fail, even if it would have pointed to the same SHA.

We have a repo on github under the Company-ProductGroup org name, let's call it Company-ProductGroup/instrumentation

When we wrote and tried to use a pipeline with the source.uri value of [email protected]:company-productgroup/instrumentation.git, the git clone failed. Correcting the capitalization of the org name let things carry on correctly.

We would expect that github org names are not case-sensitive.

Currently, no --depth parameter is applied when doing git submodule update --init --recursive $submodule

We have a large submodule that we'd like to clone shallowly.

Curly bracket expansion doesn't play nice when used as part of a URI:

resources:
- name: tools
  type: git
  source:
    uri: [email protected]:{{github-username}}/tools.git
    branch: master
    private_key: {{github-private-key}}

jobs:
- name: test
  plan:
  - get: tools

After set-pipeline passing --var github-username=bacoboy expands with extra quotes:

      uri: [email protected]:"bacoboy"/tools.git

This results in a resource error:

resource script '/opt/resource/check []' failed: exit status 128

stderr:
Identity added: /tmp/git-resource-private-key (/tmp/git-resource-private-key)
Cloning into '/tmp/git-resource-repo-cache'...
fatal: remote error: 
  %s is not a valid repository name
  Email [email protected] for help

We use git lfs for some development dependencies (~2GB of them) but we don't need them on CI (and it slows down the builds enormously).

It looks like it should be as easy as adding a param like disable_git_lfs and adding an if to the inputs script. Does that sound reasonable if I make a PR?

It would be nice that checked out branch would be added to metadata.

Hi Folks,

I know we (the data toolsmiths) were the folks who originally submitted the LFS PRs to the git resource, but we discovered they don't work! I am creating this issue to track the fact that it still definitely doesn't work, and we (the data toolsmiths) should probably submit some more PRs to actually make LFS work.

Best,
Zak + @cjcjameson

P.S. Do u ever think about space?

We are running 1.4.1 and find that randomly, a git resource which has worked fine will refuse to pull from a repo which has new commits available.

There are no errors, the resource simply spins and never times out.

This is a random issue that affects most of our pipelines which normally work without an issue.

Anyone else experience a similar issue?

Resource caching [1] prevents the git_resource from recognizing changes to the resource configuration. e.g. branch, uri

[1] https://github.com/concourse/git-resource/blob/master/assets/check#L29

The current git-lfs binary is at version 1.1.2, while the latest is at 1.2.1. Should we consider making the git-lfs github release an explicit input to the git-resource build plan?

the git resource does not seem to be able to tag a repo at the sha provided to a build, if there have been commits after that sha on the branch.

when we try to do this, we get errors on push if we don't specify rebase, and it tags the wrong sha if we do specify rebase.

a tags-only push would solve this problem for us

After cloning cf-release I see the following when running git status:

fatal: Not a git repository: /tmp/build/get/.git/modules/src/capi-release/modules/src/cloud_controller_ng
fatal: 'git status --porcelain' failed in submodule src/capi-release

Here is a pipeline that can recreate the issue on concourse v1.3.0:

resources:
- name: cf-release
  type: git
  source:
    uri: https://github.com/cloudfoundry/cf-release.git
    branch: master

jobs:
- name: check-status
  plan:
  - aggregate:
    - get: cf-release
      resource: cf-release
  - task: status
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: pivotalcfreleng/minimal
      inputs:
      - name: cf-release
      run:
        path: /usr/bin/git
        args:
        - status
        dir: cf-release

Currently, when specifying a value for tag_filter the only candidates that are considered are the ones that are reachable from the specified (or default) branch. Tags that are detached HEADs (or in other branches) are not considered since they are not retrieved during git fetch <BRANCH>.

In Maven (and Java generally) the version of an application exists in source control. So, when we tag a release, it requires a commit that contains those changes. In the Spring projects, it’s common that master never has a release on it. Version numbers go from one snapshot to another from the perspective of master, and the release hangs off and has to be directly targeted.

Based on a Slack conversation I'd like to advocate that any behavior having to do with tags (tag_filter is the only one that's obvious to me) ignore branches completely. All tags, regardless of what they're reachable from HEAD or not, should be considered. This policy should hold true for other future features having to do with tags as well.

If you give Concourse an SSH private key that has a passphrase the only error you will see is when you click the resource is something like unable to check. Only experienced Concoursers recognize the missing hashes for a git resource and suggest connectivity problems with the private key. We figured the issue out but I figured I would create an issue in the hopes some kind of relatively simple check can be added to help future devs.

I think it would be a nice tweak to reduce mistakes when people first start and also kills a line of .yml in most people's pipelines.

We have a pipeline that is not getting triggered on a change to dir/file.md when defining the glob dir/**/*. When using dir/* it works.

Locally the commands git log --format='%H' --reverse $ref..HEAD -- dir/**/* and git log --format='%H' --reverse $ref..HEAD -- dir/* produce the same output.

Hi,
We have a pipeline where the get params include a list of submodules that should be cloned. This works fine when checking out the source-code before building. However after building we push a tag back to the repository and this is where the problem occurs.

After the tag is pushed the source-code is pulled again, this time all submodules are cloned. How can we avoid this full submodule clone?

Best regards Andreas Knifh

A put command to a github resource is always a git push --tag. We use tags to mark release versions and commits deployed to specific environments. Right now, we have to delete all tags locally in our scripts before pushing from Concourse. We would prefer to be able to just specify in our plan configuration that we don't want to push tags, or for this to be the default behavior.

Spoke with @vito about this over the w/e:

[ci-skip] is nice but it stops anyone else from pulling those same commits in your repo. Proposing something like this to fix the problem:

resources:
- name: source-code
  type: git
  source:
    uri: [email protected]:concourse/git-resource.git
    branch: master
    disable_ci_skip: true

Hi,

In our company is mandatory to use a proxy to connect to internet sites. We have a git repository in internet and only configured for ssh access, so we need to create a tunnel using corkscrew across the proxy (http://www.techrepublic.com/blog/linux-and-open-source/using-corkscrew-to-tunnel-ssh-over-http/).

We tested in ours concourse workers instances and it is working, but for using that in git-resource we need to setup the ssh config.

Before to create a fork to implement this functionality and use it, we want to know if we could implement and donate this code to this repository, and which are the steps to follow to colaborate.

We think that this functionality is useful to big companies with a complex environments, so if you want we can colaborate.

What do you think? Could you answer the questions above?

Thanks,
Carlos León

From README "Tracks the commits in a git repository."

The git repository check determines the SHA limited to the paths that are not filtered. When the resource is obtained at getting phase with depth - the depth is based on the overall repository. This difference results in git failing because the target sha is unrecognizable.

Philosophically speaking since the resource is limited to several paths - the depth parameter in the get operation should be considered as depth for the filtered paths - not the overall repo. Easier said than done, though. There are strategies to obtain the needed depth - check this question for ideas: http://stackoverflow.com/questions/39935546/get-git-sha-depth-on-a-remote-branch/39941969#39941969

Note that since the resource check is cloning the whole repo - if there is a mechanism to store the current depth - it can be used as starting point and this will reduce significantly the number of iterations until it gets to the right one.

As a hot patch, the depth might be ignored - if filters on paths are used ... or it might be reported as an issue when the pipeline validation is made. Note that the connection combining these two will result in "random" failures is hard to make.

When using tag_filter, the check command reports the tag name instead of the hash in the version list

But then the in command returns the hash not the tag

I think it should report the tag if available.

For large repos with years of history, it'd be useful to support the --depth option when cloning a repo.

At some point recently (I think it's recent), merge commits have stopped being detected. For example, given a repository's commit log:

and the resource history:

Note that the following commits aren't listed:

• 76730de
• be5d390
• 52d3397
• c135c4d

The common thread is that these are all merge commits. We use merge commits to delineate units of work, but the thing that has flagged this for us is that we use them to trigger Pivotal Tracker transitions.

Using BDD style description, because it's what I know.

Given

I have a branch called foo
Foo is a feature branch off develop
My git resource is configured as branch: foo
The last check of my git resource pointed to sha aaa

When

I merged foo to develop as a merging commit - creating sha bbb
And pushed develop to github
And updated my git resource config to branch: develop
And fly configured concourse

Then

My git resource never triggered with bbb
And I waited a long time

When

I commited another change to develop - creating sha ccc

Then

My git resource triggered with ccc
And never showed bbb