kubernetes / git-sync Goto Github PK
View Code? Open in Web Editor NEWA sidecar app which clones a git repo and keeps it in sync with the upstream.
License: Apache License 2.0
A sidecar app which clones a git repo and keeps it in sync with the upstream.
License: Apache License 2.0
When I add username
and password
to git-sync
command it failed with error:
ERROR: can't create .netrc file: error setting up git credentials exit status 255: error: could not lock config file //.gitconfig: Permission denied
Step 5 : COPY git-sync /git-sync
lstat git-sync: no such file or directory
Today I found a pod that is running since Mon, 27 Mar 2017 19:20:53 -0300
has 1745 zombie child processes. The deployment is configured with this env variables:
- name: GIT_SYNC_REPO
value: http://xxxxxxxxx.git
- name: GIT_SYNC_DEST
value: data
- name: GIT_SYNC_WAIT
value: "300"
- name: GIT_SYNC_USERNAME
value: xxxxxxxxx
- name: GIT_SYNC_PASSWORD
value: xxxxxxxxx
- name: GIT_SYNC_MAX_SYNC_FAILURES
value: "999999"
- name: GIT_SYNC_PERMISSIONS
value: "755"
- name: GIT_SYNC_ROOT
value: /git
image: gcr.io/google_containers/git-sync-amd64:v2.0.4
11349 ? Sl 0:00 \_ /usr/bin/containerd-shim 014cc79294083c25fe69e13acb30cb35977585b086806466ff269c3841116225 /var/run/docker/libcontainerd/014cc79294083c25fe69e13acb30cb35977585b086806466ff269c3841116225 docker-runc
11366 ? Ssl 0:08 | \_ /git-sync
11416 ? Z 0:00 | \_ [git-credential-] <defunct>
10949 ? Z 0:00 | \_ [git-remote-http] <defunct>
30826 ? Z 0:00 | \_ [git-remote-http] <defunct>
21644 ? Z 0:00 | \_ [git-remote-http] <defunct>
8790 ? Z 0:00 | \_ [git-remote-http] <defunct>
27684 ? Z 0:00 | \_ [git-remote-http] <defunct>
15016 ? Z 0:00 | \_ [git-remote-http] <defunct>
1260 ? Z 0:00 | \_ [git-remote-http] <defunct>
20780 ? Z 0:00 | \_ [git-remote-http] <defunct>
7507 ? Z 0:00 | \_ [git-remote-http] <defunct>
26587 ? Z 0:00 | \_ [git-remote-http] <defunct>
13586 ? Z 0:00 | \_ [git-remote-http] <defunct>
32364 ? Z 0:00 | \_ [git-remote-http] <defunct>
19541 ? Z 0:00 | \_ [git-remote-http] <defunct>
6487 ? Z 0:00 | \_ [git-remote-http] <defunct>
25368 ? Z 0:00 | \_ [git-remote-http] <defunct>
12562 ? Z 0:00 | \_ [git-remote-http] <defunct>
30987 ? Z 0:00 | \_ [git-remote-http] <defunct>
18188 ? Z 0:00 | \_ [git-remote-http] <defunct>
4824 ? Z 0:00 | \_ [git-remote-http] <defunct>
24131 ? Z 0:00 | \_ [git-remote-http] <defunct>
11333 ? Z 0:00 | \_ [git-remote-http] <defunct>
30182 ? Z 0:00 | \_ [git-remote-http] <defunct>
17435 ? Z 0:00 | \_ [git-remote-http] <defunct>
3827 ? Z 0:00 | \_ [git-remote-http] <defunct>
22832 ? Z 0:00 | \_ [git-remote-http] <defunct>
10001 ? Z 0:00 | \_ [git-remote-http] <defunct>
28827 ? Z 0:00 | \_ [git-remote-http] <defunct>
16080 ? Z 0:00 | \_ [git-remote-http] <defunct>
2909 ? Z 0:00 | \_ [git-remote-http] <defunct>
22296 ? Z 0:00 | \_ [git-remote-http] <defunct>
9650 ? Z 0:00 | \_ [git-remote-http] <defunct>
.......
I've tried alpine and size is reduced by ~100 mb but the mv version that comes with it doesn't have --no-target-directory
option
Inputs?
I'm specifically trying to add -vvv
to the ssh command by setting GIT_SSH_COMMAND=ssh -vvv
in my pod. PR #29 added behavior that sets GIT_SSH_COMMAND
instead of using a wrapper, which make sense but make this impossible.
I think it would be nice if GIT_SSH_COMMAND
could still be used to set flags since that's how google and stackoverflow suggest people enable more verbose logs. However, I'm not sure if it would make sense to try and merge the user defined set of arguments with the settings git-sync needs to configure, it seems risky and prone to weird errors.
It might be easier to just add a new option to git-sync for enabling verbose ssh commands.
Does anyone have opinions?
When I update the repo, app recognizes a new fetch is required but misreads 'head':
I1012 16:30:46.724601 1 main.go:353] update required
I1012 16:30:46.856616 1 main.go:265] rev HEAD resolves to 55f6f1fe46a888a40a2f0764f33acdfb169cebf6
E1012 16:30:46.857805 1 main.go:156] error syncing repo: error running command: exit status 128: "fatal: '/usr/local/src/repo/rev-55f6f1fe46a888a40a2f0764f33acdfb169cebf6' already exists\n"
$ git --no-pager log --decorate=short --pretty=oneline -n 1
108a2acc0d9da231bbf9420ada28277c3c68f9c6 (HEAD -> master, origin/master, origin/HEAD) fix #7
Because of #32 I tried switching to SSH, but the container keeps crashing again.
The logs are:
$ kubectl logs dp-analytics-blog-2fjnv -c git-sync
I0202 13:28:42.608581 1 main.go:162] starting up: ["/git-sync"]
E0202 13:28:43.307843 1 main.go:169] error syncing repo: error running command: exit status 128: "Cloning into '/git'...\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"
Just to test, I replaced the container command with:
# command: ["cat"]
# args: ["/etc/git-secret/ssh"]
I got the SSH key from the output, removed all my local keys from the SSH agent, imported that key (from the container) with ssh-add
and it worked to clone the repo. So it must be something from the container?
My config:
apiVersion: v1
kind: ReplicationController
metadata:
name: my-rc-name
spec:
replicas: 1
template:
metadata:
name: my-pod-name
labels:
application: my-app
component: my-comp
environment: my-env
spec:
containers:
- name: git-sync
image: gcr.io/google_containers/git-sync:v2.0.4
# command: ["cat"]
# args: ["/etc/git-secret/ssh"]
imagePullPolicy: Always
volumeMounts:
- name: git-secret
mountPath: /etc/git-secret
env:
- name: GIT_SYNC_REPO
value: "[email protected]:GITHUB_USER/GITHUB_REPO.git"
- name: GIT_SYNC_SSH
value: "true"
- name: GIT_SYNC_BRANCH
value: master
- name: GIT_SYNC_DEST
value: git
- name: GIT_SYNC_DEPTH
value: "1"
volumes:
- name: html
emptyDir: {}
- name: git-secret
secret:
secretName: github-secret-ssh
defaultMode: 0400
--dest
and $GIT_SYNC_DEST
are somewhat obtuse.
OS: Mac OS X Sierra 10.12.6
Make version: GNU Make 3.81
Steps: make test
Error:
$ make test
Running tests:
ok k8s.io/git-sync/cmd/git-sync 0.005s
? k8s.io/git-sync/pkg/version [no test files]
Checking gofmt: PASS
Checking go vet: PASS
building: bin/amd64/git-sync
container: e2e/git-sync-amd64:v2.0.5-13-g267a78b-dirty
test root is /tmp/git-sync-test.28622
Initialized empty Git repository in /private/tmp/git-sync-test.28622/repo/.git/
testcase head-once: PASS
testcase default-sync: make: *** [test] Error 1
The issue seems to be the pkill git-sync
it is unable to match a process and returns 1. The container remains also running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0e6fe071a1ef e2e/git-sync-amd64:v2.0.5-13-g267a78b-dirty "/git-sync --logto..." 3 minutes ago Up 3 minutes modest_jennings
And is matched by ps
:
$ ps auxw | grep git-sync
user 26736 0.0 0.0 2451236 828 s003 S+ 8:52PM 0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn git-sync
user 26525 0.0 0.1 556638724 9780 s003 S 8:47PM 0:00.40 docker run -i -u 501:20 -v /tmp/git-sync-test.28622:/tmp/git-sync-test.28622 e2e/git-sync-amd64:v2.0.5-13-g267a78b-dirty --logtostderr --v=5 --wait=0.1 --repo=/tmp/git-sync-test.28622/repo --root=/tmp/git-sync-test.28622/root --dest=link
I actually want to work on issue #54, so I thought I first get all tests working, but if irrelevant I'm happy to drop this issue
As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.
The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".
Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)
Thanks so much, let me know if you have any questions.
(This issue was generated from a tool, apologies for any weirdness.)
[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md
A period of network issues seems to have surfaced an issue with git-sync for us.
Git-Sync can get its self stuck in a crash-loop if it exits after it's done the initial clone. It's very difficult to tell if this is failing after the clone on the initial sync or on a subsequent sync failure that is over the limit of GIT_SYNC_MAX_SYNC_FAILURES
. Both cases results in the same error message, but we can expect both cases to happen in production.
Here are some example logs:
ross@xenon:~$ kubectl logs kube-applier-6949c7675b-g626g -c git-sync
I0820 20:43:42.280722 1 main.go:164] starting up: ["/git-sync"]
I0820 20:43:47.214284 1 main.go:326] cloned ssh://<redacted>.git
I0820 20:43:47.215581 1 main.go:270] syncing to HEAD (c5d769d3bd5d18c99e22f0b3e06e1d1280170808)
E0820 20:43:52.225918 1 main.go:171] error syncing repo: error running command: exit status 128: "fatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"
# Wait for Kubernetes to restart the pod.
ross@xenon:~$ kubectl logs kube-applier-6949c7675b-g626g -c git-sync
I0820 20:44:09.678487 1 main.go:164] starting up: ["/git-sync"]
E0820 20:44:09.679825 1 main.go:171] error syncing repo: error running command: exit status 128: "fatal: destination path '/git' already exists and is not an empty directory.\n"
From that point on the container will never work again, the entire pod must be restarted for it to work. Unless I'm mistaken the source of the errors isn't important since it could come from anything between git-sync and the git server being broken temporarily.
Should I instead try to force the pod to restart if git-sync restarts? That seems a little extreme to me.
E0803 13:22:58.321408 1 main.go:186] error syncing repo: error running command: exit status 1: "Cloning into '/git'...\n/git/.git: Permission denied\n"
My make command
make container REGISTRY=local VERSION=latest
The command I execute after build
docker run --rm local/git-sync --repo=https://github.com/kubernetes/git-sync --branch=master --wait=30
What I already tried
RUN mkdir /git/
RUN chmod 775 /git/
This is a core repository. As such, it needs to use the same merge automation as the rest of the project
I have a PR open that will address this at an org-wide level: kubernetes/test-infra#9342. It will:
/approve
pluginCan one of the repo maintainers here drop an LGTM (or objections) on the linked PR? Alternatively, if I hear no objections by Monday 10am PT of next week, I will merge the PR.
Hi, thanks for sharing this ๐
Do you have any idea about how to notify another container to reload files immediately after git-sync successfully ran?
My use-case is that
(1) git-sync periodically checks my git repo containing PHP/Ruby/etc web app sources
(2) though my app doesn't support hot-reloading on symlink flip (which git-sync does for atomicity of updates),
(3) I'd like my app to reload/restart once git-sync updated sources
I had thought of sending signals from git-sync to notify my app for reloading.
Kubernetes doesn't currently support sending signals between containers/pods kubernetes/kubernetes#24957
OS: Mac OS X Sierra 10.12.3
Steps: make test
Error::
Running tests:
ok k8s.io/git-sync/cmd/git-sync 0.046s
? k8s.io/git-sync/pkg/version [no test files]
Checking gofmt: PASS
Checking go vet: PASS
building: bin/amd64/git-sync
Sending build context to Docker daemon 8.736 MB
Step 1/10 : FROM alpine:3.4
---> 0766572b4bac
Step 2/10 : MAINTAINER Tim Hockin <[email protected]>
---> Using cache
---> 0eac37411a22
Step 3/10 : ADD bin/amd64/git-sync /git-sync
---> Using cache
---> 65a54a0c5443
Step 4/10 : ENV GIT_SYNC_DEST /git
---> Using cache
---> 6bade5a07d01
Step 5/10 : RUN apk update --no-cache && apk add ca-certificates coreutils git openssh-client
---> Using cache
---> 8cb820470a33
Step 6/10 : RUN mv /usr/bin/ssh /usr/bin/ssh-binary
---> Using cache
---> c55cce37bd39
Step 7/10 : ADD ssh-wrapper.sh /usr/bin/ssh
---> Using cache
---> 1a61fe46952d
Step 8/10 : RUN chmod 755 /usr/bin/ssh
---> Using cache
---> 029ee069f00d
Step 9/10 : USER nobody:nobody
---> Using cache
---> cca0e247c918
Step 10/10 : ENTRYPOINT /git-sync
---> Using cache
---> 71aaf60e7e1b
Successfully built 71aaf60e7e1b
container: e2e/git-sync-amd64:v2.0.4-2-g5e2d975
test root is /tmp/git-sync-test.17388
Initialized empty Git repository in /private/tmp/git-sync-test.17388/repo/.git/
testcase head-once: PASS
testcase default-sync: FAIL: /tmp/git-sync-test.17388/root/link does not exist or is not a symlink
ytussupbekov 2457 0.1 0.1 556625368 10996 s000 S+ 12:21AM 0:00.02 docker run -i -u 2064180707:113584762 -v /tmp/git-sync-test.17388:/tmp/git-sync-test.17388 e2e/git-sync-amd64:v2.0.4-2-g5e2d975 --logtostderr --v=5 --wait=0.1 --repo=/tmp/git-sync-test.17388/repo --root=/tmp/git-sync-test.17388/root --dest=link
ytussupbekov 2461 0.0 0.0 2442020 808 s000 S+ 12:21AM 0:00.00 grep git-sync
make: *** [test] Error 1
Cause: The reason being /tmp
directory itself in Mac OS X is a symlink to /private/tmp
. Suggested fix could be to identify architecture darwin
-> and run tests in /private/tmp
. For other architecture, keep it as it is. Though I am not entirely sure on the state of test_e2e.sh
and usefulness of having the test running in Mac OS X. Thoughts?
I ran the example on README and it gave me this below error:
$ docker run -d -v /tmp/git-data:/git gcr.io/google-containers/git-sync-amd64:v2.0.5-13-g267a78b --repo=https://github.com/ngtuna/k8s_notes --branch=master --wait=30
827186f9ea9ad2194eb252386126722c980bb707460a951b179122ab7b53b833
Tunas-MacBook-Pro:git-sync tuna$ docker logs -f 827186f9ea9ad2194eb252386126722c980bb707460a951b179122ab7b53b833
ERROR: --dest must be a bare name
Usage of /git-sync:
During some initial tests following error came up:
text error syncing repo: error running command: exit status 128: "fatal: destination path '/etc/source' already exists and is not an empty directory.\n"
This seems to only happen on subsequent sync runs. Before the directory is empty and syncs successfully.
Some ENVs for a better context:
- name: GIT_SYNC_BRANCH
value: master
- name: GIT_SYNC_SSH
value: "true"
- name: GIT_SYNC_ONE_TIME
value: "false"
- name: GIT_SYNC_WAIT
value: "2"
- name: GIT_SYNC_ROOT
value: /etc/source/
- name: GIT_SYNC_DEST
value: /etc/source/files
Split ENV configuration into own documentation file.
docs/config.md
Instead of failing hard it should log an error and not take down the whole pod.
When I set the one-time
flag along with a revision via the rev
flag the program goes into the sleepForever
call on https://github.com/kubernetes/git-sync/blob/master/cmd/git-sync/main.go#L185, never reaching the test for the one-time
flag on line 187.
Would you accept a PR on this or you prefer to tackle it yourselves?
When it starts up, it checks out the correct revision of the correct branch. The next run, it checks the latest revision of the master branch, even if it says it checks out from origin/kubernetes
:
This are the variables that I'm running with:
I replaced the git repo user/name, because it's a private repo.
GIT_SYNC_BRANCH=kubernetes
GIT_SYNC_SSH=true
GIT_SYNC_DEST=git
[email protected]:my-github-user/my-github-repo.git
And the logs:
I1026 17:09:48.023905 1 main.go:302] cloned [email protected]/my-github-repo.git
I1026 17:09:48.025887 1 main.go:251] syncing to HEAD (82abe61bbd80b47d2275178d7fed190a89ed621a)
I1026 17:09:48.809439 1 main.go:259] added worktree /git/rev-82abe61bbd80b47d2275178d7fed190a89ed621a for origin/kubernetes
I1026 17:09:49.259858 1 main.go:279] reset worktree /git/rev-82abe61bbd80b47d2275178d7fed190a89ed621a to 82abe61bbd80b47d2275178d7fed190a89ed621a
I1026 17:09:50.471179 1 main.go:353] update required
I1026 17:09:50.471212 1 main.go:251] syncing to HEAD (eb893e897072ba4b788e832b0ccab7138d860b06)
I1026 17:09:50.956608 1 main.go:259] added worktree /git/rev-eb893e897072ba4b788e832b0ccab7138d860b06 for origin/kubernetes
I1026 17:09:51.569854 1 main.go:279] reset worktree /git/rev-eb893e897072ba4b788e832b0ccab7138d860b06 to eb893e897072ba4b788e832b0ccab7138d860b06
Hi there,
Would be nice if we could sparse checkout a specified list of folders/files.
I just ran into the issue, that the symlink created for GIT_SYNC_DEST is not an absolute link therefore it needs to be a subdir of GIT_SYNC_ROOT to function. Furthermore the GIT_SYNC_ROOT directory also has to be mounted as volume or else the symlink might fail due to filesystem boundaries.
Add flag and logic to disable symlink usage and just pull in new changes via git pull.
This could be used as a workaround for various issues and additional use-cases, but introduces inconsistent state so this should be heavily warned about.
Flag: GIT_SYNC_ATOMIC
If this is true then git-sync
cannot be used for the static-file based server along with nginx.
Could there be any other issue for this behaviour?
The ssh-wrapper seems unnecessary to me, as the command can be set as:
GITHUB_SSH_COMMAND="ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i /etc/git-secret/ssh"
Hello,
I am using this version of git sync 1a9138765af75007d88f77c985f4f2af200b1227
I build a docker image using make container REGISTRY=X TAG=Y
And I am using following setup:
apiVersion: v1
kind: Pod
metadata:
name: server
spec:
containers:
- image: nginx
name: nginx
volumeMounts:
- mountPath: /mypath
name: git-volume
- image: MY_IMAGE:git-sync-amd64:v2.0.4-6-g1a91387
name: git-sync
env:
- name: GIT_SYNC_REPO
value: "git@MY_REPO"
- name: GIT_SYNC_SSH
value: "true"
#command:
#- tail
#- -f
#- /dev/null
args:
- -dest
- foo
- --v=9
volumeMounts:
- mountPath: /etc/git-secret
name: git-secret
- name: git-volume
mountPath: /git
volumes:
- name: git-secret
secret:
secretName: creds
defaultMode: 256
- name: git-volume
emptyDir: {}
But I cannot clone the repository.
When I use the commend version with tail -f /dev/null
and execute into the container and I run manually:
# export GIT_SSH_COMMAND="ssh -vv -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i/etc/git-secret/ssh"
# git clone git@MY_REPO
...
Load key "/etc/git-secret/ssh": Permission denied
...
# ls -l /etc/git-secret/ssh
lrwxrwxrwx 1 root root 10 Feb 2 16:21 /etc/git-secret/ssh -> ..data/ssh
# -l /etc/git-secret/..data/ssh
-r-------- 1 root root 1675 Feb 2 16:21 /etc/git-secret/..data/ssh
# whoami
nobody
I was able to get git-sync working with these settings. I also was having trouble getting GIT_SYNC_ONE_TIME
working and so just added an arbitrarily high GIT_SYNC_WAIT
to achieve that single first pull/clone.
{
"name": "git-sync",
"image": "gcr.io/google_containers/git-sync-amd64:v2.0.4",
"imagePullPolicy": "Always",
"env": [{
"name": "GIT_SYNC_REPO",
"value": "[email protected]:group/test-repo"
}, {
"name": "GIT_SYNC_BRANCH",
"value": "master"
},{
"name": "GIT_SYNC_DEST",
"value": "master"
}, {
"name": "GIT_SYNC_SSH",
"value": "true"
}, {
"name": "GIT_SYNC_WAIT",
"value": "99999"
}],
"securityContext": {
"runAsUser": 0
},
"volumeMounts": [{
"name": "git-secret",
"mountPath": "/etc/git-secret"
}, {
"name": "repo",
"mountPath": "/git"
}]
}
My other container (to which the above acts as a sidecar) includes the following settings:
{
"name": "git-secret",
"mountPath": "/etc/git-secret"
}, {
"name": "repo",
"mountPath": "/home/jovyan/repo"
}
So that has given me ssh capability back in the main container and synced files to /home/jovyan/repo
, which is all fine.
Perhaps the next part is by design, but maybe someone more familiar with the workings of this can explain or perhaps suggest a work-around.
My main objective is to clone a repo to a compute environment I'm working with and have the ability to push back changes to the repo before I leave that compute environment.
So in my main container running that compute environment, the repo
has been cloned and mounted to /home/jovyan/repo
and, as per the logs from the git-sync container below, folders have been added for master
and rev-f9d0af287de930b7495145c716b1d4fcca88673e
.
cloned [email protected]:group/test-repo.git
2017-02-26T10:35:14.794399416Z I0226 10:35:14.794334 1 main.go:268] syncing to HEAD (f9d0af287de930b7495145c716b1d4fcca88673e)
2017-02-26T10:35:15.341912965Z I0226 10:35:15.341767 1 main.go:281] added worktree /git/rev-f9d0af287de930b7495145c716b1d4fcca88673e for origin/master
2017-02-26T10:35:15.346189283Z I0226 10:35:15.346089 1 main.go:301] reset worktree /git/rev-f9d0af287de930b7495145c716b1d4fcca88673e to f9d0af287de930b7495145c716b1d4fcca88673e
Back on my compute environment, it is the repo
folder that is the git folder (see output from running git status
) and eventhough both the master
and rev-f9***
folders have .git
sub-directories (see red arrow below), these aren't recognised as git repos.
The problem is, based on this new imposed structure, git thinks I have deleted all my files from my repos root folder (see 3) and added them into sub-folders (see 1). If I add these changes and push back to my repo, then it's going to get all screwed up.
Am I using this tool the wrong way, or was it not intended to manage the pushing
changes back to the repo, but instead just emphasising the pulling
part.
I would appreciate anyone's insights on this. Thanks.
Running the container as root is not perfect. Especially, when other containers using the files are non-root.
Hi there.
is it possible to synchronize git submodule as well?
if possible, I'd like to know how to use arguments.
[e.g]
_env:
Due to nobody:nobody not having enough permissions the chmod fails and a ssh secret file can't be used.
Currently working on using defaultMode so that can be removed completely.
I've been trying to understand what is happening, and I'm a little mystified. This seems like something that would have been caught much earlier, so this is most likely a user issue.
What happens is when I try to sync a tag, I get the following error. Conversely, when I use a git hash, things seems to work as expected. Some of the variables are set in the environment, if there are any that would be useful to know, please let me know.
/git # mkdir /test
/git # GIT_SYNC_BRANCH="master" GIT_SYNC_REV="latest" GIT_SYNC_DEPTH="100" GIT_SYNC_ROOT=/test /git-sync
I1105 22:09:10.266422 306 main.go:179] starting up: ["/git-sync"]
I1105 22:09:10.616432 306 main.go:354] cloned [email protected]:
I1105 22:09:10.617700 306 main.go:285] syncing to latest (e9f359bb7ccbfe2e912f386373c3bcf45b497fbc)
I1105 22:09:10.869244 306 main.go:298] added worktree /test/rev-e9f359bb7ccbfe2e912f386373c3bcf45b497fbc for origin/master
I1105 22:09:10.872336 306 main.go:318] reset worktree /test/rev-e9f359bb7ccbfe2e912f386373c3bcf45b497fbc to e9f359bb7ccbfe2e912f386373c3bcf45b497fbc
I1105 22:09:11.085670 306 main.go:405] update required
I1105 22:09:11.085703 306 main.go:285] syncing to latest ()
I1105 22:09:11.295549 306 main.go:298] added worktree /test/rev- for origin/master
E1105 22:09:11.296808 306 main.go:186] error syncing repo: error running command: exit status 128: "fatal: ambiguous argument '': unknown revision or path not in the working tree.\nUse '--' to separate paths from revisions, like this:\n'git [...] -- [...]'\n"
If this happens not to be user error, I would suggest it likely is not expected behaviour.
Not sure why it is so, but there is no git-sync:latest
image in the container registry.
This is causing the demo code to break.
Ref: gcr.io/google_containers/git-sync
๐ >docker build -t myrepo/hugo hugo/
Sending build context to Docker daemon 5.632kB
Step 1/13 : FROM golang
...
Submodule path 'would-have-been-cool-in-the-80s': checked out '7933ce7d6aad4ecf38edc217340c1004dd8f01fb'
Failed to recurse into submodule path 'sublime-hugo-theme'
The command '/bin/sh -c git clone --recursive https://github.com/spf13/hugoThemes.git /themes' returned a non-zero code: 1
I tried the demo setup and played around a bit with the files.
There seems to be an issue with the git-sync
writing the checked out files resulting in a SystemEvent: REMOVE
in Hugo resulting in an empty website. Manually editing the files from the git-sync
container works.
My current setup is with Hugo --verboseLog=true
enabled.
Steps taken:
.md
files$ kubectl logs my-hugo-pod-kgijd hugo -f
+ '[' '!' -d /src/git/site/themes ']'
+ ln -s /themes /src/git/site/themes
++ eval echo server '--source=${HUGO_SRC}' '--theme=${HUGO_THEME}' '--buildDrafts=${HUGO_BUILD_DRAFT}' '--baseUrl=${HUGO_BASE_URL}' --watch '--destination=${HUGO_DEST}' --appendPort=false --log --verbose '--verboseLog=${HUGO_VERBOSE_LOG}'
+++ echo server --source=/src/git/site --theme=hugo-theme-bootstrap4-blog --buildDrafts=true --baseUrl=minion-address:30258 --watch --destination=/dest --appendPort=false --log --verbose --verboseLog=true
+ hugo server --source=/src/git/site --theme=hugo-theme-bootstrap4-blog --buildDrafts=true --baseUrl=minion-address:30258 --watch --destination=/dest --appendPort=false --log --verbose --verboseLog=true
INFO: 2016/10/26 23:58:42 hugo.go:405: Using config file: /src/git/site/config.toml
INFO: 2016/10/26 23:58:42 hugo.go:517: using a UnionFS for static directory comprised of:
INFO: 2016/10/26 23:58:42 hugo.go:518: Base: /src/git/site/themes/hugo-theme-bootstrap4-blog/static
INFO: 2016/10/26 23:58:42 hugo.go:519: Overlay: /src/git/site/static/
INFO: 2016/10/26 23:58:42 hugo.go:551: syncing static files to /dest/
Started building sites ...
INFO: 2016/10/26 23:58:42 site.go:1489: found taxonomies: map[string]string{"tag":"tags", "category":"categories"}
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/tags/themes/page/1" translated to "tags/themes/page/1/index.html"
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/tags/go/page/1" translated to "tags/go/page/1/index.html"
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/tags/hugo/page/1" translated to "tags/hugo/page/1/index.html"
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/tags/templates/page/1" translated to "tags/templates/page/1/index.html"
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/tags/development/page/1" translated to "tags/development/page/1/index.html"
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/tags/golang/page/1" translated to "tags/golang/page/1/index.html"
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/categories/golang/page/1" translated to "categories/golang/page/1/index.html"
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/categories/development/page/1" translated to "categories/development/page/1/index.html"
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/post/page/1" translated to "post/page/1/index.html"
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/page/1" translated to "page/1/index.html"
INFO: 2016/10/26 23:58:42 htmlredirect.go:116: Alias "/page/1" translated to "page/1/index.html"
WARN: 2016/10/26 23:58:42 site.go:2373: "404.html" is rendered empty
Built site for language en:
0 draft content
0 future content
0 expired content
5 pages created
0 non-page files copied
10 paginator pages created
6 tags created
2 categories created
total in 79 ms
WARN: 2016/10/26 23:58:42 hugo.go:568: Skip DataDir: lstat /src/git/site/data: no such file or directory
WARN: 2016/10/26 23:58:42 hugo.go:573: Skip I18nDir: lstat /src/git/site/i18n: no such file or directory
WARN: 2016/10/26 23:58:42 hugo.go:578: Skip LayoutDir: lstat /src/git/site/layouts: no such file or directory
Watching for changes in /src/git/site/{content,static,themes}
WARN: 2016/10/26 23:58:42 hugo.go:568: Skip DataDir: lstat /src/git/site/data: no such file or directory
WARN: 2016/10/26 23:58:42 hugo.go:573: Skip I18nDir: lstat /src/git/site/i18n: no such file or directory
WARN: 2016/10/26 23:58:42 hugo.go:578: Skip LayoutDir: lstat /src/git/site/layouts: no such file or directory
Serving pages from /dest
Web Server is available at //minion-address:30258/ (bind address 127.0.0.1)
Press Ctrl+C to stop
.md
files.INFO: 2016/10/26 23:52:28 hugo.go:703: Received System Events: ["/src/git/site/content/index.md": REMOVE "/src/git/site/content/post/hugoisforlovers.md": REMOVE "/src/git/site/content/post/creating-a-new-theme.md": REMOVE "/src/git/site/content/post/migrate-from-jekyll.md": REMOVE "/src/git/site/content/post/goisforlovers.md": REMOVE "/src/git/site/content/post": REMOVE "/src/git/site/content/post": REMOVE "/src/git/site/content": REMOVE]
Change detected, rebuilding site
2016-10-26 23:52 +0000
Source changed /src/git/site/content/index.md
Source changed /src/git/site/content/post/hugoisforlovers.md
Source changed /src/git/site/content/post/creating-a-new-theme.md
Source changed /src/git/site/content/post/migrate-from-jekyll.md
Source changed /src/git/site/content/post/goisforlovers.md
Source changed /src/git/site/content/post
Source changed /src/git/site/content
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/tags/themes/page/1" translated to "tags/themes/page/1/index.html"
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/tags/go/page/1" translated to "tags/go/page/1/index.html"
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/tags/golang/page/1" translated to "tags/golang/page/1/index.html"
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/tags/development/page/1" translated to "tags/development/page/1/index.html"
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/tags/hugo/page/1" translated to "tags/hugo/page/1/index.html"
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/tags/templates/page/1" translated to "tags/templates/page/1/index.html"
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/categories/development/page/1" translated to "categories/development/page/1/index.html"
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/categories/golang/page/1" translated to "categories/golang/page/1/index.html"
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/post/page/1" translated to "post/page/1/index.html"
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/page/1" translated to "page/1/index.html"
INFO: 2016/10/26 23:52:28 htmlredirect.go:116: Alias "/page/1" translated to "page/1/index.html"
WARN: 2016/10/26 23:52:28 site.go:2373: "404.html" is rendered empty
Built site for language en:
0 draft content
0 future content
0 expired content
0 pages created
0 non-page files copied
9 paginator pages created
6 tags created
2 categories created
total in 47 ms
Manually editing the files Works
In a separate test case, if I ssh into the git-sync
container and manually edit the files, Hugo correctly sees the file as modified and regenerates the correct number of pages.
Hugo Logs
INFO: 2016/10/27 00:01:07 hugo.go:703: Received System Events: ["/src/git/site/content/post/hugoisforlovers.md": WRITE]
Change detected, rebuilding site
2016-10-27 00:01 +0000
Source changed /src/git/site/content/post/hugoisforlovers.md
INFO: 2016/10/27 00:01:07 site.go:1096: rereading /src/git/site/content/post/hugoisforlovers.md
INFO: 2016/10/27 00:01:07 site.go:1489: found taxonomies: map[string]string{"tag":"tags", "category":"categories"}
INFO: 2016/10/27 00:01:07 htmlredirect.go:116: Alias "/tags/templates/page/1" translated to "tags/templates/page/1/index.html"
INFO: 2016/10/27 00:01:07 htmlredirect.go:116: Alias "/tags/hugo/page/1" translated to "tags/hugo/page/1/index.html"
INFO: 2016/10/27 00:01:08 htmlredirect.go:116: Alias "/tags/development/page/1" translated to "tags/development/page/1/index.html"
INFO: 2016/10/27 00:01:08 htmlredirect.go:116: Alias "/tags/themes/page/1" translated to "tags/themes/page/1/index.html"
INFO: 2016/10/27 00:01:08 htmlredirect.go:116: Alias "/tags/go/page/1" translated to "tags/go/page/1/index.html"
INFO: 2016/10/27 00:01:08 htmlredirect.go:116: Alias "/tags/golang/page/1" translated to "tags/golang/page/1/index.html"
INFO: 2016/10/27 00:01:08 htmlredirect.go:116: Alias "/categories/development/page/1" translated to "categories/development/page/1/index.html"
INFO: 2016/10/27 00:01:08 htmlredirect.go:116: Alias "/categories/golang/page/1" translated to "categories/golang/page/1/index.html"
INFO: 2016/10/27 00:01:08 htmlredirect.go:116: Alias "/post/page/1" translated to "post/page/1/index.html"
INFO: 2016/10/27 00:01:08 htmlredirect.go:116: Alias "/page/1" translated to "page/1/index.html"
INFO: 2016/10/27 00:01:08 htmlredirect.go:116: Alias "/page/1" translated to "page/1/index.html"
WARN: 2016/10/27 00:01:08 site.go:2373: "404.html" is rendered empty
Built site for language en:
0 draft content
0 future content
0 expired content
5 pages created
0 non-page files copied
10 paginator pages created
6 tags created
2 categories created
total in 82 ms
Step 5/7 fails when building for arm on a Raspberry pi 3 running 4.4.50-hypriotos-v7+
HypriotOS/armv7: pirate@node01 in ~/git-sync on master*
$ make container REGISTRY=leozilla/git-sync TAG=0.1.0 ARCH=arm
Step 5/7 : RUN apk update --no-cache && apk add ca-certificates coreutils git openssh-client
---> Running in a7d10f45e7b2
/bin/sh: apk: not found
The command '/bin/sh -c apk update --no-cache && apk add ca-certificates coreutils git openssh-client' returned a non-zero code: 127
Makefile:109: recipe for target '.container-leozilla_git-sync_git-sync-arm-v2.0.5-15-gdac3f77' failed
I tried to change the RUN instruction to
Step 5/7 : RUN busybox /bin/apk update --no-cache && busybox /bin/apk add ca-certificates coreutils git openssh-client
---> Running in 4a2dd9194a9e
apk: applet not found
The command '/bin/sh -c busybox /bin/apk update --no-cache && busybox /bin/apk add ca-certificates coreutils git openssh-client' returned a non-zero code: 127
Makefile:109: recipe for target '.container-leozilla_git-sync_git-sync-arm-v2.0.5-15-gdac3f77-dirty' failed
Is this project supposed to work on arm/with busybox?
Almost half of the commits in this repo never made it to a released version.
Maybe it's time to cut a new release so we can get more feedback from the users and people don't have to compile their own image.
Just wondering if anyone has a preferred liveness probe / healthcheck command to ensure that a git-sync sidecar is healthy. Without this, it seems there could be a risk of the git-sync sidecar silently failing, causing an expected config change not to be delivered.
I tried unsuccessfully to clone a private repo on Bitbucket. With repositories in github works wonderfully. This works for Bitbucket repos? Thanks.
k8s cluster: v1.7.1
git-sync: v2.0.4/v2.0.5
The container fails to start because the credentials aren't passed to the git
command. However, whats interesting is that if you ssh
in to the container and run git-sync --help
you can visibility see the env vars have been read by the go program and are set as "defaults".
If I override the container entrypoint and run `./git-sync -username -password then it works just fine.
This obviously isn't ideal because you can see the credentials in plaintext.
Platform specific repos are up to date: https://gcr.io/v2/google_containers/git-sync-amd64/tags/list
The plain git-sync is not. v2.0.2 is still missing. https://gcr.io/v2/google_containers/git-sync/tags/list
cc @thockin
Wondering if this can be used as a standalone docker container outside of kubernetes. I see in docs/ssh.md, it mentions using kubernetes secrets. I'd like to use git over ssh, with public key auth, but just on a standalone VM running docker.
Thanks!
2016/09/03 08:37:57 error syncing repo: error cloning repo "git clone --no-checkout -b master ssh://[email protected]:22/repo/_git/web /git": exit status 128: Cloning into '/git'...
Host key verification failed.
fatal: Could not read from remote repository.Please make sure you have the correct access rights
and the repository exists.
I'm getting a host verification error, which I don't even see how it's possible with the ssh_wrapper. In kubernetes I use the demo yaml-files and a secret.yaml as follows:
apiVersion: v1
kind: Secret
metadata:
name: git-creds
namespace: zeptio
type: Opaque
data:
ssh: "base64-private-key"
This is the command I run on my local host for testing this out:
docker run --rm -e GIT_SYNC_REPO="ssh://[email protected]:22/repo/_git/web" -e GIT_SYNC_SSH=true -v /home/name/web:/git -v /home/name/git-secret:/etc/git-secret gcr.io/google_containers/git-sync
Where /home/name/git-secret contains my non-base64 coded private key.
If I run
git clone --no-checkout -b master ssh://[email protected]:22/repo/_git/web /git
on my local machine, using the ssh key in /home/name/git-secret, everything works fine.
Any idea what could be happening here?
I'd like to contribute into the project adding webhook call on successful sync.
The case is: we trigger Prometheus to reload it's alerts \ rules stored in git repo.
The parameters are as below:
var doWebhook = flag.Bool("webhook", envBool("WEBHOOK_ENABLED", false),
"trigger webhook on sucessfull git sync")
var webhook = flag.String("webhook-url", envString("WEBHOOK_URL", ""),
"the url to send a request to when git synced")
var webhookMethod = flag.String("webhook-method", envString("WEBHOOK_METHOD", "POST"),
"the HTTP method url to use to send the webhook")
var webhookStatusCode = flag.Int("webhook-status-code", envInt("WEBHOOK_CODE", 200),
"the HTTP status code indicating successful triggering of reload")
We have observed a situation where a command, in our case git ls-remote -q origin refs/heads/master
gets stuck running for ~2h.
It succeeds on the retry, but we are left with a stale repository for hours.
Could we add a general timeout of say 5 minutes or any value that would cover most sane operations to: cmd.CombinedOutput()
Line 469 in 296093b
Or allow callers to set timeout in per operation basis. Happy to PR, need guidance on design.
kube-applier-5d6658fb78-7fvnx git-sync I1004 14:45:20.740365 30 main.go:374] local hash: 0c50140fce1d54b1b603649635eae3aaf61223fc
kube-applier-5d6658fb78-7fvnx git-sync I1004 14:45:20.740387 30 main.go:375] remote hash: 0c50140fce1d54b1b603649635eae3aaf61223fc
kube-applier-5d6658fb78-7fvnx git-sync I1004 14:45:20.740397 30 main.go:380] no update required
kube-applier-5d6658fb78-7fvnx git-sync I1004 14:45:20.740413 30 main.go:196] next sync in 0s
kube-applier-5d6658fb78-7fvnx git-sync I1004 14:45:20.740452 30 main.go:435] run("/git/repo"): git rev-list -n1 HEAD
kube-applier-5d6658fb78-7fvnx git-sync I1004 14:45:20.742580 30 main.go:435] run("/git/repo"): git ls-remote -q origin refs/heads/master
kube-applier-5d6658fb78-7fvnx git-sync E1004 16:50:02.298279 30 main.go:176] unexpected error syncing repo: error running command: exit status 128: "fatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"
kube-applier-5d6658fb78-7fvnx git-sync I1004 16:50:02.298310 30 main.go:177] waiting 0s before retrying
kube-applier-5d6658fb78-7fvnx git-sync I1004 16:50:02.298348 30 main.go:435] run("/git/repo"): git rev-list -n1 HEAD
kube-applier-5d6658fb78-7fvnx git-sync I1004 16:50:02.300759 30 main.go:435] run("/git/repo"): git ls-remote -q origin refs/heads/master
kube-applier-5d6658fb78-7fvnx git-sync I1004 16:50:07.961125 30 main.go:374] local hash: 0c50140fce1d54b1b603649635eae3aaf61223fc
kube-applier-5d6658fb78-7fvnx git-sync I1004 16:50:07.961161 30 main.go:375] remote hash: 78015b311e3755beabca38040f30d34d0d68b97a
kube-applier-5d6658fb78-7fvnx git-sync I1004 16:50:07.961176 30 main.go:377] update required
kube-applier-5d6658fb78-7fvnx git-sync I1004 16:50:07.961222 30 main.go:270] syncing to HEAD (78015b311e3755beabca38040f30d34d0d68b97a)
kube-applier-5d6658fb78-7fvnx git-sync I1004 16:50:07.961244 30 main.go:435] run("/git"): git fetch --tags origin master
kube-applier-5d6658fb78-7fvnx git-sync I1004 16:50:12.410303 30 main.go:435] run("/git"): git worktree add /git/rev-78015b311e3755beabca38040f30d34d0d68b97a origin/master
kube-applier-5d6658fb78-7fvnx git-sync I1004 16:50:16.435896 30 main.go:283] added worktree /git/rev-78015b311e3755beabca38040f30d34d0d68b97a for origin/master
Ref: #58
The git-sync container keeps crashing:
This are the variables I tried to run it with:
Environment Variables:
GIT_SYNC_REPO: https://github.com/GITHUB_USER/GIT_REPO.git
GIT_SYNC_BRANCH: master
GIT_SYNC_DEST: git
GIT_SYNC_USERNAME: <set to the key 'username' in secret 'github-secret'>
GIT_SYNC_PASSWORD: <set to the key 'password' in secret 'github-secret'>
Of course, the GITHUB_USER
and the GIT_REPO
are set to the correct values when I deploy,
The Image I'm using is:
gcr.io/google_containers/git-sync:v2.0.4
$ kubectl logs dp-analytics-blog-kgsnh -c git-sync
ERROR: can't create .netrc file: error setting up git credentials exit status 255: error: could not lock config file //.gitconfig: Permission denied
Restructure demo to reflect the most demanded use-case.
Caddy + git-sync with auto-tls.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.