Hi,

I use caddy-docker for a while now and I love it. I want to use the git add-on.

1) My Caddyfile

http://example.com {

        root example

        git {
        repo [email protected]:example-org/staticsite
        branch master
        key /etc/id_rsa
        interval 30
        }
}

2) Docker exec:

/srv
total 8
-rw-r--r--    1 root     root         516 May 19 15:05 index.html
drwxr-xr-x    6 root     root        4.0K Jun 14 20:10 example/

/srv/example/www
total 16
drwxr-xr-x    2 root     root        4.0K Jun 14 20:02 css/
drwxr-xr-x    2 root     root        4.0K Jun 14 20:02 images/
-rwxr-xr-x    1 root     root        3.6K Jun 14 20:02 index.html*
drwxr-xr-x    2 root     root        4.0K Jun 14 20:02 js/

Two challenge here.

1. As I use one container for 1 site, I want caddy to accept traffic for any request, not only example.com. So the Caddyfiel should use 0.0.0.0

2. I don’t know how caddy is suppose to know where to serve files over /srv/example/www

Here is how I imagine it could work but the container does not even run with this Caddyfile:

3) Imaginary Caddyfile

0.0.0.0
root /svr

    git {
        repo [email protected]:example-org/staticsite /svr
        branch master
        key /etc/id_rsa
        interval 30
    }

Thank you in advance!
https://twitter.com/askpascalandy

Not sure if that's the right the title, but here's the behavior:

(Caddyfile)
git [email protected]:user/myproject

Will in reality pull:

https://[email protected]:user/myproject

Which is not the indented behavior, as I'm expecting it to pull via SSH. Doing this way apparently fixes the problem:

git [email protected]:user/myproject {
    key .ssh/id_rsa
}

But I don't want to specify a key. Thanks for reading :)

Edit: I appreciate if someone could test if this issue can be replicated, I haven't tried to do the same on another machine.

This is a PHP website using HHVM via FastCGI with a Bolt.cm installation.

SSH keys are properly configured in both VPS and BitBucket

My Caddyfile:

http://xxxxxx.somewhere.net {
        gzip

        git {
                repo     [email protected]:myuser/mywebsite.git
                branch   master
                key      /root/.ssh/id_rsa
                path     .
                hook     /bitbucketwebhook
        }

        log ./access.log
        errors ./error.log

        root ./mywebsite

        rewrite {
                r /thumbs/(.*)$
                to /index.php?{query}
        }

        rewrite {
                r \.(?:ico|ICO|css|CSS|js|JS|gif|GIF|jpe?g|JPE?G|png|PNG|ttf|TTF|woff|WOFF|woff2|WOFF2)$
                to /{path}?{query}
        }

        rewrite {
                r /phpmyadmin/(.*)$
                to /{path}?{query}
        }

        rewrite {
                r .*
                to /index.php?{query}
        }

        rewrite {
                r /bolt/(.*)$
                to /index.php?{query}
        }

        rewrite {
                r /async/(.*)$
                to /index.php?{query}
        }

        fastcgi / /var/run/hhvm/sock php
}

But every time I retry the webhook, Caddy responds with 403 Forbidden:

Using Caddy 0.8 with Git module downloded from the website
Debian 8 64bits

^ Title explains everything. Same as for GitLab and GitHub.

Some Links:

• General Explanation
• Documentation
• Demo Listener Project

Consider the case where I set up a github webhook with a secret. The github handler decides whether to handle a request based on headers. It will ONLY validate the signature if it determines it is a github request.

If I send a raw POST with no github headers, no signature, and just a json object with a ref, the generic handler will pick it up and pull the repository. I would argue that this should not be. It allows anyone on the internet, regardless of authentication of any kind to make my server pull code and execute commands.

Hi, thanks for awesome plugin!

Unfortunately, it's impossible to use it with default gitolite installation.

Gitolite url: [email protected]:repo is not parsed correctly because of username gitolite@:

caddy[7380]: 2016/04/12 08:08:38 invalid git url [email protected]:reponame

I think it caused by this code (git@ username is hardcoded)

When url is replaced with git@..., then error is disappeared.
But it's still impossible to clone repo, because there are no git user on the server.

I don't want to post it publicly, you can contact me at echo dGhlYmxhemVoZW5AdGhlYmxhemVoZW4uY29tCg== | base64 -d

I've tried to pull from Gogs, but the following won't work:

http://localhost:4500/...

It tries to fetch the https version:

https://localhost:4500/...

That won't work:

fatal: unable to access 'https://localhost:4500/foo/bar': gnutls_handshake() failed: An unexpected TLS packet was received.

There might be many reasons to reload caddy after updating git repo.

In my case I use markdown and templates with markdown directive. Example:

localhost {
  root /var/www/html

  git {
    repo myrepo.git
    path /var/caddy-repo
    then rsync -a --delete --force --update /var/caddy-repos/content/ /var/www/html
  }

  ext .md

  markdown / {
    template templates/default.html
    template nice templates/nice.html
  }
  
}

When I update content/templates/nice.html, without reloading caddy it won't update the template file.

But if I add then sudo systemctl reload caddy to Caddyfile, it gets trapped to reload and git fetch loop.

Is it possible to internally somehow reload configuration? Or is it an issue of markdown directive?

Hi,

Looking at what this does, I think we can also use it for CoreDNS. I'm wondering how minimal the code changes need to be to make this happen.

Or willing to accept PRs that make this middleware work for both? (If possible).

Not strictly an issue, but rather a question.
Here's what I need:
git clone --depth 1 https://github.com/yitzhakbg/levpoem.git /srv/levpoem

Q: How and where to specify the "--depth 1" arguments?

I try to deploy some small apps with caddy git, but it shouldn't updated automatically. Just should clone the current git repo during deployment.

Could you add an interval "once" option instead of the time in seconds?

Originally filed here, but asked by @abiosoft to file in your repo.
caddyserver/caddy#434

I have git enabled in my Caddyfile for a site. If I start caddy with the -quiet=true option I would expect no output on startup at all. But I still see the output of the git initialization every time. The core caddy startup output is the only part that seems to be suppressed.

$ /usr/local/bin/caddy -conf="/home/glenn/www/Caddyfile" -quiet=true
From https://github.com/grempe/www.rempe.us
 * branch            master     -> FETCH_HEAD
Already up-to-date.
2015/12/13 13:34:22 https://github.com/grempe/www.rempe.us.git pulled.
^C

$ /usr/local/bin/caddy -conf="/home/glenn/www/Caddyfile" -quiet=false
Activating privacy features... done.
From https://github.com/grempe/www.rempe.us
 * branch            master     -> FETCH_HEAD
Already up-to-date.
2015/12/13 13:35:28 https://github.com/grempe/www.rempe.us.git pulled.
rp1.rempe.us:https
rp1.rempe.us:http

$ caddy -h
Usage of caddy:
...
  -quiet=false: Quiet mode (no initialization output)
...

I would expect this to quiet all output, not just part of it, from the startup sequence no matter what config options I have provided in the Caddy file for git or any other options.

Hello,

the script for adding new hosts to ~/.ssh/known_hosts always doubles the size of the file. Mine got 600MB and that make every pull very slow.

Also I think the middleware shouldn't reorder as sensitive file as known_hosts is. It should just append necessary lines. Also stripping comments from ssh-keyscan can be done.

My repo contains some stuff that I don't want to serve. I only want to serve the www sub-folder. Right now I have path set to ../git and then I copy the relevant data to the webroot.

Ideally I would checkout to $SOME_PATH/git and then have my webroot be $SOME_PATH/git/www but that would also require caddy to boot when the root doesn't exist (currently attempting that crashes caddy).

Can we have a param to set the path relative to the working-directory instead of to the webroot?

Sometimes the commit history would be force updated, is it possible to support to change hook action to this:

git fetch
git reset  --hard origin/master

It will make deployment more robust than using pull.

I detect problems in git add-on.
When I run the go command, git-addon can not see it. Despite GOPATH & GOROOT registered in the global profile and in the profile of the current user (root). Caddy-server is started using supervisord user = root.

If I write the absolute path to go, then the command is executed, but the execution stage climbs the error that the program does not see the packages used in the program, although they are installed in the system. If I run go install manualy in the program folder, then the command exec without errors.

I am new to Caddy and would like to use this add on. Caddy's own docs don't seem to specify how to install add-ons.

Is there a generic way to do this? If not, how can install this add-on?

Hello,

Currently I am unable to use caddy git to clone a repo where authentication is passed in the repo URL.

Config:

site.com {
    tls off
    root /srv/domain/
    git https://user:[email protected]/project/site.com.git
}

Caddy Output:

2017/07/11 17:46:54 [WARNING] TLS disabled for http://
Cloning into '/srv/domain'...
fatal: could not read Password for 'https://[email protected]': No such device or address
2017/07/11 17:46:54 exit status 128
Cloning into '/srv/domain'...
fatal: could not read Password for 'https://[email protected]': No such device or address
2017/07/11 17:46:54 exit status 128
Cloning into '/srv/domain'...
fatal: could not read Password for 'https://[email protected]': No such device or address
2017/07/11 17:46:54 exit status 128
2017/07/11 17:46:54 exit status 128
2017/07/11 17:46:55 [WARNING] Root path does not exist: /srv/domain/
Activating privacy features... done.

Tested on version 0.10.4

Thank you again for this awesome plugin.

Following up #29

Now I can use http://, but can't specify the port:

http://localhost:12345/user/repo.git

Gives me the following error:

remote: No such host at :80
fatal: repository http://localhost/12345/user/repo.git/ not found

I don't know much about the implementation, but maybe you could just use the URL given by the user instead of trying to validate it?

Cheers.

This builds the site but doesn't serve:
http://levpoem.co.il {
root /var/www/srv/levpoem
tls off
git {
repo https://github.com/yitzhakbg/levpoem.git
path ../../src/levpoem # relative to root. In actuality, /var/www/src/levpoem
branch master
then runhugo
}
}
When I comment out caddy-git, the site just built (above) serves OK:
http://levpoem.co.il {
root /var/www/srv/levpoem
tls off

git {

repo https://github.com/yitzhakbg/levpoem.git

path ../../src/levpoem # relative to root. In actuality, /var/www/src/levpoem

branch master

then runhugo

}

}
It only serves when run without caddy-git. Any ideas/suggestions? How to debug?
This doesn't work either (build for localhost, serve on levpoem.co.il):
http://localhost {
root /var/www/srv/levpoem
tls off
git {
repo https://github.com/yitzhakbg/levpoem.git
path ../../src/levpoem # relative to root. In actuality, /var/www/src/levpoem
branch master
then runhugo
}
}
http://levpoem.co.il {
root /var/www/srv/levpoem
tls off
}

Not exactly sure where i'm screwing this up. I'm running docker container: https://github.com/abiosoft/caddy-docker . When I run the container i get errors:

Warning: Identity file /root/.ssh/id_rsa not accessible: No such file or directory.
Warning: Permanently added the RSA host key for IP address 'x.x.x.x' to the list of known hosts.
Permission denied (publickey)

Yet I can connect like so:

root@ip-10-1-0-15:~# ssh -T [email protected]
authenticated via a deploy key.

You can use git or hg to connect to Bitbucket. Shell access is disabled.

This deploy key has read access to the following repositories:

project/myrepo: -- root@ip-10-1-0-15

Using Caddy 8.2 and Caddyfile is:

git {
repo [email protected]:project/repo.git
branch master
key /root/.ssh/id_rsa
path .
hook /_webhook
}

Any ideas would be much appreciated.

The instruction says:

You can also set up a webhook to pull immediately after a push. 

How can I set it up to pull immediately only after a push?

Thanks,
Andrew

How would I go about recursively cloning a Git repo that uses submodules?

After pulling the repository only a new commit will trigger the then commands
https://github.com/abiosoft/caddy-git/blob/master/git.go#L81

This behaviour prevents to check commit coverage by signed tags if the tags were added later on. To understand my use case: https://carmebar.github.io/2016/10/29/making-a-mirror-of-a-gitbased-blog.html#preparing-caddy

I propose to check for a change in tags too.

Thanks for your work.

Allowing scripts to be executed from /tmp is a security concern for may places (although I don't fully understand why).

In any case, I'm experiencing this problem at present, and it seems the plugin should adhere to environment configuration for temporary files.

I'm guessing this is a fairly unique request, but I need the ability to support multiple hooks on the same repo -- the difference is the branch.

You can see the full Caddyfile here, but here’s the relevant bits:

git github.com/grindjs/docs {
	hook        /hooks/deploy-docs {$DOCS_SECRET}
	hook_type   github
	branch      master
	key         /apps/.ssh/id_rsa
	path        ../resources/docs/master
	interval    -1
}

git github.com/grindjs/docs {
	hook        /hooks/deploy-docs {$DOCS_SECRET}
	hook_type   github
	branch      0.7
	key         /apps/.ssh/id_rsa
	path        ../resources/docs/0.7
	interval    -1
}

git github.com/grindjs/docs {
	hook        /hooks/deploy-docs {$DOCS_SECRET}
	hook_type   github
	branch      0.6
	key         /apps/.ssh/id_rsa
	path        ../resources/docs/0.6
	interval    -1
}

Basically I have docs for different versions of the framework in branches that can be updated, however when I push to a branch other than master, I get the following:

2017/07/19 02:00:45 github webhook ignored. Error: found different branch 0.7
2017/07/19 02:02:13 github webhook ignored. Error: found different branch 0.6

I'm guessing the plugin is hitting the first match and then abandoning?

I have a Caddyfile with two sites, one using the git directive. Caddy doesn't start listening for connections until the pull and then command have run. The server should not be blocked while the git directive is doing its thing. Possibly, the git directive should do something more fine-grained like serve a 500 at its root path until the command finishes.

It looks like Gitlab also supports secret tokens for webhooks. Could we get this supported too? 👍

Hi there,

First, thanks for your great plugin!
I have been running through an issue for 2 days now, which is pretty weird to me.

What I want is to clone a private repository using my SSH key.

When I try running caddy as a service (Debian 8) using: sudo systemctl start caddy
And then query for its status using: sudo systemctl status caddy
Caddy tells me that it could not clone from remote repository because of my public key.
I assumed it was related to my SSH key permissions, so I tried running the exact same command
that my service runs when I start it. Here, Caddy has had no problem cloning my repository.

Unfortunately, I cannot manage to find where the issue is. My caddy.service file is located at
/etc/systemd/system/caddy.service and the user as which the service should run is well set.

My caddy.service file:

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network.target

[Service]
Restart=on-failure

; User and group the process will run as.
User=inad
Group=www-data

; File descriptors limit.
LimitNOFILE=1048576

ExecStart=/usr/local/bin/caddy -agree=true -email=myemail -conf=/etc/caddy/Caddyfile
ExecReload=/bin/kill -USR1 $MAINPID

StartLimitInterval=600

[Install]
WantedBy=multi-user.target

My Caddyfile:

domain.com {
    root /var/www/my-site

    git my-git-repo.git {
        key    /home/inad/.ssh/id_rsa
    }
}

The file /home/user/.ssh/id_rsa has permissions 600, and owner:group is inad:inad.
As I said, when I run manually the ExecStart command from caddy.service in my shell,
it correctly clones the repository, but using systemctl, it does not.

Any help is welcome, thank you in advance!

I'm still seeing the behaviour mentioned on #16 with a download of caddy 0.8.2 + git.

Are the changes supposed to be available in the binaries already?

It would be useful to have the add-on always output something to the console when a webhook is received.

Currently, it seems that when a webhook is received for a branch other then the one specified, it is completely ignored and not shown anywhere.

This caused me to believe the test hooks I was sending were not being received correctly, and spent a significant time debugging it.

A simple message saying a hook was received and ignored since it is not the configured branch would be very useful in situation like these, as most git software I use sends the "test" webhooks as the last commit to the default (master) branch.

With path $CLONE_HERE, the path cloned to turns out to be /var/tmp/\$CLONE_HERE.

Remember that nasty bug where git pull was being called twice concurrently, and they were conflicting with each other and causing the server to not start successfully? (Happened when a server block was associated with more than one host.) I fixed that in 0ac8bf5 by calling the Setup functions only once per server block, rather than once per host per server block.

Well this turned out to be problematic because then the Setup function didn't know which host it was setting up for which was, frankly, kinda dumb on my part. So I reverted the change in a configfix branch so that Setup is called once per host per server block again. This way, like before, each time Setup is called, it's passed a server.Config that belongs to just that one host.

But now it has the possibility of executing your Startup callback more than once (the callback that executes the git pull command) and concurrently - like it did before. But now there are new fields on the Controller that you can use to do something just once per server block, namely the OncePerServerBlock() function 😄

You can use it like this:

// the returned error is the error returned from the function you pass in
err := c.OncePerServerBlock(func() error {
    // this executes immediately as long as it hasn't
    // been executed for this server block yet
    return nil
})

So I would use this function to do your appending to the c.Startup or c.Shutdown slices, for example - or whatever you want to run only once per server block (i.e. once per appearance in the Caddyfile).

Let me know if you have any questions!

It would be cool if it will be possible that git add-on perform multiple commands and run long-running commands, then we will abandon supervisord.

Please don't assume shells are in /bin. This breaks caddy-git on FreeBSD (and probably others that install shells in /usr/local). You could either get a shell path from /etc/shells, or use #!/usr/bin/env {shell}.

In docs you mention private repos won't work on windows. Is this a platform issue with Windows, or a case of testing and implementing something straightforward?

Do you have any pointers on what might need to be done to implement this on windows?

The default configuration is also serving the .git folder. For public repositories, this is not an issue, but it is if you are using private ones.

Is it possible to specify a port to use within the repo URL for SSH?

I've tried using something like git [email protected]:222/mdcollins05/git-repo-name.git {...} and it still attempts to connect on port 22.

Any suggestions or is this not currently possible? Thanks!

Hi! Application runned by Git add-on goes down after several hours & don't restart.

This is error log of application (created by caddy)
13/Jan/2016:19:57:22 +0300 [ERROR 502 /] unreachable backend

This is few strings from caddy.log

[GIN-debug] Listening and serving HTTP on :9000
[GIN-debug] [ERROR] listen tcp :9000: bind: address already in use

This is Caddyfile

mimozaflowers.ru, www.mimozaflowers.ru {
        root /root/gocode/src/github.com/pavlik/mimozaflowers
        gzip
        log /var/log/mimoza_access.log

        errors {
                log /var/log/mimoza_error.log
                #404 404.html # Not Found
        }

        git {
                path ./
                repo https://github.com/pavlik/mimozaflowers
                hook /webhook secret
                then go install
                then_long /root/gocode/bin/mimozaflowers
        }

        proxy / localhost:9000
}

This is my app https://github.com/pavlik/mimozaflowers/blob/master/server.go

I would appreciate if you help to understand what is the reason off app shutdown...

I tried to set the git repo by an env variable like that.
git { repo {$repo} branch {$branch} path / interval {$interval} }

But it looks like it isn't replaced?
parse https://$%7Brepo%7D.git: invalid URL escape "%7B"

Hi there 👋

Thank you for your work on the git plugin for caddy, this is really useful. However I came across an issue that makes things a little complicated for me.

In your examples in the README, there is the following section:

git github.com/user/site {
    path  ../
    then  hugo --destination=/home/user/hugosite/public
}

However even when skimming through the code of this Repo I wasn't able to find out what the working directory of the then command would be. I expect that to be the directory where the repo is cloned to, but that seems not to be the case. At least not on Windows.

In the example above, it would be nice to know where to point hugo if the site root and the clone directory are not the same. I was trying to automatically build a website with caddy using caddy with the following script

localhost:5000 {
    # serve the built hugo site
    root ./www/public

    git https://github.com/my/repo {
        path ..
        # hugo should take the checked out source and build the website according to the config file in the repo
        then hugo
    }
}

It seems a bit unnecessary to have the full git history cloned to a production server, especially if the commit history grows long. I'm wondering if it's a good idea to have the option to use git clone --depth=1 instead of cloning everything.

Thanks.

It says that the git path is relative to root. Root according to my understanding, is the top level directory from which pages are served. If so, the site's sources live in the public page. Isn't that like saying (using hugo as an example):
hugo -d . etc...
instead of hugo -d public etc...??
generating the html in the same directory.
Please clarify.

https://github.com/abiosoft/caddy-git/blob/master/BUILDING.md either should be deprecated or removed from the repository since caddy now builds plugins automatically on their servers.

I want to use Sourcegraph for caddy-git code search, browsing, and usage examples. Can an admin enable Sourcegraph for this repository? Just go to https://sourcegraph.com/github.com/abiosoft/caddy-git. (It should only take 30 seconds.)

Thank you!

I am getting the following error when trying to use this extension:

Cloning into '/srv/html/git'...
fatal: cannot exec '/tmp/.git_ssh.18': Exec format error
fatal: unable to fork
2015/11/23 10:23:41 exit status 128

since that snippet (script) in this case is supposed to be a wrapper i presume that it needs to be a valid script, so it should start with a shabang.

The building readme file contains a reference to github.com/caddyserver/caddydev. This repo does not exist or is private. Please update documentation.

I'd like to be able to supply -s recursive -X theirs as an argument to git pull. (In fact it's maybe worth considering as a default?)

This isn't currently possible at all, to the best of my knowledge, since although:

[pull]
    twohead = theirs

gives the latter (and can be supplied as a --template in Args), it still results in an error because -s recursive is missing.

It's also not immediately clear that Args should refer to clone, and clone alone, so my suggestion is to deprecate it and have CloneArgs and PullArgs.

Alternatively, since I think this is a good default, change pull to fetch and checkout.

I'm not familiar with Go, but I've had a look at the code and I think I could probably manage a PR if this does sound like something your interested in supporting.

Many thanks,

I'm having issues getting my web hook to work. On Caddy start it does pull the latest code from bitbucket as expected. But for some reason pulling using the webhook is not working. I'm thinking that the webhook should never reach the rewrite rule, no? The key file should be working because it is pulling on container start, just not on code push.

Any help would be much appreciated. Here is my info:

CADDYFILE VERSION 0.8.2:

mydomain
tls myemail
#browse
fastcgi / 127.0.0.1:9000 php # php variant only
startup php-fpm # php variant only

git {
       repo    [email protected]:mygitrepo.git
       branch   master
       key      /root/.ssh/id_rsa
       path     ../../src
       hook     /_webhook
       then    cp -pR /srv/src/laravel /srv/
       then    sh -c "cd /srv/laravel && composer update"
       then    sh -c "cd /srv && chown -R nobody.nobody laravel"
}

root /srv/laravel/public

log access.log
errors error.log

rewrite {
    if {file} not favicon.ico
    to {path} {path}/ /index.php?{query}
}

BITBUCKET SET:

Under integrations and Webhook i have the following hook:
https://mydomain/_webhook

ERROR from bitbucket:

Response from https://mydomain/_webhook
HTTP status: 403
Elapsed time: 1590ms
Request time: 7 minutes ago (Tuesday, April 26th 2016, 3:56:10 pm)