Mirror remote repository feature about gitblit HOT 69 CLOSED

I have a plan for this now.  I will implement a "mirror repository" feature which allows
you to clone a remote repository and optionally set an auto-pull frequency (never,
15mins...1 week).  Not sure if RefSpec should be controllable or if it should be hard-wired
to basically everything.  The repository will have the usual controls (freeze, access
restriction, etc) and it will be up to the user to ensure that the mirror is not really
committed to... or if it is, the commits must be on a separate branch.  This feature
would be most useful for teams tracking a 3rd party dependency.

Pushed this back to the next release.

Issue 348 has been merged into this issue.

Moving to next release

Hi, is this feature implemented?

Nope.

Are you referring to the federation client ? 
The last snapshot doesn't crash , but 1.3.0 only pulls the first 
state of the repository . Subsequent calls don't pull the latest changes 
Is that related ? 

This feature is a little bit like the federation client.  Github offers a mirror feature
- this would be exactly like that.  Clone & periodically fetch a specific repo from
any source.

Federation client doesn't (properly) pull.  I wasn't aware of that, please report this
as a separate issue and attach any relevant console messages.

This feature would really make gitblit my all in one tool. I prefer having a full local
clone of any library I use. This also involves using git-svn sometimes. I vote for
this feature!

Yeah, I could use this myself; I just need to _do_ it.  I can tell you, however, that
git-svn would not be on the menu.  Gitblit is a pure Java solution and git-svn isn't
implemented by JGit.  Even if I wanted to implement my own git-svn I believe SVNKit's
license is incompatible with the Apache license.  Maybe not - I can't recall the details
but it seemed like an issue.

When you get around to implementing this feature, I could have a look at implementing
this for SVN. I have no need to push back to svn repo's, I just want to have a copy
of them when I work with them.

Perhaps a 2 pass system, where a background script just clones and maintains a normal
git-svn repository (or even git-hg), and then Gitblit mirrors from that. That's how
I currently have things set up on my dev server, with a cronjob.

Hi James
¡To_do_it!, yeah... an idea for what it help.... :)
You should dial the repository as a mirror, not accept push (it may not solve conflicts),
only fetch from origin and then pull (auto merge) with the branch head (default of
origin - no possibility to change).
An periodicity/interval should be scheduled.

P.S 
This I have done with a linux script and crontab hourly - don't know if you remember
I told you -.

Thk

I pushed an implementation of a mirror executor to master.  There is no ui for initially
creating the mirror so you'll have to git clone --mirror {yada}.  SSH probably won't
work.  Credentials also probably won't work... but who knows.  It runs periodically
for all identified mirrors.

I suppose that I'll do a "clone repo" page at some point, but it's not high on my list
right now.

Live mirror here: http://next-gitblit.rhcloud.com/summary/jgit.git

I added some more mirror repos and move the jgit mirror so the above link is now invalid.

Thank you James! I'll have a look this weekend, or monday when i'm back at the office
:) 

I know bug trackers aren't supposed to be used for the "me too" and "+1" comments but
with this item I just couldn't resist ;-) So here goes: Thanks James! Really cool and
useful feature!

Just to clarify a few things...
1) Will it be possible to say which branches to mirror? I don't think this is actually
a necessary feature but thought I'd ask anyway.

2) I assume that you'll be able to have your own branches in a mirrored repository?
So the remote branches/tags/commits, etc. will all be present in the mirror along with
any branches created specifically in the local mirror.

3) Will there be an option for "Just in time" syncing with the remote I.e. The mirror
does a "pull" from the remote on a pull from or push to the mirror? You could have
a fairly long sync interval to catch the times when the local mirror is quiet but would
be up-to-date whenever any operations occur.

4) Will I be able to convert existing repositories into mirrors? Will there be a UI
element such as a "Convert to Mirror" button and URL field to do this or will I just
run...

$ git remote add --mirror=fetch origin <url>

...in the appropriate repositories and GitBlit will pick them up as mirrors (after
a restart)?

-----
5) Will this lead to a bi-directional mirror? Essentially commits pushed to the local
(mirror) repository would be pushed to the remote, the push to the local would fail
if the push to remote fails, etc.

Cheers,
Alex

alex:
1. A true git mirror always includes all branches, no need to specify. It's a full
100% copy of the server.

2. I wouldn't recommend that. I'd recommend starting a new clone from your mirror,
and push to that. Leave the mirror as-is.

3. This is not really what mirroring is intended for in git.

4. Why would you want to mirror your own repositories?

I suggest you do some reading into git clone --mirror :)

robindegen:
I suppose that's the crux of the questions, is this essentially a "git clone --mirror"
or something above and beyond that?

So the use case I'm thinking of is (I'll use jgit as an example)...
* I'm working on a team that's adding a feature to jgit.
* We "mirror" jgit to a central GitBlit instance.
* We created a branch for our work in the GitBlit jgit mirror for the teams work.
* We'd like the master (and possibly any/all of the public jgit branches) kept up-to-date
automatically.
* We can browse the latest work on jgit from out GitBlit instance, more easily merge
changes into our feature-branch, etc. all done via GitBlit rather than needing to interact
manually with the remote jgit repo.

I wasn't suggesting I mirror my own repositories. Like you, I like to have a clone
of a 3rd party library in my GitBlit instance. Until now I've just cloned the remote
repository to my machine, added my Gitblit instance as a remote and then pushed to
GitBlit. To keep my GitBlit instance up to date I've subsequently (on my local machine)
pulled from origin and pushed to GitBlit. My GitBlit copy may also include branches
that I've created for my own work. So I was wondering whether I'd be able to convert
this type of repo in GitBlit into a "mirrored" one and in doing so it would replace
the manual "pull from origin and push to GitBlit phase" and do that automatically for
me on the GitBlit server.

I'm also trying to avoid having direct Git command line access on the GitBlit server,
setting cron jobs, etc. I.e. consolidating configuration in GitBlit so it's easier
to manage, backup, etc. in a corporate environment. 

is this essentially a "git clone --mirror": yes, as described in comment #18

@alex: I understand what you want.  It was what I planned to do 2.5 years ago when
I opened the issue but that is not how it works presently.  Gitblit will refuse pushes
to mirror repositories which makes your topic branch idea a no-go.

This will eliminate the pull/push you are doing now to update your manual mirror BUT
it won't co-exist with your own topic branches.  The mirror repo is a pristine copy
of upstream, not a hybrid of your content and upstream.

The mirror executor eliminates the need for fetch cron jobs, but it does not yet eliminate
the requirement for direct Git command line access on the server to initially create
the mirror.  Baby steps.

Thanks James for confirming that.

Yes most definitely one step at a time, I realise you're very busy. 

It just so happens that essentially the scenario I've talked about has come up in conversation
in work and I noticed coincidentally that there'd been activity on this feature so
I thought I'd ask some questions.

If I can help at all then I'd be happy to. I'm not sure how much time I'd be able to
devote but I would try to give as much as I can. If there's anything in particular
you'd like me to do then let me know. Obviously I'm not familiar with the code but
I'm sure I'd pick it up soon enough. UI changes would take me longer but the Java parts
won't be a problem.

@alex: what about (brainstorming)...

this part is standard distributed workflow
1. you mirror jgit - it's a pristine, upstream copy
2. you fork jgit - ~alex/jgit.git
3. you code on topic branches and push to your fork ~alex/jgit.git

this part is different and not completely thought out :)

Gitblit (optionally?) tracks the fork origin.

4. gitblit prevents pushes to your fork on branches that are tracking the fork origin
5. gitblit automatically updates tracked branches in forks OR maybe there is a button
to manually update cloned branches in forks

This would get you automatic "git pull" ref updates from upstream->mirror->fork->working
copy but the tradeoff would be blocking pushes to tracked branches.

One of the things that bugs me about Github is that inexperienced contributors sometimes
fork, commit on master, push master, and then open a PR from their master to mine.
 That's fine for a one-off, but this creates pain if they contribute again - and re-use
their master.  There is a high probability there is a merge somewhere in there which
screws up the works.  That is partially what I am trying to avoid in 4+5 while offering
you a "mirror" experience for tracked refs.

Now I don't know what happens for new refs pushed to the origin.  They would presumably
propagate to the forks, but it gets messy with potential collisions.

Like I said, not completely thought out - and related to, but not exactly, what this
issue is about.

One mirror behaves something like a working copy in some aspects, unless now gitblit
can upgrade (fecht -> http://git-scm.com/book/en/Git-Branching-Remote-Branches) and
then make a pull to the master branch only.
To do this (for now) you need to create the repo on the command line using a mirror
-> https://help.github.com/articles/duplicating-a-repository.
Of what it's to give some functionality to mirror repo, choosing which branch you want
to see, but beware, this branch is not the one doing the merge, that is responsible
for the origin repo manager/users, Gitblit only does the merge after fetch of the branch
that is marked as master.

Regards

James:
4. That may be a nice feature to have. Even better would be if this is user based (just
like the other userrights)
5. To me this is low priority, just a "nice to have". I usually manually pull from
the mirror directly, and then push changes back to the fork if any.

I guess this all boils down to properly using Git. The real question here is if it
should be enforced by the software or not. A pull request should never be done on master
really. Just instruct people to work on a branch, and merge with your master on their
end prior to a PR. This is the git workflow as done by Linus Torvalds (linux kernel),
and we also work like that in our company.

You can't never push a mirror over Gitblit, as Gitblit can't resolve conflicts in the
branches, that could result, for the action of fetch - pull acts as a third party.

@eguer
Very true. With my 4 I meant the pushes you can do on a forked cloned repo. Not directly
to the mirror ofcourse. In my opinion there's no reason to ever push to a mirror at
all. It's only designed for having a full local copy of somebody else's project/library
to keep safe.

Here you can see clearly the difference between fetch and pull -> https://help.github.com/articles/fetching-a-remote

@eguervos: I don't understand what you are trying to say in #29.  I think it's important,
but I just can't figure it out.  Can you reword?

@robindegen: User rights.  I don't understand how that comes into play.  Can you elaborate?

For point 4, where you disallow pushing to one of the branches that are actually on
the mirror.

So the workflow would be as you described. You mirror something. You fork it, and then
based on your userrights you're allowed to push to branches that also exist on the
mirror or not. Regular users should be prevented, and they can only push to their own
branches from that fork. Some admin user might need to push to it for whatever reason.

If we (ever) allow pushes to a branch sourced from a mirror then I don't think we can
have point 5 because we end up in a merge situation - which is what I think eguervos
is trying to say.

One alternative to this would be to clone the mirror & push a completely new repo with
no fork relationship to the mirror.

Hmm that is true. Good point. Then I guess point 4 should just not be made posible.
I can't seem to think of any good reason to really need to be able to anyway.

I guess your alternative is only interessting if the project you originally cloned
from  doesn't exist anymore, so there will be no more remote updates for it. Then you
could opt to convert it to a normal repository.

"Then I guess point 4 should just not be made posible. I can't seem to think of any
good reason to really need to be able to anyway."

So... you propose leaving everything as-is and not implementing points 4 & 5?
Or do you mean, yes, implement 4 (and maybe 5) but forget the permissions loop-hole?

I mean the latter.. have gitblit disallow the push. That way it prevents the user from
causing merge trouble.

@James
Just that after a fetch should be a pull to the master branch, since I have observed
that in the absence of pull does not update deleted branches and the possibility to
display a particular branch, not necessarily the master.

@robindegen
Read this https://groups.google.com/forum/#!topic/gitblit/vsc9MEWaXs4

@guervos:  Here is where this gets special.

The pull workflow assumes you have two refs which can both change and because of that
a merge may be required.  Point 4 says the fork does not *own* the cloned branch, the
mirror does.  This would allow Gitblit to update forks from the origin and reset (ref-update)
all tracked branches to the mirror's state -- except HEAD which would point to whatever
branch the fork owner wanted.

Gitblit need make a pull (git pull <remote> - as the master branch as not changed) followed
after the fetch (git fetch <remote>). Because when do not pull in the mirror, does
not delete the origin branches are found deleted. The fetch, don't merge anything and
it's necessary if you want to work on a clone of the repository mirror that you have
in Gitblit server.

This problem I have found when compiling branch in mirror repos clones from my server.

Furthermore, I can have as many copies of a mirror as branches have, in order to follow
the work of a team, this will eliminates the need to constantly check out between branches.

"Gitblit need make a pull (git pull <remote> - as the master branch as not changed)"

Basically, true.  If a fork's tracking branches become special (currently they are
not) then after fetching from the origin (refs/remotes/origin/xxx), the tracking branches
(refs/heads/xxx) would forced to match their origin counterparts (refs/remotes/origin/xxx).

Since Gitblit forks are bare repositories, you can not "pull".  "pull" requires a working
directory to resolve merge failures - which a bare repo does not have.  And we won't
care about merge failures - because we are not merging - we are tracking the origin.
 The git equivalent would be to fetch, as you say, and then use the update-ref plumbing
command to change the sha-1 that the tracking branches (refs/heads/xxx) point to.

"Furthermore, I can have as many copies of a mirror as branches have, in order to follow
the work of a team, this will eliminates the need to constantly check out between branches."

I don't understand this point.

"Furthermore, I can have as many copies of a mirror as branches have, in order to follow
the work of a team, this will eliminates the need to constantly check out between branches."

Is the reason of why having the repository with a name in order to differentiate it
from other mirror of the same but different branches showing.

Essentially a mirror is a working copy, in fact Gitblit with that repo should behave
the same way, the difference lies in we can clone a mirror and it's useful, that can
only be achieved if gitblit doing pull master branch.
The 1st clone (mirror) is set by the HEAD, the fetch does not move, if you clone this
repo find that what you have cloned is dated when cloned (created the mirror) as the
fetch put the commits to the work area and not in the working directory, and this is
important to know, if you want to clone a mirror and working with it.

It sounds like you are using multiple Gitblit instances (one per developer?) on repos
with working copies.  This will only be for server-side forks in a centralized environment.

or team
...on repos with working copies. <- yes
...This will only be for server-side forks in a centralized environment. <- or mirror
with automatic scheduled updates

Mirror that behave like working copies, preventing pull, but allowing be cloned.

When you show log of a repository that is working copy that have been made various fetch,
in reality what you see is the work area overlying from the working directory, the
current working directory is where the HEAD sign.
If you make a clone of this, that drag to your clone is where the HEAD points not the
working area, that may be much higher, when you do a "show log" of this clon, is when
you realize you don't have reality  the work done to date, but the one on the date
you created the mirror.

Now, it's clear.

By definition, a "git mirror" is a *bare* repository with a refspec of "refs/*:refs/*"
(and a mirror=true flag).  It does not have a working copy (checkout) and can't be
treated as a normal checkout.  Every time I have used the word mirror in this discussion,
that is what I have meant.  I'm not inventing an alternate definition of "mirror",
just using standard "git clone --mirror".

A Gitblit fork is also a bare repository and can not be treated as a checkout.  Anything
else is a repository clone with a working copy.  

Despite the fact that you can use Gitblit with working copies, as you do... I'm not
going to touch non-bare repositories with this feature.  Gitblit is meant to be a centralized
git server.  It does have some support for client-side use, but for now I'm not going
to consider that use.

Mirror works fine, with what you've done so far, it is enough (so far).

Thk so much

+1 Me too! ;)

No seriously, this sounds like good stuff. We, too, are currently using "git clone
--mirror" with subsequent "git remote update" in a cronjob. It would be perfect to
have all of that in the Gitblit UI, but having Gitblit replace the cronjob would already
be cool.

Here is another thought from how we use this. We mirror not only to have a remote site's
repos locally, but also use the mirrors to implement business continuity. That is to
say that if the remote site goes down, we switch the mirror repos on the mirroring
site to be the active master that developers can push to who have been using the remote
repo so far. Once the remote site is operational again and the repos hosted there can
be used for pushes again (i.e. have been synced back) we switch the local repos back
to mirror mode.
Of course, it would be great to be able to do that with Gitblit support. =)

I have a snapshot build running from a5c69b6b7 on our server. I will let you know if
we run into any issues with mirrors.

@Florian: That was part of the intention of the Federation feature but I don't think
it ever quite got there.

At first i thought my repo did not show up, but i had to clear cache. Seems to work
nicely now. I will check in a few days if it actually synched properly.

@James: My mirrors do not appear to be updating. It identifies them as a mirror properly
(showing the icon), but I mirror cloned for example the linux kernel 2 days ago, and
it now just shows last commit 2 days ago, even though there have been recent comments
the last 2 days.

Is there anything else i need to set up in the config or something? Currently i'm running
git commit a5c69b6b7...

Sounds like the mirror executor is not running; it's disabled by default.
Did you enable it?

git.enableMirroring=true

Changing this requires a restart.

Thanks, I'll ty that when in the next maintenance tonight.

Well, what would be the difference between the federation and the mirror feature then?
The difference I can see is that you don't need the remote using Gitblit for the mirroring
to work. Which is what I am looking for since unfortunately not all of our sites are
using this brilliant tool.

One difference is that the federation feature was written 2 years ago and the mirror
feature was written last week.  :)

Mirror is a focused subset of federation suitable for use with generic http/https/git
servers.
Federation was written to continuously replicate one gitblit to another: repos, users,
settings, etc.  If new repos are added to Gitblit A, and Gitblit B pulls from Gitblit
A - Gitblit B automatically clones Gitblit A's new repositories.

The Mirror is only about updating refs for a specific repository that you have manually
configured as a mirror.  You could use the Mirror feature to pull from another Gitblit.

Federation can also be used in other clever ways since it supports repository subsets
and various permission levels.

I guess you could summarize as:
Federation = Backup/replication
Mirror = Tracking 3rd party repo

Well, I would have liked to use Gitblit at every site in order to enable federation,
to implement the scenario described above. I it is out of the scope for Mirroring,
reading your explanation.

Turning it on seems to have worked fine. My mirrors are properly being updated now.

I'm simplifying this issue.  Gitblit now has a mirroring mechanism and it's been working
well for months.  Separate issues can be opened for enhancing use of the mirroring
mechanism (e.g. create mirror from web ui).

1.4.0 released.

Hello git experts ,

I have tried like below 


git clone --mirror [email protected]/numerics.git:29418/repo/tools/
git remote update
git push --mirror git@localhost:29418/numerics.git:29418/repo/tools/

with above I can get all refs and branches  but  if I will do/create local branch and
I will do git push few fixes to my local branch & to master branch in git@localhost:29418/numerics.git:29418/repo/tools/


then how I will merge or add the latest changes from [email protected]/numerics.git:29418/repo/tools/
to my local repository git@localhost:29418/numerics.git:29418/repo/tools/

Do I need to create like 

[remote "origin"]
    url = [email protected]/numerics.git:29418/repo/tools/
    mirror = true
[remote "mirrors"]
    url = git@localhost:29418/numerics.git:29418/repo/tools/

or git push to mirror and rename to again origin 

Hi, you need make a fork of this repo-branch to push in. Don't push o work
with mirror , mirror it's read only o to fork. Mybe can federate? I don't
remenber if this is it.
El 21/10/2014 06:17, <[email protected]> escribió:

eguervos is right.  You need to think of a git mirror as an actual mirror.  It reflects,
it does not alter/add.

In the scenario you are describing you have 3 repos:

  [Github]       [Gitblit]
      |              |
      |              |
      +----[Clone]---+

Gitblit does not have any special features for working with working copy clones other
than viewing them.

In this 3-repository scenario you would use command-line git from your clone to pull
from Github and push to Gitblit.

The mirror feature allows you to keep a bare clone in Gitblit of your Github (or any
other source) repository.  This is a one-way, perfect-copy scheduled operation.

  [Github]------>[Gitblit]

One perhaps overlooked feature of Gitblit mirrors is that if your source repository
is a Gitblit-hosted repository which uses the BranchTicketService (for example https://dev.gitblit.com/summary/gitblit.git),
a Gitblit mirror of this repository will automatically include the complete ticket
information.  It will be read-only - because it's a mirror - but everything will be
there.