Giter Club home page Giter Club logo

git-mirror's Introduction

git-mirror

This is a script for one-way repo mirroring. It supports Github, Gitlab, and CodeCommit repositories.

Pre-requisites

Setup SSH access for every backend used:

Install pip dependencies:

$ pip install -r requirements.txt

Setup API credentials (Optional)

If you expect the script to create mirror repositories that do not exist (that is, the origin exists but one or more of its replicas don't), then you must follow these steps for each host:

  • Github: git-mirror currently does not support creating Github repos.

  • Gitlab:

    • Create a personal token and save it in the environment variable GITLAB_TOKEN.
    • Optionally set GITLAB_NAMESPACE to a namespace id if you wish the new repository to be created in a particular namespace instead of your personal namespace.
  • CodeCommit: Setup AWS CLI and its credentials.

How it works

git-mirror takes as input a manifest file listing the repositories that need to be mirrored, structured as the following example:

[
    {
        "guid": "my-repo",
        "description": "A script for one-way repo mirroring, supports Github, Gitlab, and CodeCommit",
        "skip": false,
        "origin": "[email protected]:hazimavdal/git-mirror.git",
        "replicas": {
            "gitlab": "[email protected]:replica-set-01/my-repo-replica01.git",
            "aws": "ssh://git-codecommit.us-east-2.amazonaws.com/v1/repos/my-repo-replica02"
        }
    }
]

In this example we have the repository my-repo whose origin is Github and it needs to be mirrored to both Gitlab and CodeCommit. When the script is run, it will do the following:

  1. Clone each repository in the manifest from the URL specified by origin (if it hasn't been cloned already). It is assumed the user running the script can git clone that repo.

  2. git push --mirror to each replica in the replicas field. If the target repo does not exist, the script will try creating it.

Usage

The script has two sub-commands with the following global arguments:

  • -m, --manifest: the manifest file (defaults to ./repos.json)
  • -v, --log-level: log verbosity level (defaults to info)
  • -l, --log-file: full path of the log file name (defaults to .logs/git-mirror.log)
  • --dry-run: if set the script runs in dry-run mode without causing any side effects (defaults to false).
  1. mirror: this is the main operation that performs one-way mirroring for all repositories in a manifest file. The command takes an additional argument:
  • -d, --repo-dir: the directory name where repos should be cloned into (defaults to .repos)

Example usage:

./src/git-mirror.py  mirror                                    \
                    --manifest repos.sample.json               \
                    --log-file .logs/git-mirror.log            \
                    --repo-dir .repos                          \
                    --log-level debug
  1. integrity: this command checks if the state of remotes matches what is specified by the manifest file. For every repo in the manifest, git-mirror checks if the HEAD of every replica matches the HEAD of the repository's origin.
./src/git-mirror.py  integrity                                 \
                    --manifest repos.sample.json               \
                    --log-file .logs/git-mirror.log            \
                    --log-level debug

Recommended usage

I run this script as a cron job on a VM that keeps my repos of interest in sync across hosts. My repos.json is kept in a separate repository and is pulled every time the cron job runs. This way, I can just modify this file and have the changes take effect seamlessly.

Known issues

  • The CodeCommit API ignores everything after a period (.) in the name of the repository that is being created. For example, if you attempt to create the repo hello.world, it will be created as hello, not hello.world. The script will fail in this case. However, if the repository already exists (i.e. not created through the script), then it works without issue.

git-mirror's People

Contributors

hazimavdal avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.