- Manually push to mirror
- Push to mirror using GitLab CI
GitLab has a native mirroring feature and supports both pull and push directions; in pull, it periodically fetch any changes (including addition or removal of branches) from another location and push them to your GitLab repository, whereas in push, it periodically updates of your GitLab repository to the mirrored repository.
I’ve been using this feature mirror Hexojs repository to my GitLab account as a backup. It works well so far, especially when the repository has multiple branches. However, what if you has a simpler repository setup, like my blog which only has two branches and you want to get your hands dirty. Let’s find out.
Manually push to mirror §
The easiest to sync
origin location (GitLab repo in this case) to another location (GitHub repo) is to add another remote,
$ git remote add mirror https://github.com/<user>/<project>
after you push commits to
origin, push to
mirror as well.
$ git push origin master # Or `git push -u` $ git push mirror master # Enter your GitHub credential
Managing multiple SSH identities §
The previous guide assumes HTTP authentication, but what if you (want to) use SSH authentication for both GitLab and GitHub. If you haven’t set up SSH authentication yet, read on.
Generate and add a new SSH key to GitLab and GitHub §
For this guide, I’ll be using ED25519 keypair, instead of the default RSA. This SSH keypair will be your account SSH key and has the same privilege as a personal access token; it covers all the repo that your account has access to, which is different from deploy key also known as per-repository key.
$ ssh-keygen -t ed25519 -C '[email protected]' # Save the key to /home/user/.ssh/id_gitlab_username # The email address is only used for label, not authentication # Enter a password when prompted, that password will be used # to encrypt the private key
id_gitlab_username will be generated in “~/.ssh” folder; they are public and private key respectively, remember to back them up.
Add the public key to your GitLab account in the “SSH Keys” setting. Repeat this step to generate another keypair for your GitHub account, name the file accordingly. The private key is the credential that you will use to authenticate your account.
Configure SSH §
Create a new file in “~/.ssh/config” with the following content,
#github account Host github.com-username HostName github.com User git IdentityFile ~/.ssh/id_github_username Preferredauthentications publickey IdentitiesOnly yes #gitlab account Host gitlab.com-username HostName gitlab.com User git IdentityFile ~/.ssh/id_gitlab_username Preferredauthentications publickey IdentitiesOnly yes
config and private keys must have
chmod 600 permission, adjust accordingly.
Add GitLab and Github as trusted hosts §
When connecting to the SSH server for the first time, you will encounter the following message,
The authenticity of host 'gitlab.com (22.214.171.124)' can't be established.. This is because SSH also verifies the authenticity of the server. To avoid that message, you can import the respective key fingerprints into “~/.ssh/known_hosts”.
$ ssh-keyscan gitlab.com 2>&1 | grep -vE '^#' > ~/.ssh/known_hosts $ ssh-keyscan github.com 2>&1 | grep -vE '^#' >> ~/.ssh/known_hosts
2>&1 | grep -vE '^#' is to remove the comment “# github.com:22 SSH-2.0-babeld-76c80caa”.
Update remote repository location §
The final step is to update the
mirror remote repository locations.
$ git remote set-url origin [email protected]:username/project.git $ git remote set-url mirror [email protected]:username/project.git
It’s a good practice to verify the values,
$ git remote -v mirror [email protected]:username/project.git (fetch) mirror [email protected]:username/project.git (push) origin [email protected]:username/project.git (fetch) origin [email protected]:username/project.git (push)
Next time you push or pull from the repository, you will be prompted with a password that you previously set during
ssh-keygen, to decrypt your private key and authenticate your account.
Use multiple origin location (alternative) §
Alternatively, a neat trick I found is that you could also add multiple locations to the
origin, so that whenever you push the commits, git will automatically push to all locations. This shortcut also applies to SSH authentication.
$ git remote set-url origin --push --add https://gitlab.com/<username>/<project> $ git remote set-url origin --push --add https://github.com/<username>/<project>
The first command will replace the default push location of
origin and only the second command onward will add a new location, so make sure you add both GitLab and GitHub.
As always, verify it.
$ git remote -v origin https://gitlab.com/curben/blog.git (fetch) origin https://gitlab.com/curben/blog.git (push) origin https://github.com/curben/blog.git (push)
Push to mirror using GitLab CI §
Part of this guide is also found in my previous post, Using Git and Git+SSH in GitLab CI, this time I add more explanation for clarity.
The first thing is to set up a deploy key, a public key set in the targeted GitHub repository, in which its private key counterpart will be used by GitLab to authenticate to the GitHub. Deploy key is a per-repository key, thus applies only to a repository; this effectively isolate the privilege of the key, compromise of a deploy key would not affect other repositories, unlike account key or personal access token which affects all of your repo. Nothing stops you from using the same deploy key on multiple repos, but that practice is highly discouraged.
In this use case, since deploy key will only be used in a CI environment, it is not necessary to save it “
/.ssh” folder–you would use your account SSH key to authenticate from your workstation. I use “/Desktop” folder here as it would be easier to locate the keypair when configuring the CI. As always, backup your SSH key.
$ ssh-keygen -t ed25519 -C '[email protected]' # Save the key to /home/user/Desktop/id_github_repo # The email address is only used for label, not authentication
id_github_repo will be generated in “~/Desktop” folder; they are public and private key respectively, remember to back them up.
On your GitHub repository (that will mirror your GitLab repository), go to Settings → Deploy keys → Add deploy key, paste the content of
id_github_repo.pub and tick Allow write access.
On your GitLab repository, navigate to Settings → CI / CD → Variables, add a new Var variable named
GH_PRIVATE_KEY and add the content of
id_github_repo, make sure Protect variable is ticked so that the key is only imported to the “master” branch (which is a protected branch by default).
Add another Var variable named
SSH_KNOWN_HOSTS and the output of
ssh-keyscan github.com 2>&1 | grep -vE '^#'. I explained the necessity of this step in previous section.
Add a new job named
mirror in your repository’s .gitlab-ci.yml. Optionally, you could move Import SSH key step to
before_script if preferred, it does not make any practical difference. However, if you want to use Update GitHub mirror step in
after_script, Import SSH key has to be moved there as well–
script are executed in the same shell, while
after_script is executed in a different shell.
image: node:alpine mirror: before_script: # Install prerequisites - 'which ssh-agent || (apk update && apk add openssh-client git)' script: ## Import SSH key # Launch ssh-agent - eval $(ssh-agent -s) # Add private key from CI variable - echo "$GH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null # Create ~/.ssh folder if not exist - mkdir -p ~/.ssh # Restrict the folder privilege - chmod 700 ~/.ssh # Add GitHub key fingerprint - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts - chmod 644 ~/.ssh/known_hosts ## Update GitHub mirror # ssh connect to GitHub - ssh -T [email protected] || ":" # workaround) force exit code 0 # Add a new remote location called "mirror" - git remote add mirror [email protected]:curbengh/blog.git # Discard changes before checking out branch - git reset HEAD --hard # Push "master" branch - git checkout master && git push mirror master # Uncomment below to push other branch(es), change "branchX" according to your need #- git checkout branchX && git push mirror branchX rules: # Only trigger through push event in master branch - if: '$CI_COMMIT_REF_NAME == "master" && $CI_PIPELINE_SOURCE == "push"' when: always # Only trigger through "Run pipeline" in master branch - if: '$CI_COMMIT_REF_NAME == "master" && $CI_PIPELINE_SOURCE == "web"' when: always