Distributed Git or keeping your remote in sync

In my current setup I have laptop I'm working on, and small local server to play with.

Let's pretend we want to have some web site with bunch of files hosted on server.

We may:

manually ssh into server and edit files there
keep files locally and rsync them to remote
use git 🤔

Goal:

as usual store repo in github
have local clone to edit files and on server the same clone to host site
once if push changes - they should automatically arrive to server
ideally server may want to run some scripts once changes arrived

Before proceeding there are two topics worth mentioning:

bare repo - aka git init --bare - is an option when we want to have an central repo stored somewhere not in github and in our case not as usefull because we want rather forcibly push changes to remote
setup without 3rd parties like github is possible

Here is how can we do it without github

On remote run following:

mkdir ~/github.com/mac2000/demo
cd ~/github.com/mac2000/demo
git init
echo hello > README.md
git add .
git commit -m init

# allow remote push
git config receive.denyCurrentBranch updateInstead

Locally run:

git clone ssh://mini/~/github.com/mac2000/demo
cd demo
echo foo > foo.txt
git add .
git commit -m foo
git push

After push, you should see your changes on remote - profit 🎉

Hooks

pre-receive

There is one more thing we should take care of - if there are some changes on remote - our push will be rejected, and we won't know why, so on remote create following pre-receive hook

touch .git/hooks/pre-receive
chmod +x .git/hooks/pre-receive
vim .git/hooks/pre-receive

#!/usr/bin/env bash
set -e

if [[ $(git --work-tree=.. status --short | wc -l) -ne 0 ]]
then
  echo "[pre-receive]: rejecting push beacause of local changes"
  exit 1
fi

now, whenever you push changes from your local machine, at least it will be clear what went wrong

notes:

from pre-receive hook we can not do things like git clean, git reset, ...
pre-receive hook is ran in .git directory, that's why we passing --work-tree=..

post-receive

Here is second one, for exapmle lets pretend we have node.js project

touch .git/hooks/post-receive
chmod +x .git/hooks/post-receive
vim .git/hooks/post-receive

#!/usr/bin/env bash
set -e

eval "$(/opt/homebrew/bin/brew shellenv)"

echo "[post-receive] npm install"
npm install --quiet --prefix ..

notes:

hooks will not have your profile, as a result empty PATH, so either use full path to executables, or like in this example eval brew env
do not forget about working directory
stdout and stderr of this script will be printed on local machine while pushing changes 💪

With this in place it is already possible to do whatever tricks we want

github

and for github setup is quite simple

let's pretend we have created github.com/mac2000/demo, so on both machines run

git clone https://github.com/mac2000/demo

to allow remote push, on remote machine, run:

git config receive.denyCurrentBranch updateInstead

then prepare pre-receive and post-receive hooks as described above

on local machine our goal now is to push to two remotes instead of one (github)

to see configured remotes run:

git remote show origin

to add one more remote to origin run:

git remote set-url --add --push origin https://github.com/mac2000/demo
git remote set-url --add --push origin ssh://mini/~/github.com/mac2000/demo

note: we need to run this command twice, because initially this setting is unset and as soon as we ran first one, we will loose github, after running this commands git remote show origin should display both push urls

and finally try to change, commit, push something - everything is wired up and you should see log similar to this one:

Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 7.66 KiB | 7.66 MiB/s, done.
Total 5 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: [post-receive] npm install # 👈 post-receive script in action
remote:
remote: added 66 packages, and audited 67 packages in 438ms
remote:
remote: 14 packages are looking for funding
remote:   run `npm fund` for details
remote:
remote: found 0 vulnerabilities
To ssh://mini/~/github.com/mac2000/demo # 👈 pushed to our remote
   6c32237..79b1693  main -> main
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 7.66 KiB | 7.66 MiB/s, done.
Total 5 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
To https://github.com/mac2000/demo  # 👈 pushed to github
   6c32237..79b1693  main -> main

troubleshooting

note if push to one origin fails, it won't stop git and it will try to push to others

in such case there is possibility to catch following issue:

on remote you have some changes
as result push is prevented
but push to github succeed

so keep an eye on git push logs

alternatives

in my case, initially, i wanted to have something independed of github/bitbucket/gitlab/... which is kind of solved

there are other ways to configure something similar is you are stick to github

you may want to configure webhook on github side, it will call an endpoint on your server, which will git pull changes

alternativelly you may provide ssh key to github and create some kind of github action that will ssh into your server and do the work

both cases require server to be exposed somehow to external network and obviously you tied up with github

scp, rsync

also there are

scp -r . mini:~/github.com/mac2000/demo

but as you may guess it will be way to slow

alternativelly we may want

rsync -avz --delete --filter=':- .gitignore' -e ssh . mini:~/github.com/mac2000/demo

but at the very end, having git is an ultimate answer to the question - what files are changed on remote

ticks

also, with git hooks we can do all sorts of tricks, e.g.

inf post-receive hook we are receiving oldrev newrev refname stdin which allows us to answer the question - what files were changed, e.g.:

#!/usr/bin/env bash
set -e

while read oldrev newrev refname
do
  echo "oldrev: $oldrev"
  echo "newrev: $newrev"
  echo "refname: $refname"
  git diff --name-only "$oldrev" "$newrev"
done

you may want to run your script only for main branch

while read oldrev newrev refname
do
  echo "oldrev: $oldrev"
  echo "newrev: $newrev"
  echo "refname: $refname"
  if [ "$refname" = "refs/heads/main" ]
  then
    echo "main branch pushed - do the workd"
  fi
done

for git diff options are:

git diff --name-only "$oldrev" "$newrev" - list file names
git diff --name-status "$oldrev" "$newrev" - list file names prefixed with one letter change code, e.g. A - added, M - modified, D - deleted
git diff --diff-filter=AM --name-only "$oldrev" "$newrev" - list file names filtered by change, in this example only added or modified files will be printed

let's imagin we have nodejs project served by nginx

so our script may become something like (pseudo)

if changed_files.includes('package.json') {
  npm install
}

npm run build

if changed_files.includes('nginx.conf') {
  nginx -s reload
}

if changed_files.includes('demo.plist') {
  launchctl load ~/Library/LaunchAgents/demo.plist
}

...

also, what's important you are not forced to write your script in bash, here is an example of such script written in node

#!/usr/bin/env /opt/homebrew/bin/node

import { createInterface } from "readline";
import { execSync } from "child_process";

const { oldrev, newrev, refname } = await read();
console.log({ oldrev, newrev, refname });
// remote: {
// remote:   oldrev: 'e1be22c77fc76247341a376204d277a75c8b08a5',
// remote:   newrev: '2632b65ca1c3be923a25d9bd53bcb8f6d52b9ddc',
// remote:   refname: 'refs/heads/main'
// remote: }

const files = changes(oldrev, newrev);
console.log({ files });
// remote:   files: [ '.gitignore', 'foo.txt', 'package-lock.json', 'package.json' ]

if (refname === "refs/heads/main" && files.includes("package.json")) {
  console.log("package.json changed, running npm install");
  try {
    execSync("npm install", { stdio: "inherit" });
    // remote: /bin/sh: npm: command not found - do not forget about paths, or eval brew
  } catch (e) {
    console.error("npm install failed", e);
    process.exit(1);
  }
}

async function readStdin() {
  const rl = createInterface({ input: process.stdin, terminal: false });
  return new Promise((resolve) => {
    const lines = [];
    rl.on("line", (line) => lines.push(line));
    rl.on("close", () => {
      // technically there may be more than one line, aka push with commits to different branches
      resolve(lines);
    });
  });
}

async function read() {
  const lines = await readStdin();
  const [oldrev, newrev, refname] = lines[0].split(/\s+/);
  return { oldrev, newrev, refname };
}

function changes(oldrev, newrev) {
  return execSync(`git diff --name-only ${oldrev} ${newrev}`)
    .toString()
    .trim()
    .split("\n")
    .map((l) => l.trim())
    .filter(Boolean);
}