local caching for git repos

[ link: git-caching | tags: git scm | updated: Wed, 03 Jan 2007 10:20:54 ]

I try to minimize the amount of data I pull from git repositories. To do this I have a directory on my file server that has a bunch of clones of git (and hg and previously bk) repositories. All of these are exported and mounted on my other machines in /site/scm/. I will refer to this as cache :)

Next, I have a cron job that regularly updates those trees from the their upstream counterparts. All my working copies are cloned from those repositories using the --local --shared mode, or using --reference if I think I will be committing upstream any time soon.

Using local git cache to build an unmodified tree

If I just want to build a kernel I would run:

git-clone --local --shared -n /site/scm/git/linux-2.6.git linux-2.6-build
git-tag -l
git-checkout -b my-build-branch v2.6.18.1

This will create a new clone in linux-2.6-build, and pretty much leave the tree empty -- the result is a tree that's about 500k in size. Next, I look at the tags available and decide I want to build v2.6.18.1. I do a checkout onto a new branch called my-build-branch. The result is a tree with all actual history stored on the file server, and only the v2.6.18.1 version in the build directory.

Setting up a working tree with commits to upstream

Next, if I am working on a project that will end up generating commits I want to push up, then I would run something like this:

git-clone -n --reference /site/scm/xelerance/openswan.git \
            git+ssh://vault.xelerance.com/.../openswan.git/ \
            openswan-work
git-checkout -f -b my-public public
echo "Push: my-public:public" >> .git/remotes/origin

This one is a bit more complicated. The git-clone will create a repository that is seeded from vault.xelerance.com, but only if the objects are not present in the local cache. All pulls and pushes are done with vault over ssh, but before any objects are downloaded, the local cache is first checked. Next step is to create a working branch called my-public on which I will make my changes to the upstream public branch. Finally, I record that I want to push my changes from my-public to public upstream.

Using personal working branches and upstream-tracking branches

The reason why having distinct my-public and public branches is not initially obvious. The local public branch tracks the upstream repository's public branch. Initially, it contains none of my local changes, just what other developers have pushed. I do all of my work on my-public, and only push after I have merged those public changes with my code.

A common interaction would be

  • git fetch - updates the public branch, and possibly other branches, with upstream's changes
  • git pull . public - update current working branch, my-public, with changes from public
  • edit, compile, test, commit
  • git fetch - get the changes done in parallel
  • git pull . public - merge the other developer's work into mine
  • git push - push changes on my-public to public upstream
  • git push - update local public branch to track latest `public in upstream

Finally I should mention that the my-* branches never leave my repository. They are only used to differentiate local branches from upstream-tracking branches.

Update script

Oh... and finally, getting back to the cache, here is the script that updates all my repositories in my cache.

#!/bin/bash

for d in */.git/../ ; do 
        ( 
        cd $d 
        echo
        echo ================= $(pwd)
        echo
        remotes=$(cd .git/remotes/ ; ls | grep -v '~$')
        branches=$(cd .git/branches/ ; ls | grep -v '~$')
        for o in $remotes $branches ; do
                echo ----------------- fetch $o
                git fetch $o
        done
        echo ----------------- repack
        git repack -a -d 
        )
done

Merging multiple remotes into one cache

Here is another great feature of git. By default the only remote that git tracks is origin. But, you can have multiple entries under .git/remotes/ directory that allows you to track multiple remote trees that share common ancestry.

I think this is most evident in the linux tree.

$ cd /site/scm/git//linux-2.6.git/

$ ls .git/remotes/       
origin  v2.6.15.y  v2.6.16.y  v2.6.17.y  v2.6.18.y

$ cat .git/remotes/origin 
URL: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
Pull: master:origin

$ cat .git/remotes/v2.6.18.y
URL: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.18.y.git
Pull: master:v2.6.18.y

This allows me to compare stable trees with Linus trees. But best of all, when I clone this tree -- as shown above -- there is only one place that stores all the common ancestry: the cache.

Bart's Blog

/git-caching

About

I am an embedded Linux software developer and consultant operating under Jukie Networks Inc in Ottawa, Canada.

RSS Feed - Full Content

RSS Feed - Full Content

ipv6 ready

Valid XHTML 1.0 Transitional

Created with Vim

Created with Perl




Tags

abiword adam android apt asm ata audio bacula bash bios board-game boot bug busybox bzr c code conference debian debug desktop devel disk dns dpkg dream email embedded firefox font fs fun g1 gdb geode git git scm git-find git-graft go google gpg grub hardware html hw inkscape ion3 ipsec ipv6 ipv6summit.ca irc irssi joke kdb kernel klips kqemu kvm laptop lastfm ldap lighttpd linux lua lvm mail make mercurial meta mmc mouse-free mp3 mpd music mutt net nexusone nfs oclug ogre ols oom OOo openswan opteron osx ottawa perl photography power pxe qemu raid redhat rogers ruby sata sbc scm screen script scsi security sheeva shell skype spam squid ssd ssh suse svn tags talk thinkpad ubuntu uml unix urxvt usb uzbl vi video vim vimgit virt vpn vserver web weechat wifi wind wmii wmiirc-lua work x x86 x86emu xen xen-box-setup xfs xterm zsh

Posts

[ 20130409092758 ]
automount mmcblk devices

[ 20130316190507 ]
tunnelbroker vs IRC

[ 20110616180255 ]
HDD -> SDD

[ 20110517204617 ]
how to manually create a 6in4 tunnel

[ 20110501202915 ]
Presentation slides published!

[ 20110320190458 ]
What is IPv6, and why should I care?

[ 20110309230106 ]
Ottawa IPv6 Summit 2011

[ 20110309225149 ]
I am now an IPv6 Sage

[ 20101214145558 ]
It's a Holiday Miracle

[ 20101103095130 ]
ipv6 on your desktop in 2 steps

[ 20101020113026 ]
growing a live LVM volume

[ 20100915175649 ]
distributing DNS list through radvd

[ 20100908173402 ]
Canadian ipv6 drought

[ 20100903190623 ]
ipv6 certification

[ 20100723192259 ]
m4a to mp3

[ 20100723134522 ]
git 1.7.2 is out

[ 20100523092001 ]
console=ttyS0 with grub2

[ 20100426162644 ]
How many times is my function used within an executable?

[ 20100412100135 ]
vmlinux on Ubuntu

[ 20100328152951 ]
where your WIND coverage ends

[ 20100326113354 ]
Nexus One live in Ottawa on WIND Mobile

[ 20100311133411 ]
the WIND excitement

[ 20100302160743 ]
sata hotswap pico-HOWTO

[ 20100227093435 ]
serving http content out of a git repo

[ 20100225104049 ]
pimped out zsh prompt

[ 20100221184126 ]
live termcasting of your terminal over telnet

[ 20100215083725 ]
Debian on UBIFS upgrade on SheevaPlug

[ 20100214205532 ]
nexus one

[ 20100130150527 ]
skype on Debian Linux (64bit)

[ 20100125191021 ]
I am so peeved at Rogers

[ 20091205130455 ]
running really nice

[ 20091120225902 ]
notmuch for vim

[ 20091113102221 ]
squid and apt

[ 20091104195344 ]
using WIP branches to save every edit

[ 20091104194146 ]
using git workflows to avoid loosing intermediate changes

[ 20091019183503 ]
reflections on ACM Reflections

[ 20091013174248 ]
bacula rejected Hello command

[ 20091007235046 ]
pimping out git log

[ 20091002094400 ]
virtualization primer

[ 20090922110756 ]
adding an external encrypted volume under Debian

[ 20090905112529 ]
the dreaded process of rooting Rogers Dream

[ 20090830174551 ]
Cyanogen's recipe for Cupcake/Donut-like pastry

[ 20090827100232 ]
simpler android rooting

[ 20090821095427 ]
androids don't like water

[ 20090820215358 ]
prettier function tracing

[ 20090816151854 ]
Baking a cupcake

[ 20090814131559 ]
Hello Android!

[ 20090813100025 ]
sim unlocking a G1

[ 20090731185522 ]
rooting the droid

[ 20090727210307 ]
getting into android

[ 20090714094340 ]
importing an old project into git

[ 20090702113222 ]
Why pick Git?

[ 20090628131255 ]
select loop for X events

[ 20090625181315 ]
portable printf

[ 20090623225800 ]
switching to uzbl

[ 20090622223147 ]
Linux Symposium

[ 20090622214023 ]
bringing git-format-patch to bzr

[ 20090612215638 ]
nfs local caching with fscache and cachefilesd on Lenny

[ 20090610202041 ]
Scott Chacon smacks git around

[ 20090610150039 ]
how would you read a file into an array of lines

[ 20090609215141 ]
libguestfs

[ 20090609000208 ]
tiding up the PATH

[ 20090608232531 ]
only showing relevant messages in mutt by default

[ 20090608010405 ]
git-vim hacking

[ 20090530223801 ]
mark-yank-urls: fix bug allowing shell to interpret the url

[ 20090509212648 ]
wmiirc-lua v0.2.8 release

[ 20090509113125 ]
wmiirc-lua moving to github

[ 20090504101605 ]
two terminals one PWD

[ 20090501172645 ]
splitting files out of a commit

[ 20090409155905 ]
git workflow: git amend

[ 20090401112030 ]
how old are these files in git?

[ 20090330181138 ]
sles 11 on kvm

[ 20090323194942 ]
android true type font

[ 20090322203939 ]
popen with stdin, stdout, and stderr

[ 20090320214228 ]
shrinking URLs

[ 20090304004744 ]
readlater

[ 20081112150409 ]
splitting patches with git

[ 20081011081638 ]
creating busybox symlinks

[ 20081002215121 ]
wmiirc-lua v0.2.5 release

[ 20080916155113 ]
git-svn strangeness

[ 20080915112959 ]
installing git man pages quickly

[ 20080913112345 ]
wmiirc-lua v0.2.4 release

[ 20080825100454 ]
Kernel Walkthroughs - booting

[ 20080813210205 ]
Linux Kernel Booting

[ 20080719211329 ]
printable OLS/2008 schedule

[ 20080715214447 ]
wmiirc-lua updates

[ 20080713194704 ]
Git Screencast

[ 20080713143429 ]
four steps to reproducible Debian installs

[ 20080705150651 ]
USB2.0 enclosure benchmark

[ 20080703230924 ]
Linux Kernel Walkthroughs posted

[ 20080702113602 ]
Introducing the Ottawa Ruby folks to Git

[ 20080628160732 ]
Authenticating Linux against OSX LDAP directory

[ 20080627142123 ]
Canada Day Events 2008

[ 20080613162541 ]
Linux Kernel Walkthroughs

[ 20080510083828 ]
is my usb device connected to a fast port?

[ 20080509111534 ]
show more git info on zsh prompt

[ 20080430104202 ]
git-vim

[ 20080412100337 ]
color your word

[ 20080404105620 ]
show current git branch on zsh prompt (2)

[ 20080303200359 ]
how to track multiple svn branches in git

[ 20080301134220 ]
fixing X for GeodeLX

[ 20080108002540 ]
kvm nfs hang

[ 20080107160836 ]
screen -c relative path bug

[ 20080105132854 ]
WeeChat spell suggestions

[ 20071219221358 ]
show current git branch in zsh

[ 20071217141037 ]
wmiirc-lua kitchen sink repository

[ 20071212100316 ]
protecting sshd from OOM killer

[ 20071204234232 ]
wmiirc-lua v0.2.1 remembers a bit more

[ 20071013205336 ]
wmiirc-lua v0.2 has suspend and raw modes

[ 20070929112345 ]
wmiirc-lua debianization

[ 20070924104140 ]
zsh tip of the day - global aliases

[ 20070915094213 ]
wmiirc-lua v0.1.1

[ 20070913130838 ]
comparing two directories

[ 20070909204125 ]
git-rebase --interactive

[ 20070908115905 ]
zsh tab completion awesomeness

[ 20070902000736 ]
wmiirc in lua

[ 20070831150306 ]
debugging with -dbg libraries

[ 20070831142646 ]
svn status like output in git

[ 20070829141847 ]
Git Cheat Sheet

[ 20070821142038 ]
switching to abiword

[ 20070811105746 ]
forwarding ssh and X through screen

[ 20070807112531 ]
git-svnup

[ 20070724082355 ]
reducing power consumption

[ 20070722123734 ]
Makefile template

[ 20070722002649 ]
less, colourful

[ 20070719162359 ]
irssi docs

[ 20070716114553 ]
qemu eats up /dev/shm

[ 20070710214512 ]
git-clean in svn land

[ 20070705113139 ]
ipw2200 not working

[ 20070627191916 ]
git slides updated

[ 20070618220649 ]
unpopular debian packages on my system

[ 20070611125852 ]
git-svn with multiple branches

[ 20070517085321 ]
Linus on Git at Google

[ 20070510134551 ]
vim modelines insecure

[ 20070504205042 ]
bios disassembler

[ 20070504124124 ]
dd hex arguments

[ 20070503013555 ]
urxvt mouseless url yanking

[ 20070502211941 ]
gitdiff.vba v2

[ 20070419234350 ]
india

[ 20070418155857 ]
zsh fun

[ 20070418143632 ]
pipe to pastey.net

[ 20070418094151 ]
vimgrep alias

[ 20070416202545 ]
mouse-free

[ 20070406141850 ]
ATA messages via SCSI layer

[ 20070330221019 ]
GITDiff vim plugin

[ 20070329011735 ]
git presentation for OCLUG

[ 20070328123631 ]
fixing vim's [[ and ]] for bad code

[ 20070316092236 ]
pxeboot and nfsroot with debian

[ 20070312134706 ]
etc snapshots with git

[ 20070307004041 ]
remote power switch

[ 20070222215355 ]
klips-less openswan git tree

[ 20070221041316 ]
git caching for v1.5.x

[ 20070218002214 ]
klips loses zlib

[ 20070209172606 ]
vim and linux CodingStyle

[ 20070207205427 ]
my kqemu install

[ 20070204100100 ]
leaner meaner openswan

[ 20070115111917 ]
wmii+ruby xlock action

[ 20070112131252 ]
cloning xterms in wmii+ruby

[ 20070102010551 ]
wmii+ruby talk for OCLUG

[ 20061228220641 ]
dump and restore

[ 20061218100219 ]
C style

[ 20061101002027 ]
fetching all git branches from remote

[ 20061028111607 ]
local caching for git repos

[ 20061020145437 ]
automatic version creation with git

[ 20061018213306 ]
wmii w/ ruby wmiirc

[ 20061018201907 ]
small fonts

[ 20061007151802 ]
google-codesearch from vim

[ 20060928020813 ]
shell commands

[ 20060920093957 ]
letting screen apps use the ssh-agent

[ 20060908223613 ]
mpdscribble stream support

[ 20060907125149 ]
glGo on ubuntu/dapper amd64

[ 20060906163240 ]
lbdb and mutt

[ 20060902145643 ]
vim tutorial

[ 20060902135722 ]
fixing your terminal

[ 20060828124713 ]
apt-get pdiffs

[ 20060824224842 ]
256 colour xterms

[ 20060824152658 ]
dynamic IPcomp

[ 20060824145428 ]
inkscape++

[ 20060818150516 ]
tags/cscope for system headers

[ 20060805131557 ]
opteron 170, part 4

[ 20060805101941 ]
opteron 170, part 3

[ 20060803233234 ]
opteron 170, part 2

[ 20060802210126 ]
opteron 170

[ 20060729144129 ]
OLS keysigning / 2006

[ 20060728105500 ]
git-find findings

[ 20060727162941 ]
starting on git-find

[ 20060727113632 ]
git-graft and git-find brainstorm

[ 20060726224531 ]
pretty function tracing

[ 20060713174723 ]
uml and multiple network segments

[ 20060707182236 ]
lastfm artist and title to clipboard

[ 20060706162256 ]
reverting a git changeset

[ 20060629212003 ]
user #3

[ 20060628083456 ]
firefox crashes with form input

[ 20060621151402 ]
git vs hg

[ 20060612222204 ]
ldap account management

[ 20060612194523 ]
stupid ldap

[ 20060608092157 ]
rpm hell is right

[ 20060605095726 ]
OpenSSH VPNs

[ 20060604114317 ]
Lenovo lost a customer

[ 20060601234010 ]
generating html colourized sourcecode

[ 20060601211716 ]
ion3 greatness and acting on X selections

[ 20060526085644 ]
software RAID10 performance

[ 20060525234148 ]
learning to love git

[ 20060524121638 ]
recent vim7 articles

[ 20060516095748 ]
bootstrapping debian on my sbc

[ 20060428145140 ]
entropy injection

[ 20060423140628 ]
Adam

[ 20060414202507 ]
converting mp3s to CD

[ 20060413232836 ]
secure apt-get

[ 20060412194423 ]
xen domain0 on debian

[ 20060410220525 ]
LVM2 on RAID1 mirror

[ 20060410102824 ]
building a RHEL4 kernel w/ kdb support

[ 20060407230939 ]
xen on debian

[ 20060407230818 ]
Upgraded look

[ 20060330131334 ]
Flattered by a copy

[ 20060328165153 ]
vim7 from source

[ 20060308123539 ]
Perl, Catalyst, CPAN, and Debian

[ 20060308123302 ]
last.fm

[ 20060128124841 ]
carcassonne and zombies

[ 20060120135931 ]
Election Humour

[ 20050925130002 ]
ldap on debian

[ 20050528190034 ]
Error while mapping shared library sections

[ 20050124130158 ]
IRC over email gateway

[ 20050110225522 ]
brute force attacks sshd?

[ 20050108095026 ]
LDAP authentication (part 1)

[ 20041124130146 ]
sawfish workspace themes

[ 20041113082651 ]
Mini-DV to divx using mencoder

[ 20041004084525 ]
notes on vserver

[ 20040922104334 ]
fast kernel logging

[ 20040803104122 ]
Digital Rebel for sale... GONE

[ 20040714202912 ]
OpenOffice resources

[ 20040603175746 ]
photo editing

[ 20040601082817 ]
WRT54G

[ 20040503205227 ]
Digital Rebel

[ 20040420200136 ]
Open Office templates

[ 20040326082602 ]
bash vi editing mode

[ 20040315204142 ]
debian install CDs

[ 20040312155542 ]
change of jobs

[ 20040308091554 ]
spamassasin extras

[ 20040305163216 ]
cool debian tools

[ 20040305155708 ]
first post!





Bart Trojanowski
http://www.jukie.net/~bart
bart@jukie.net