git caching for v1.5.x

[ link: git-1.5-caching | tags: scm git | updated: Wed, 21 Feb 2007 04:52:54 ]

I wrote about git caching several months back. The term, git caching, was something I had given a local repository that can be used as a reference for multiple projects. New features in the recently released git 1.5.x requires that I blog again about this great tool.

Recap: I am working on a linux patch -- klips to be specific. I have more repositories then I know what to do with. Git has this cool feature where it can point to another directory to find it's object files, this is called alternate or reference repository.

Git 1.5.0

First, you will need git 1.5.0 or better, as it added some new branching and remote management features. If you need to build it follow these steps:

    $ git clone git://git2.kernel.org/pub/scm/git/git.git
    $ cd git
    $ make configure
    $ ./configure --prefix=/usr
    $ make all doc
    $ sudo make install install-doc

( you may need my patch otherwise you will not be able to have dots in your remote names )

The cache

Now, let's make a new cache for the linux tree:

    $ mkdir -p /site/scm/git/linux.git
    $ cd /site/scm/git/linux.git
    $ git init-db

Now we add some repositories we want to track:

    $ git remote add linux git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    $ git remote add linux-v2.6.15.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.15.y.git
    $ git remote add linux-v2.6.16.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.16.y.git
    $ git remote add linux-v2.6.17.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.17.y.git
    $ git remote add linux-v2.6.18.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.18.y.git
    $ git remote add linux-v2.6.19.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.19.y.git
    $ git remote add linux-v2.6.20.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.20.y.git

So far we have no objects for these repositories. To fetch them all I used to maintain a script that got them one by one. But now we have a single git command to update everything:

    $ git remote update

And it not only gets all remotes, it also gets all branches of all remotes.

Wow!

If you forgot some, you can add them and update again:

    $ git remote add linux-kgdb git://git2.kernel.org/pub/scm/linux/kernel/git/trini/linux-2.6-kgdb-master.git
    $ git remote add linux-kgdb-testing git://git2.kernel.org/pub/scm/linux/kernel/git/trini/linux-2.6-kgdb-testing.git
    $ git remote update

Now you can schedule this at night time using cron so you always have a fresh cache. Note that the intent is not to clone this repository locally. It does not have to be upto date all the time. Once a day is just fine.

Working repositories

Say I want to work on a patch for the kernel. I clone as I would normally, but I add a --reference option and point it at my cache.

    $ git clone --reference /site/scm/git/linux.git \
            git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git \
            linux-work

Now linux-work has the latest from Linus, but it didn't have to download it from the kernel.org server. It just asked the server for the object hashes and fetched all that it could from the reference. Here is what's inside:

    $ git branch -a
    * master
      origin/HEAD
      origin/master

Note that 1.5.x tracks all branches off the remote. When you git fetch origin it will get any new branches that Linus created.

Git 1.5.x also separates local and remote branches. The -a shows all branches. By default all remote branches that are tracked will have the remote's name as a prefix. You can access them with that name:

    $ git log -1 origin/master
    commit c8f71b01a50597e298dc3214a2f2be7b8d31170c
    Author: Linus Torvalds <torvalds@woody.linux-foundation.org>
    Date:   Tue Feb 20 20:32:30 2007 -0800

        Linux 2.6.21-rc1

How references work

Should you need to, you can change the reference to a superset repository (ie, it contains all the objects that your working repo needs) as the location is just a path to another repo:

    $ cat .git/objects/info/alternate
    /site/scm/git/linux.git/.git/objects/

Chaining reference repositories

Ok, here comes a(nother) neat part. You can have an alternate point to a repository that is itself using an alternate.

Why? Well, recall I am working on KLIPS, not on Linux directly. KLIPS has it's own git repository that is based on Linux. Locally my main KLIPS clone references the linux cache, as described above. This KLIPS tree is my cache. My actual work repository references this cache.

Bart's Blog

/git-1.5-caching

About

I am an embedded Linux software developer and consultant operating under Jukie Networks Inc in Ottawa, Canada.

RSS Feed - Full Content

RSS Feed - Full Content

ipv6 ready

Valid XHTML 1.0 Transitional

Created with Vim

Created with Perl




Tags

abiword adam android apt asm ata audio bacula bash bios board-game boot bug busybox bzr c code conference debian debug desktop devel disk dns dpkg dream email embedded find firefox font fs fun g1 gdb geode git git scm git-find git-graft go google gpg grub hardware html hw inkscape ion3 ipsec ipv6 ipv6summit.ca irc irssi joke kdb kernel klips kqemu kvm laptop lastfm ldap lighttpd linux lua lvm mail make mercurial meta mindblown mmc mouse-free mp3 mpd music mutt net nexusone nfs oclug ogre ols oom OOo openswan opteron osx ottawa perl photography power pxe qemu raid redhat rogers ruby sata sbc scm screen script scsi security sheeva shell skype spam squid ssd ssh suse svn tags talk thinkpad ubuntu uml unix urxvt usb uzbl vi video vim vimgit virt vpn vserver web weechat wifi wind wmii wmiirc-lua work x x86 x86emu xen xen-box-setup xfs xterm zsh

Posts

[ 20181212145129 ]
improving find -exec efficiency

[ 20130409092758 ]
automount mmcblk devices

[ 20130316190507 ]
tunnelbroker vs IRC

[ 20110616180255 ]
HDD -> SDD

[ 20110517204617 ]
how to manually create a 6in4 tunnel

[ 20110501202915 ]
Presentation slides published!

[ 20110320190458 ]
What is IPv6, and why should I care?

[ 20110309230106 ]
Ottawa IPv6 Summit 2011

[ 20110309225149 ]
I am now an IPv6 Sage

[ 20101214145558 ]
It's a Holiday Miracle

[ 20101103095130 ]
ipv6 on your desktop in 2 steps

[ 20101020113026 ]
growing a live LVM volume

[ 20100915175649 ]
distributing DNS list through radvd

[ 20100908173402 ]
Canadian ipv6 drought

[ 20100903190623 ]
ipv6 certification

[ 20100723192259 ]
m4a to mp3

[ 20100723134522 ]
git 1.7.2 is out

[ 20100523092001 ]
console=ttyS0 with grub2

[ 20100426162644 ]
How many times is my function used within an executable?

[ 20100412100135 ]
vmlinux on Ubuntu

[ 20100328152951 ]
where your WIND coverage ends

[ 20100326113354 ]
Nexus One live in Ottawa on WIND Mobile

[ 20100311133411 ]
the WIND excitement

[ 20100302160743 ]
sata hotswap pico-HOWTO

[ 20100227093435 ]
serving http content out of a git repo

[ 20100225104049 ]
pimped out zsh prompt

[ 20100221184126 ]
live termcasting of your terminal over telnet

[ 20100215083725 ]
Debian on UBIFS upgrade on SheevaPlug

[ 20100214205532 ]
nexus one

[ 20100130150527 ]
skype on Debian Linux (64bit)

[ 20100125191021 ]
I am so peeved at Rogers

[ 20091205130455 ]
running really nice

[ 20091120225902 ]
notmuch for vim

[ 20091113102221 ]
squid and apt

[ 20091104195344 ]
using WIP branches to save every edit

[ 20091104194146 ]
using git workflows to avoid loosing intermediate changes

[ 20091019183503 ]
reflections on ACM Reflections

[ 20091013174248 ]
bacula rejected Hello command

[ 20091007235046 ]
pimping out git log

[ 20091002094400 ]
virtualization primer

[ 20090922110756 ]
adding an external encrypted volume under Debian

[ 20090905112529 ]
the dreaded process of rooting Rogers Dream

[ 20090830174551 ]
Cyanogen's recipe for Cupcake/Donut-like pastry

[ 20090827100232 ]
simpler android rooting

[ 20090821095427 ]
androids don't like water

[ 20090820215358 ]
prettier function tracing

[ 20090816151854 ]
Baking a cupcake

[ 20090814131559 ]
Hello Android!

[ 20090813100025 ]
sim unlocking a G1

[ 20090731185522 ]
rooting the droid

[ 20090727210307 ]
getting into android

[ 20090714094340 ]
importing an old project into git

[ 20090702113222 ]
Why pick Git?

[ 20090628131255 ]
select loop for X events

[ 20090625181315 ]
portable printf

[ 20090623225800 ]
switching to uzbl

[ 20090622223147 ]
Linux Symposium

[ 20090622214023 ]
bringing git-format-patch to bzr

[ 20090612215638 ]
nfs local caching with fscache and cachefilesd on Lenny

[ 20090610202041 ]
Scott Chacon smacks git around

[ 20090610150039 ]
how would you read a file into an array of lines

[ 20090609215141 ]
libguestfs

[ 20090609000208 ]
tiding up the PATH

[ 20090608232531 ]
only showing relevant messages in mutt by default

[ 20090608010405 ]
git-vim hacking

[ 20090530223801 ]
mark-yank-urls: fix bug allowing shell to interpret the url

[ 20090509212648 ]
wmiirc-lua v0.2.8 release

[ 20090509113125 ]
wmiirc-lua moving to github

[ 20090504101605 ]
two terminals one PWD

[ 20090501172645 ]
splitting files out of a commit

[ 20090409155905 ]
git workflow: git amend

[ 20090401112030 ]
how old are these files in git?

[ 20090330181138 ]
sles 11 on kvm

[ 20090323194942 ]
android true type font

[ 20090322203939 ]
popen with stdin, stdout, and stderr

[ 20090320214228 ]
shrinking URLs

[ 20090304004744 ]
readlater

[ 20081112150409 ]
splitting patches with git

[ 20081011081638 ]
creating busybox symlinks

[ 20081002215121 ]
wmiirc-lua v0.2.5 release

[ 20080916155113 ]
git-svn strangeness

[ 20080915112959 ]
installing git man pages quickly

[ 20080913112345 ]
wmiirc-lua v0.2.4 release

[ 20080825100454 ]
Kernel Walkthroughs - booting

[ 20080813210205 ]
Linux Kernel Booting

[ 20080719211329 ]
printable OLS/2008 schedule

[ 20080715214447 ]
wmiirc-lua updates

[ 20080713194704 ]
Git Screencast

[ 20080713143429 ]
four steps to reproducible Debian installs

[ 20080705150651 ]
USB2.0 enclosure benchmark

[ 20080703230924 ]
Linux Kernel Walkthroughs posted

[ 20080702113602 ]
Introducing the Ottawa Ruby folks to Git

[ 20080628160732 ]
Authenticating Linux against OSX LDAP directory

[ 20080627142123 ]
Canada Day Events 2008

[ 20080613162541 ]
Linux Kernel Walkthroughs

[ 20080510083828 ]
is my usb device connected to a fast port?

[ 20080509111534 ]
show more git info on zsh prompt

[ 20080430104202 ]
git-vim

[ 20080412100337 ]
color your word

[ 20080404105620 ]
show current git branch on zsh prompt (2)

[ 20080303200359 ]
how to track multiple svn branches in git

[ 20080301134220 ]
fixing X for GeodeLX

[ 20080108002540 ]
kvm nfs hang

[ 20080107160836 ]
screen -c relative path bug

[ 20080105132854 ]
WeeChat spell suggestions

[ 20071219221358 ]
show current git branch in zsh

[ 20071217141037 ]
wmiirc-lua kitchen sink repository

[ 20071212100316 ]
protecting sshd from OOM killer

[ 20071204234232 ]
wmiirc-lua v0.2.1 remembers a bit more

[ 20071013205336 ]
wmiirc-lua v0.2 has suspend and raw modes

[ 20070929112345 ]
wmiirc-lua debianization

[ 20070924104140 ]
zsh tip of the day - global aliases

[ 20070915094213 ]
wmiirc-lua v0.1.1

[ 20070913130838 ]
comparing two directories

[ 20070909204125 ]
git-rebase --interactive

[ 20070908115905 ]
zsh tab completion awesomeness

[ 20070902000736 ]
wmiirc in lua

[ 20070831150306 ]
debugging with -dbg libraries

[ 20070831142646 ]
svn status like output in git

[ 20070829141847 ]
Git Cheat Sheet

[ 20070821142038 ]
switching to abiword

[ 20070811105746 ]
forwarding ssh and X through screen

[ 20070807112531 ]
git-svnup

[ 20070724082355 ]
reducing power consumption

[ 20070722123734 ]
Makefile template

[ 20070722002649 ]
less, colourful

[ 20070719162359 ]
irssi docs

[ 20070716114553 ]
qemu eats up /dev/shm

[ 20070710214512 ]
git-clean in svn land

[ 20070705113139 ]
ipw2200 not working

[ 20070627191916 ]
git slides updated

[ 20070618220649 ]
unpopular debian packages on my system

[ 20070611125852 ]
git-svn with multiple branches

[ 20070517085321 ]
Linus on Git at Google

[ 20070510134551 ]
vim modelines insecure

[ 20070504205042 ]
bios disassembler

[ 20070504124124 ]
dd hex arguments

[ 20070503013555 ]
urxvt mouseless url yanking

[ 20070502211941 ]
gitdiff.vba v2

[ 20070419234350 ]
india

[ 20070418155857 ]
zsh fun

[ 20070418143632 ]
pipe to pastey.net

[ 20070418094151 ]
vimgrep alias

[ 20070416202545 ]
mouse-free

[ 20070406141850 ]
ATA messages via SCSI layer

[ 20070330221019 ]
GITDiff vim plugin

[ 20070329011735 ]
git presentation for OCLUG

[ 20070328123631 ]
fixing vim's [[ and ]] for bad code

[ 20070316092236 ]
pxeboot and nfsroot with debian

[ 20070312134706 ]
etc snapshots with git

[ 20070307004041 ]
remote power switch

[ 20070222215355 ]
klips-less openswan git tree

[ 20070221041316 ]
git caching for v1.5.x

[ 20070218002214 ]
klips loses zlib

[ 20070209172606 ]
vim and linux CodingStyle

[ 20070207205427 ]
my kqemu install

[ 20070204100100 ]
leaner meaner openswan

[ 20070115111917 ]
wmii+ruby xlock action

[ 20070112131252 ]
cloning xterms in wmii+ruby

[ 20070102010551 ]
wmii+ruby talk for OCLUG

[ 20061228220641 ]
dump and restore

[ 20061218100219 ]
C style

[ 20061101002027 ]
fetching all git branches from remote

[ 20061028111607 ]
local caching for git repos

[ 20061020145437 ]
automatic version creation with git

[ 20061018213306 ]
wmii w/ ruby wmiirc

[ 20061018201907 ]
small fonts

[ 20061007151802 ]
google-codesearch from vim

[ 20060928020813 ]
shell commands

[ 20060920093957 ]
letting screen apps use the ssh-agent

[ 20060908223613 ]
mpdscribble stream support

[ 20060907125149 ]
glGo on ubuntu/dapper amd64

[ 20060906163240 ]
lbdb and mutt

[ 20060902145643 ]
vim tutorial

[ 20060902135722 ]
fixing your terminal

[ 20060828124713 ]
apt-get pdiffs

[ 20060824224842 ]
256 colour xterms

[ 20060824152658 ]
dynamic IPcomp

[ 20060824145428 ]
inkscape++

[ 20060818150516 ]
tags/cscope for system headers

[ 20060805131557 ]
opteron 170, part 4

[ 20060805101941 ]
opteron 170, part 3

[ 20060803233234 ]
opteron 170, part 2

[ 20060802210126 ]
opteron 170

[ 20060729144129 ]
OLS keysigning / 2006

[ 20060728105500 ]
git-find findings

[ 20060727162941 ]
starting on git-find

[ 20060727113632 ]
git-graft and git-find brainstorm

[ 20060726224531 ]
pretty function tracing

[ 20060713174723 ]
uml and multiple network segments

[ 20060707182236 ]
lastfm artist and title to clipboard

[ 20060706162256 ]
reverting a git changeset

[ 20060629212003 ]
user #3

[ 20060628083456 ]
firefox crashes with form input

[ 20060621151402 ]
git vs hg

[ 20060612222204 ]
ldap account management

[ 20060612194523 ]
stupid ldap

[ 20060608092157 ]
rpm hell is right

[ 20060605095726 ]
OpenSSH VPNs

[ 20060604114317 ]
Lenovo lost a customer

[ 20060601234010 ]
generating html colourized sourcecode

[ 20060601211716 ]
ion3 greatness and acting on X selections

[ 20060526085644 ]
software RAID10 performance

[ 20060525234148 ]
learning to love git

[ 20060524121638 ]
recent vim7 articles

[ 20060516095748 ]
bootstrapping debian on my sbc

[ 20060428145140 ]
entropy injection

[ 20060423140628 ]
Adam

[ 20060414202507 ]
converting mp3s to CD

[ 20060413232836 ]
secure apt-get

[ 20060412194423 ]
xen domain0 on debian

[ 20060410220525 ]
LVM2 on RAID1 mirror

[ 20060410102824 ]
building a RHEL4 kernel w/ kdb support

[ 20060407230939 ]
xen on debian

[ 20060407230818 ]
Upgraded look

[ 20060330131334 ]
Flattered by a copy

[ 20060328165153 ]
vim7 from source

[ 20060308123539 ]
Perl, Catalyst, CPAN, and Debian

[ 20060308123302 ]
last.fm

[ 20060128124841 ]
carcassonne and zombies

[ 20060120135931 ]
Election Humour

[ 20050925130002 ]
ldap on debian

[ 20050528190034 ]
Error while mapping shared library sections

[ 20050124130158 ]
IRC over email gateway

[ 20050110225522 ]
brute force attacks sshd?

[ 20050108095026 ]
LDAP authentication (part 1)

[ 20041124130146 ]
sawfish workspace themes

[ 20041113082651 ]
Mini-DV to divx using mencoder

[ 20041004084525 ]
notes on vserver

[ 20040922104334 ]
fast kernel logging

[ 20040803104122 ]
Digital Rebel for sale... GONE

[ 20040714202912 ]
OpenOffice resources

[ 20040603175746 ]
photo editing

[ 20040601082817 ]
WRT54G

[ 20040503205227 ]
Digital Rebel

[ 20040420200136 ]
Open Office templates

[ 20040326082602 ]
bash vi editing mode

[ 20040315204142 ]
debian install CDs

[ 20040312155542 ]
change of jobs

[ 20040308091554 ]
spamassasin extras

[ 20040305163216 ]
cool debian tools

[ 20040305155708 ]
first post!





Bart Trojanowski
http://www.jukie.net/~bart
bart@jukie.net