AlBlue’s Blog

Oh dear oh dear O2

2012-01-25T21:51:00.001Z

Today hasn't been a good day, all in all. No sooner than I had posted on the depressing state that is Google- than O2 disclose that they are sending my mobile number through to any website that I care to navigate to via my iPhone's data connection.

The fact that it was discovered today led to a veritable twitstorm on the O2 twitter feed, and the eventual blog post seemed fairly contrite as to the reasons behind it. They put it to a configuration error which accidentally led to the phone number being sent to more sites than they had intended.

This doesn't excuse the fact that there is no reason for O2 to be sending my phone number to anyone, whether they intended it or not. I don't see why my phone number is necessary – as they put it – to verify my age for certain sites. And it's not like such a header couldn't trivially be faked, nor of a phone number being any kind of guarantee of an age in any case (unless they're also sending a lot more data to those allegedly trusted providers as well).

They claim that this is "standard practice" in the industry, which makes you wonder how many other internet data providers are doing much the same thing, except they haven't cocked it up as badly as O2 have yet. It was also instructive to find out that O2 are mangling data that goes through their network; for example, this blog post highlights how O2 are in-lining style information causing semantic errors in the page itself. You can determine it for yourself by going to http://mnementh.csi.cam.ac.uk/atimport/ – if you see a red background with "Test" on it then it works as expected; however, if you browse it from an O2 equipped data phone then you'll see they have in-lined the referred CSS then it will read "If this text renders in a browser then something is very broken"

Perhaps the best thing to come out of this is the highlight on the practices that O2 are doing, and encourage the use of a VPN connection to a server which is not going to cause any further problems. O2, it seems, cannot be trusted with my business and therefore I will have to take it into my own hands.

You get what you pay for

2012-01-25T09:53:00.001Z

There has been a disturbing trend in the force recently, of companies and their applications taking a significant turn for the worse without proper design and consideration. First, Google pulled a GMail and decided that White was the new Black, and in order to not spoil an otherwise minimalist white canvas threw in a few lines of text, before the screw up squad took out Reader and Blogger too.

Last weeks design-for-idiots trend was Twitter moving awya from the highly acclaimed (and successful) AteBits Twitter neé Tweetie application, onto a design which “Optimises for consistency” across different websites and devices.

Yes, that's right – they have managed to snatch defeat from the jaws of victory. Instead of a well designed and fluid iOS interface, it's now the same piece of junk you see on an Android phone.

Let's put aside for the moment flamewars on which device is best, and focus on the reality; they are different devices, with different styles and behaviours. Tweetie even introduced the concept of pull-to-refresh, which became such a signature gesture that it's now used on many iOS based applications. The thing is, the toolbar was optimised for the things the user does frequently; like tweets, direct messages, and the standard ^… icon used to denote more options available.

Now, we have an application whose logo sits in the top ~10% of the screen and is immobile. That's right, it stays there like a wart on the bottom of humanity, refusing to budge. What good does it do there? You know you are using the official Twitter app, because it now looks like cack, and it looks like cack on other devices as well. There's consistency for you.

To seal the deal, the new “Discover” tab has affectionately been renamed the dickbar tab in homage to last time.

Fortunately, there are alternatives. Whilst Twitter is attempting to generate a revenue stream from waving its dickbar in everyone's face, other clients have quietly but politely focussed on the right kind of in-your-face design. Tweetie may be no more, but Tweetbot is on sale right now for $1, and has some nice user interface aspects to it; or rather, it doesn't have a dickbar. Other clients also exist but they're all better than the current Twitter app.

However, the point I wanted to make was that in this age of the internet, you get what you pay for. Google is going out of its way to push Plus in front of everyone's face (PLUS!) whether they like it or not (PLUS!). They've also made it glaringly obvious (PLUS!) just so that your eye is drawn away from the main purpose (the article) and towards the share button (which, let's face it, is not what you're there for).

The problem with Google is that, as the applications are hosted in the cloud, Google has the final call over which version of the software I am using. This is good in many ways – any security updates or upgrades will happen automatically, without my involvement.

The downside is that Google can also remove functionality which I use regularly, such as the ability to read posts without a glaring PLUS on the screen in the low-key icons bar.

However, the recent turn of Google ramming Google+ down everyone's throats is the real concern. Thanks to a upcoming policy change the ever present Google will tie all of your accounts and services together, and allow you to leak data. (Remember the whole buzz fiasco a few years back, when they introduced the cheesy like symbol and then subscribed all of your private messaging contacts to your publicly visible profile?) Only not content with world social domination, they are now tying in all the videos you have ever watched on YouTube with documents you have written on Docs to blog posts you have commented on Reader.

The new policy comes into effect on the 1^st of March, and by that time, my intention is to wean myself off any Google services that I may be using. I've already stopped using their search – after all, Focus on the User shows that Google results on their own are no longer to be trusted – and DuckDuckGo is my go-to search engine (remember when Google just did search? Like that.)

The other key service I need to replace is Reader. It has – until the UIdiots moved in – been the best news feed aggregator on the web, mainly because it allowed me to keep a track of what I'd read from across multiple computers. However, these days I mostly consume it from the iPhone (it's the only one they've not let the UIdiots near yet) so being able to track from multiple devices is no longer the need it once was.

Anyway, as the title suggests, you get what you pay for. And, by a happy coincidence, Google doesn't pay me to use their services. So Google – thanks for the memories. We had a great time together. But things have changed, and we've drifted apart. I still respect some of the good things you're trying to do for the web, and as an advertising behemoth you are not likely to falter any time soon (unless Apple finds a bit of loose change behind the sofa and buys you out). But I can no longer continue to support you by providing you with all my data whilst you data-rape me and splash it across externally visible profiles. Instead of gaining a social network participant, you've lost a customer. And you know what? If ever I need to find something again – I'll just ask my friends.

Git Tip of the Week: Finale

2011-12-20T09:00:00.000Z

Over the past nine months, I've been writing a series called the Git Tip of the Week series, where I write about a Git-related article every week. Part of this has been a desire to learn more about the way the Git internals works; part is to provide a reference for others to find out about as well.

However, all good things must come to an end, and writing a weekly post, whilst keeping it fresh, is a non-trivial task. In addition, finding something new (or even vaguely interesting) to write about is increasingly difficult once you've covered the standard cases and a number of esoteric ones.

So, for my final post in the series, rather than writing about something new, I thought I'd link back to the ones that I've written before, in order. Thus, if you want to share the series with others, you can refer back to this index page as a means of finding out what I wrote when. The search list is all well and good, but it only shows the top 20 most recent posts. In addition, I've added some of my other Git related articles that may be of interest, even though they weren't part of the Git Tip of the Week series.

Prior posts

March 2011

April 2011
- Aliases
- Branches
- Merging
- Tags
May 2011
- Gollum
- Stashes
- Reflogs
- Git Revisions
June 2011
July 2011
August 2011
September 2011
October 2011
November 2011
December 2011

I'd like to thank you for your time and interest in reading this series, and wish you a happy Christmas and a prosperous New Year.

Git Tip of the Week: Forking and Pulling vs Pushing

2011-12-13T09:00:00.000Z

This week's Git Tip of the Week is about the GitHub generation. You can subscribe to the feed if you want to receive new instalments automatically.

Distributed Version Control Systems have really taken off in the last few years, though they've been around for over a decade. Probably the biggest growth spurt happened because of the controversy that launched Git back in April 2005, providing a rock solid distributed version control system, modelled on a filesystem. In only a few months, Git began hosting the 2.6.12 Linux kernel source.

However, as popular as Git may have been then, it wasn't until the birth of GitHub until Git really took off. Founded in February 2008, GitHub brought Git to a much wider audience and provided a free hosting site for public Git repositories (as well as commercial plans for private repositories.) It has been argued that GitHub is one of the reasons why Git has taken off faster than others, like Hg and Bzr.

What GitHub brought was a focus on a new model; instead of creating patches (as covered last time), GitHub encouraged universal forking of the repository. So, if you want to add a change to an existing repository, you can fork it (and create your own clone), make the changes, and then send a pull request.

There's nothing significant about pull requests in the Git workflow as compared with other DVCS tools. Pushing and pulling are two key primitives in a DVCS workflow, after all. But what was novel about GitHub's approach was the way that pull requests could be sent, as an out-of-band message to the upstream repository owner suggesting the idea.

Not only that, but the upstream owner would then get a notification and be able to view the request in situ, and with the diffs as appropriate (outside of a mail client, via the web interface). Subsequent advances, such as the ability to fork-to-fix-typos, meant that anyone could suggest changes via the web without even needing to compile the code locally.

Pushing, Pulling, Patching or Proposing

As a result, Git repositories can end up with different workflows depending on the type of project and hosting environment you are using. They are:

Pushing: You have access to directly write into the repository, so you just push your changes
Pulling: Someone has changes locally and asks you to pull the change from their repository
Patching: You send the diffs/patches by a transport mechanism (bugzilla, email) for consideration
Proposing: You use a tool like Gerrit to propose changes to which can then subsequently be merged

Each project has potentially a different style of operation, and there isn't a "right" way to use a Git repository. GitHub, for example, strongly favours the Pulling model when consuming changes from others (though of course, the repository owner can do pushing directly). The Linux Kernel, both for historic reasons and also for transparency and open discussions, chooses the patching (by e-mail) model.

The final one – proposing – is a combination of both the pull, push and patch models. They're similar to GitHub's pull mechanism, in that the project's owners can see a list of all incoming changes and decide which ones to use; but the push-based upload means that the original repository doesn't have to be forked on the remote server. And finally, tools like Gerrit (which I've mentioned before) can be used to generate patches, host in-situ discussions, and even act as a Git repository for consuming by standard git fetch protocols.

GitHub's pull-based approach has certainly had a wide impact on the number of users willing to try that method. They have a note on collaborative development models on the subject:

The Fork + Pull Model lets anyone fork an existing repository and push changes to their personal fork. This model reduces the amount of friction for new contributors and is popular with open source projects because it allows people to work independently without upfront coordination.

The Shared Repository Model is more prevalent with small teams and organizations collaborating on private projects. Everyone is granted push access to a single shared repository and topic branches are used to isolate changes.

Certainly, if there are minor changes (like a typo in documentation) the fork-and-pull model, when combined with a web-based interface, can make things dramatically easier for contributors. Instead of having to need to create accounts on bug tracking systems (or tools like Gerrit), the repository can be forked, fixed, and a pull request fired off to the repository maintainers. With the merge button in GitHub, it can often be the case of allowing the fix to be merged in without the maintainer having to check the code out at all, if it's sufficiently simple. Reducing the barrier to accepting changes helps keep an active open-source project alive and open to all.

The only problem with the Fork + Pull model is being able to attribute changes by user. For example, some open-source foundations want to ensure that any changes are granted against an existing open source license (Apache or Eclipse, for example). Other projects tend not to be as strict and will happily accept contributions from anyone, with the assumption that any contributors have agreed to the license. One additional service that patches-to-bugzilla or gerrit push provide is in the acceptance of a contributor agreement, which normally states that the individual is entitled to grant the code under the specific licence. One of the side-effects of creating an account often implies (explicitly or implicitly) the agreement to follow that foundation's licensing rules.

So, there's no "right" way to do Git; different teams, foundations and projects will have their own preference for working with a particular strategy, and may evolve over time. Instead, it's useful to know what's available so that the right choice for that project can be made, understanding the different flows available.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Patches by Email

2011-12-06T09:00:00.000Z

This week's Git Tip of the Week is about how git handles patches by email. You can subscribe to the feed if you want to receive new instalments automatically.

One of the main benefits of a distributed version control system is that code changes can be pushed from one repository to another clone, and all dependent changes are pushed as well. However, that only works when you have write access to the remote repository, which in many cases you do not. One way of getting changes is by providing a patch, or a set of changes which can be applied to a remote repository at the other end.

Git started life as a distributed version control system for the Linux project, which actively uses mail lists both as a discussion mechanism and also as a distribution mechanism for patches (changes) for an existing codebase. (New features are just a special case of patching nothing to add the new code.)

To speed the processing of patches by mail, git developed tight integration with both (command-line) mail clients and of the generic Unix mbox format. Patches can be generated in the form of mail messages, and the remote end can process them with a specific command to reconstitute the changes in the git repository.

Whilst the majority of projects don't use patches by mail as a change distribution mechanism, it is useful on occasion where either a patch needs to be generated and attached to a bug tracking system, or where changes need to be sent to a remote developer who doesn't have direct access (such as through a firewall).

The convention adopted by the git developers is to format one patch per e-mail message. The subject of the message then has the first line of the git commit, prefixed with a prefix that can be overridden on the command line but which defaults to [PATCH x/y] as a means of threading them together. (Amongst other reasons, this is why the initial line of a Git commit message is suggested to be relatively short, so that it fits with a mail client's view of the subject and suggested prefix.)

Generating and sending patches

How do we generate these patches? The git format-patch will generate a patch-file-per-commit in the range required, formatted ready to go as mail messages in mbox format. The --to can be specified for which mail address the patches should be sent to – but the sending is done separately.


(master) $ git format-patch --to cdt-dev@eclipse.org HEAD~..HEAD
0001-bug-333001-Description-Scanner-Info-doesn-t-release.patch
From 9c9c692df50e5a9eb91b41cc86f57212afd78ef9 Mon Sep 17 00:00:00 2001
From: Andrew Gvozdev …
Date: Sat, 16 Jul 2011 15:16:21 -0400
Subject: [PATCH] bug 333001: Description Scanner Info doesn't release
 ICProjectDescription
To: cdt-dev@eclipse.org

---
 .../cdt/internal/core/model/CModelManager.java     |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
⋮

If the commit message has more detail than a single line, the detail will be included below the mail's subject headers. It's possible to add additional commentary below the commit message, before the patch is shown, and any text up until a combination of --- or >8 or 8< (AKA 'scissor lines') is ignored by the patch application at the other end.

These patch files can then be transmitted via mail using the git send-email command. This connects to the given SMTP server (either the one from your global ~/.gitconfig or the project's .gitconfig, or the one specified on the command line) and then sends each patch file as a separate e-mail:


(master) $ git send-email --smtp-server=smtp.gmail.com *.patch

As well as using format-patch in a separate stage, it's possible to use send-email to generate the patches and then send them immediately. (You can also configure send-mail to prompt to open an editor so that you can customise the messages before they are sent.)

Applying patches

Once the patches have been created, how do you apply them into a local clone? If you have a patch file, you can apply it with git apply:


(master) $ git apply 0001-bug-333001-Description-Scanner-Info-doesn-t-release.patch

Note, however, that this approach does not recreate the state of the world as it was on the sender's repository. Instead, the patch is applied but it only makes local changes to the repository's content instead; it does not recreate the commit (and more specifically, the hash of that commit). You can specify git apply --index and git apply --cached to get the changes put into the staging area, but this does not recreate the same commit as before.

To recreate the commit as it was exactly requires the use of git am, which stands for apply mailbox. This runs through a mailbox (which may have one or more patches in it) and recreates a commit for each one of those patches.

Fortunately, the output generated by git patch is already in mbox format; it's the purpose of the otherwise dummy From 9c9c692df50e5a9eb91b41cc86f57212afd78ef9 Mon Sep 17 00:00:00 2001 line at the top of the patch file. As a result, the patches can be treated as one message per mbox, and then applied in batch to the changes which get sent.

In fact, since mbox elements can be concatenated together, this permits patch files to be concatenated together to form a larger patch file, which can be sent as a single unit via another transfer mechanism and then applied on the remote side.

Bundles

Patches provide a way of reconstituting a repository over a not directly connected mechanism, but the purpose of patches are to enable humans to investigate the set of changes as much as getting the change there. If however the desire is to move commits from one machine to another without direct connectivity, a better alternative is to use git bundle.


(master) $ git bundle create changes.bundle HEAD~..HEAD
Counting objects: 23, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (12/12), 935 bytes, done.
Total 12 (delta 5), reused 6 (delta 0)

The format of the bundle uses the same format as the network transmission that Git uses over the network when cloning. As a result, the references contained are only those listed in the reference list.

Typically, a tag will be used to mark where the last known point was for the remote source; then, the difference between HEAD and that tag is used to build up the bundle for the remote end. Alternatively, branches can be used to simulate the branch on the remote end.

Once the bundle file has been generated, it can be sent over any transport to the remote host for reconstitution. This might involve burning to a CD, via a USB stick or some other network protocol.

On the client side, the client can run git verify to determine if all required parent commits are present in the local repository. This must be run from the client git repository that you want to fetch into.

The client views the bundle as a remote that it can pull from, much like a path to a directory can be used to pull from a local file-based repository. You can add it as a remote (e.g. git remote add changes /tmp/changes.bundle) or you can fetch from the path to the bundle itself:


(master) $ git verify /tmp/changes.bundle
The bundle contains 1 ref
b707c559636bf8e6dffb3145bd44b03de18868b3 HEAD
The bundle requires these 1 ref
3580c1087c2860fbe6ca4c1a7a6d6e1eb1669aa3 Bug 333599 - [C++0x] Initializer lists & return without type
/tmp/changes.bundle is okay
(master) git fetch /tmp/changes.bundle
From /tmp/foo.bundle
 * branch            HEAD       -> FETCH_HEAD

Once the references have been fetched into the repository (which can be referred to as FETCH_HEAD) you can then inspect the changes, fetch/merge them into the local branches or reset your master branch to that of FETCH_HEAD.

Summary

It's not always possible to have write access to the repository you want to send changes to. In those cases you can send changes out of band, either via mail (if you want human reviews) or as a bundle (if you just want to send the commits).

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Git Submodules

2011-11-29T09:00:00.000Z

This week's Git Tip of the Week is about git submodules. You can subscribe to the feed if you want to receive new instalments automatically.

In the previous post, I wrote about a tounge-in-cheek extension called git bigjobbies; the purposes of which was to store large binaries in a repository without them being part of the main branch (and thus, not appearing in the history).

The reason one might want to do that is to avoid a checkout from the server taking a significant period of time, or taking up additional space once the clone has been made. Although Git provides a good delta encoding for existing files, these tend to only work if the binary data is relatively similar. Typically, large binary assets (such as audio files, movies or even images if they've been saved in a compressed format) share little of the same binary data under the covers.

Git also has a configuration variable, core.bigFileThreshold, which can be used to set the limit at which files are stored as-is without performing any delta comparisons. Files above 512Mb (by default) are stored without any delta compressions to previous versions (though they are deflated at storage time).

The obvious solution to this problem is to store source code (and other compressible assets) in one Git repository, and then store large media assets (sound effects, in-movie videos etc.) in another Git repository. The history of one will therefore not affect history of another.

Submodules

If you're storing these as separate git repositories, how do you ensure that they are kept in sync with each other? Well, you could use tags and rely on convention to ensure that you can acquire the same version of the assets. However, tags can change (although they're not supposed to) and conventions can be circumvented.

Another way to do it is to store a pointer to the assets. (This is similar to the .bigjobbies file suggested before.) Since they are referenced by hash, as long as you can acquire the hash then you will be able to restore the asset.

Git submodules works these two concepts together, by treating a submodule as a logically checked out directory in another repository, but referring it to it by a pointer rather than a full checkout. The submodule (sub repository) can evolve at its own pace, with its own checkouts, and the parent can refer to it by a fixed hash.

Working with submodules

To add a submodule to an existing project, run git submodule add to define a local directory corresponding to the remote Git project's contents. For example, if you wanted to add the BigJobbies project earlier as a submodule, you could do:


$ git init parent
Initialized empty Git repository in parent/.git/
$ cd parent
(master) $ git submodule add http://github.com/alblue/BigJobbies/
Cloning into BigJobbies...
done.
(master) $ ls -AF
.git/		.gitmodules	BigJobbies/
(master) $ cat .gitmodules 
[submodule "BigJobbies"]
	path = BigJobbies
	url = http://github.com/alblue/BigJobbies/
(master) $ git status
# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached ..." to unstage)
#
#	new file:   .gitmodules
#	new file:   BigJobbies

Note that this has set up a .gitmodules file and created a BigJobbies directory, corresponding to the BigJobbies cloned data. However, in the git status, it shows up as a file. What's up with that?

If we add the contents, commit, and then look at the tree, we'll get our answer:


(master) $ git commit -m "Added BigJobbies submodule"
[master (root-commit) f34f140] Added BigJobbies submodule
 2 files changed, 4 insertions(+), 0 deletions(-)
 create mode 100644 .gitmodules
 create mode 160000 BigJobbies
(master) $ git ls-tree HEAD
100644 blob 8041b87daf8e7ed034c669c6c5af9d63367dcd78	.gitmodules
160000 commit e9ed329101157ce9be5dc1c2639096bd82d3fa05	BigJobbies
(master) $ (cd BigJobbies; git rev-parse HEAD)
e9ed329101157ce9be5dc1c2639096bd82d3fa05

Instead of a simple mode 100644, which is used for storing a file with rw-r--r-- permissions, 160000 is used instead. This points to a commit, unlike the tree-or-blob that we've seen before. The commit points to the current version of HEAD in the checked out submodule, which as can be seen here is e9ed329101157ce9be5dc1c2639096bd82d3fa05.

The parent repository is now pretty slim; it contains the .gitmodules file and nothing else. However, it is also versioned in lock-step with the BigJobbies repository. Anyone who wants to clone this repository will find they can resolve the repository, albeit with a separate step:


(master) $ cd ..
$ git clone parent clone
Cloning into clone...
done.
$ cd clone
(master) $ ls BigJobbies/
(master) $ git submodule sync
Synchronizing submodule url for 'BigJobbies'
(master) $ ls BigJobbies/
(master) $ git submodule update
Cloning into BigJobbies...
done.
Submodule path 'BigJobbies': checked out 'e9ed329101157ce9be5dc1c2639096bd82d3fa05'
(master) $ ls BigJobbies/
LICENSE.txt	Movies		README.md	git-bigjobbies

In other words, we can clone the parent without acquiring any of its children. However, to populate the child submodules, we need to run a git submodule update command, which brings in the new code. (You also need to run the update when the remote repository has changed contents which you want to acquire as well.)

Parent-child relationships

Sometimes you want to be able to couple two repositories together, such as a game development project with its media assets, or a set of binary releases with a source project. It's tempting to think of these relationships as the binaries being part of the source project (or a submodule), or the media assets as part of the game source (or a submodule).

However, it's often better to reverse the dependency links between these sorts of repository dependencies. In other words, instead of a having a source repository with a child submodule of the binary assets, have a binary assets repository with a submodule of the source.

Flipping the relationship in this way allows you to treat the source repository as a standalone unit, which doesn't need references to the large binaries, but permits a full checkout of the parent repository (which does have the binaries).

For projects where the source has no need for the binaries (like in the precompiled packages for open-source projects) this distinction can save references to upstream binary repositories which may get accidentally checked out (especially if other submodules are used).

It's also possible to put the source and the binaries in two completely independent repositories, then knit them together with a higher level git repository (with two submodules). The parent can then be used as a top-level 'release' repository, whilst still allowing the binaries and source code to be acquired independently.

Finally, one advantage of having the binary (larger) repository being the parent, is that it will still work if you clone it with git clone --depth 1. When you use the --depth 1 flag, you're essentially saying that you don't want any of the history, just the latest commit on that branch. The latest commit will have a pointer to the source code's branch (which will have the full history) and so this permits you to check out a single (latest) version of the binary with access to the full source code's history.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Git BigJobbies

2011-11-22T09:00:00.000Z

This week's Git Tip of the Week is about git bigjobbies. You can subscribe to the feed if you want to receive new instalments automatically.

This tip covers two aspects; firstly, a means to show how Git can easily be extended, and secondly, a means to show that Mercurial's large file support can be implemented relatively easily on top of Git's object store. It should be noted that git bigjobbies is not intended for production use, but as a learning experiment.

Recap of object stores

The git database stores a set of objects by hash. These hashed objects may point to one of objects, trees or commits. Ultimately, a branch (or tag) in Git is just a pointer to a commit, which points to prior commits and a tree; trees points to a recursive graph of trees and blobs.

As a result, you can stick anything you want into a Git repository, provided it's inserted into the hashed object database. In addition, when you clone/fetch/pull from a Git repository, you don't necessarily get everything that the repository contains; you instead get all the reachable commits (and thus transitively, reachable trees and blobs) for the ones you don't have yet. (In the case of a clone, the set of things you have is the empty set which makes the calculation trivial.)

However, you don't get the objects that aren't reachable when you clone. So, failed experiments that didn't work, suggested changes that were not accepted in a Gerrit workflow (or reworked to provide a different implementation), or just branches or offshoots that you're not interested in, are not downloaded when you clone a repository. (Commits which are directly ancestral are of course brought down; only the divergent parts are not downloaded.)

Unreachable objects are ultimately pruned by the garbage collector. Working from known list of roots (e.g. tags, branches) the git gc can work out what objects are no longer reachable from any reference, and ultimately prune them from the record.

We can use the object database to our advantage, to store out-of-band object data in a repository which is not reachable from the branch, but is still referenced in refs and thus resolvable from the centralised decentralised version control system. Enter:

Git Bigjobbies

Git Bigjobbies is an extension I created to demonstrate out-of-band objects being stored in a Git repository. Note that this is neither supported nor recommended. With that out of the way, what does it do and how does it work?


(master) $ touch empty
(master) $ git bigjobbies add empty
(master) $ git status
# On branch master
# Untracked files:
#   (use "git add <file>…" to include in what will be committed)
#
#	.bigjobbies
#	.gitignore
nothing added to commit but untracked files present (use "git add" to track)
(master) $ cat .bigjobbies
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 empty
(master) $ cat .gitignore
empty

The extension writes the object into the database with git hash-object -w, then concatenates it to a file .bigjobbies. Although the object isn't in the tree or referenced by a commit, it still exists in the database. As a result, we can resolve the contents using the hash and into the filesystem, which we record in the .bigjobbies file. Provided this is committed into the branch, we can resolve the file using the hash alone.

But how do we prevent the object being garbage collected when it's not available? Through the general refs/ directory. If an object is referenced from a refs/ file, it will be seen as in use and therefore not garbage collected.

To write a ref, we just need to echo the hash out to a file in the refs directory. It doesn't matter what it's called – so for simplicity we just write out the hash value as the name. To separate it from ordinary git tags and branches, we use refs/bigjobbies/e59de..391 as the name.

Now, when we resolve the objects, we get the contents from the hash in the local store (if it exists); and if not, we resolve via the origin refs/bigjobbies/369de..391 remote reference. As with the Mercurial largefiles extension, it doesn't download the contents of the files unless they're needed; but on the downside, it does need the files to be downloaded ahead of time in order to work off-line. Let's look at how it would work in a clone:


$ cd /tmp
$ git clone /tmp/example other
Cloning into other...
done.
$ cd other
(master) $ ls -a1
(master) $ ls -a1
.
..
.bigjobbies
.git
.gitignore
(master) $ cat .git/refs/bigjobbies/e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
cat: .git/refs/bigjobbies/e69de29bb2d1d6434b8b29ae775ad8c2e48c5391: No such file or directory
(master) $ git bigjobbies resolve
(master) $ ls -a1
.
..
.bigjobbies
.git
.gitignore
empty
(master) $ cat .git/refs/bigjobbies/e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391

The resolve command has dynamically brought in the reference from the remote server and resolved the contents of the file in the local repository. Furthermore, any large files in interim commits will not be resolved, unless they too are mentioned in the .bigjobbies file.

Summary

The point of this was to demonstrate how easy it is for a git extension to be made. All you need do is put the executable with git-bigjobbies as the prefix, and you can run it with git bigjobbies.

In addition, it's a good exercise in understanding how the Git repository works. References are just pointers to hashes, and objects can be stored and referenced by those same hashes. From this, the entire Git tool suite is written; a combination of C and other scripting languages (for example, git-svn is largely written in Perl, and GitHub operates mostly out of Ruby).

You can clone the BigJobbies.git repository from the GitHub repository at http://github.com/alblue/BigJobbies/. The repository that you clone already has some BigJobbies in them; if you do a git bigjobbies resolve at any of the points which have a .bigjobbies file, you will find them downloaded. (Note that an implementation bug relies on the remote being called origin, in case you do git clone -o other.)

Come back next week for another instalment in the Git Tip of the Week series.

Declarative Services vs Objects

2011-11-16T20:00:00.000Z

I'm coming to the conclusion there is an opportunity for improvement of the Declarative Services specification. Right now, DS needs to be able to instantiate a class and then mutate it as services come and go, instead of creating and disposing of classes on demand.

This poses somewhat of a problem if you want to publish objects in the OSGi service registry of types which you don't control. According to the Declarative Services specification, you need to have an object which has a default constructor. The object is then fully configured with a number of bind and unbind methods, which means the object goes through a series of states where it is not fully valid.

There's also the category of classes which aren't under your control (e.g. an existing API) or are acquired elsewhere (e.g. a third party library). These often can't be extended or otherwise mutated.

Example

Let's say we want to have a list of URLs stored as services in the registry. We'll be publishing URL objects as the service (and using java.net.URL as the interface type as well; it is a contrived example, after all). Now, the URL doesn't have a default constructor; every URL must have a url. So we can use it for registering a component which needs a service (e.g. BookmarkService) quite happily. But we can't register a URL with DS.

This presents us with a challenge. We might have a component which can register URL services, but given a 1..n BookmarkService and no URLs published, Declarative Services can't help us. In essence, there's no way to have a delayed URL service in DS.

What you can do is provide a URLFactoryService with DS. This can be activated on demand, and a registerURLs() method called which in turn registers a bunch of URL objects in the registry; and once that's done, DS can kick off the BookmarkService.

Unfortunately, there's no way of being able to tell DS that the URLFactoryService is something which is capable of generating URL objects, so DS never knows it needs to start the URLFactoryService to acquire the URLs.

Solution

The solutions are either to not use DS to populate the initial set of URLs (and require an earlyStart or startlevel hack to bring the initial bundle on-line) or to modify DS in a way which allows the implementation class to be a factory for the interface type but without being a subtype of it. The existing factory specification currently requires that the factory class is still an instance of the interface class; all that changes is the cardinality.

Here's what the solution might look like:


<scr:component xmlns:scr="http://www.osgi.org/xmlns/scr/v1.1.0" name="URLdemo">
   <implementation class="URLService"/>
   <service>
      <provide interface="java.net.URL"/>
   </service>
</scr:component>

In this case, the URLFactoryService doesn't implement/extend java.net.URL. But this tells DS that when this component is activated, it gets a URL out of it, which is what we want DS to know.

We could use a new API, like ServiceOnDemand, which is a generified type that can provide us with a service when we need it:


public interface ServiceOnDemand<T> {
  public T getService();
}

If the URLFactoryService implemented the ServiceOnDemand<URL> interface, then a call to getService would return us with the URL object that could subsequently be registered on our behalf into the service registry. We can still use the bind and unbind calls as before; they just affect the ServiceOnDemand dataset. After each change of the properties (and providing the component was valid) the getService() could be treated as a factory for these object types, being called only when it is in a fully configured state.

For tear down, it becomes just as easy. Instead of removing the URLFactoryService from the registry, we can just remove its previously generated URL objects. The new DS would have to co-ordinate to know which services were registered and associated with which factory; but this shouldn't be significantly different to the way the factory works at the moment.

We could also use this in conjunction with the DS factory class. The key difference here is that the factory class must implement the interface provided, which isn't necessarily always possible.

Summary

DS, and in particular its lazy activation of components, is a great way of providing systems which evolve into being. Unfortunately, the current system is constrained to allowing only objects to be registered to which the user has full control. Although the above used URL as an example, the same could also be said for network-discovered services such as over LDAP or DNS service records, or even database connections (which typically can't be subclassed since they are binary only drivers). Finally, this could be the missing link between config admin and declarative services, which have never really played well together. Being able to use DS to fire up a factory, which consumes its configuration and then is able to generate multiple services would allow us all to avoid the pain and effort that is otherwise trying to control start ordering in order to provide services dynamically.

What else can Google screw up?

2011-11-15T23:55:00.001Z

As if screwing up reader wasn't enough, they appear to have moved the screw-up-squad over to Blogger.

For years (i.e. since Blogger started), blog posts have had a title and a post. Fairly easy to distinguish; one shows up as the title of the page, and one shows up as the body. Feeds recognise this, the page renders this – it's pretty much a given.

However, thanks to the Google+Doom operation, the Blogger codebase now appears to confuse a blog post with a giant + post. So instead of using the title given to derive the URL (e.g. http://alblue.bandlem.com/2011/11/git-tip-of-week-git-notes.html), it now uses the first few sentences of the posts. Given that all of my Git Tip of the Week posts start with "This week's Git Tip of the Week is about" it now turns out that any post I create (this month) is called http://alblue.bandlem.com/2011/11/this-weeks-git-tip-of-week-is-about-git.html. Fricking useless.

Fortunately, you can work around this – as this post does – by making a dummy post with "What else can Google screw up?" as the first paragraph. Post it to Blogger, and it generates what the old URL looked like; you can then edit the post and remove the dummy sentence from the post.

Git Tip of the Week: GC and Pruning

2011-11-15T09:00:00.001Z

This week's Git Tip of the Week is about git gc. You can subscribe to the feed if you want to receive new instalments automatically.

Paul Webster recently wrote Where the git did that go? in relation to the incredible disappearing commits. Some said that a version control system shouldn't be able to do this; but it's actually all part of Git's functionality. Let's look at what happened.

A git repository stores commits in a transitive closure (a real closure, not a lambda) from the reachable items through to every commit (and every tree and blob). It's not possible to remove a commit – and therefore the trees and blobs that make up that repository.

So, how is it possible to lose data with Git? Well, if you are using a standard Git repository, you can create branches with git branch; and delete them with git branch -d. When you delete a branch, you remove the pointer to the last commit – but you don't actually lose the commits.

In addition, the git reflog, which we covered previously, stores a list of the previous branch pointers. In other words, even if you delete a branch, the reflog has got your back.

Generally speaking however, only repositories with working directories have reflogs; bare repositories tend not to. There is a config option, git config core.logAllRefUpdates, which can be used to force it on all repositories – or disable it completely if it's not needed.

Even without a reflog, the commits aren't removed immediately. If you run a git gc, which repacks the repository into a more efficient structure, it will export non-referenced commits as loose objects. (You have to ensure that there aren't any branches or tags or reflogs to see this behaviour; if there's an existing pointer then it will not evict the object from the packfile.)

Running a git fsck will check that all objects are present as expected. You can also see what is no longer referenced; running git fsck --unreachable will show you which commits are no longer reachable due to deleted branches or removed tags. Running git fsck --unreachable daily and mailing reports will give a good early warning of commits about to disappear if it's a concern.

Objects which are no longer referenced can be evicted with git prune; though this is a low-level operation which is often called from git gc. By default it will not remove commits newer than 2 weeks old, and of course the commits that are reachable from that; so provided the branch (or tag) deleted has recent commits, it will stay around in the git repository for up two a fortnight afterwards.

Avoiding future issues

Both branches and tags can be deleted; and when invoking a remote push operation a missing branch (or tag) on the client side can invoke a delete; for example: git push github :refs/heads/master will delete the 'master' branch off the remote repository known as github. If this is in a script, such as git push github $COMIT:refs/heads/master and the variable is misspelled (therefore evaluates to the empty string) this can inadvertently delete the branch. (The same is true for tags in :refs/tags/.)

A remote repository can disable such operations with the setting receive.denyDeletes to prevent any ref deletion, and avoiding non-fast-forward branches with the receive.denyNonFastforwards. If either of these are set, then deletes have no operation and pushes cannot overwrite code which doesn't strictly follow it in history. (This is occasionally a useful operation; it may be necessary to provide a means to elevate this in certain situations if necessary.)

In addition, ensuring that branches have core.logAllRefUpdates will ensure that the repository still keeps the history of the branches, at least for gc.reflogexpire and gc.reflogexpireunreachable days.

Summary

Whilst git can be used, there are powerful options which can tweak or constrain its behaviour. In the face of scripts which have full access to the remote repository, it is advisable to have a more controlled set of options rather than the default you-can-do-anything approach. With this knowledge in mind, you should be able to set your options appropriately for your environment.

Come back next week for another instalment in the Git Tip of the Week series.

Flash in the pan

2011-11-09T09:46:00.001Z

According to ZDNet, Adobe is to stop development on Flash Mobile players.

Our future work with Flash on mobile devices will be focused on enabling Flash developers to package native apps with Adobe AIR for all the major app stores. We will no longer adapt Flash Player for mobile devices to new browser, OS version or device configurations. Some of our source code licensees may opt to continue working on and releasing their own implementations. We will continue to support the current Android and PlayBook configurations with critical bug fixes and security updates.

This doesn't mean that Adobe is out of the Flash business entirely, but it does mean that adverts and other interactive content intended for viewing on mobile devices will gradually be transitioned over to HTML5. It also signifies an increase in their HTML5 generation tools, which is ultimately Adobe's income stream from Flash.

Whether Google keeps supporting Flash for Android or not remains to be seen; and if Android stops supporting it, theni t's quite likely that it will be dead in the water. I suspect that it will continue to live on in the next Android and Windows release, but probably the subsequent one it's unlikely to happen.

JavaScript is the new bytecode

Instead, HTML5 is being seen as the new application runtime. Long predicted by Jobs, the advantage of an open runtime is that multiple companies can compete on providing the most efficient experience, whilst disassociating it from the content creation experience. Various toolkits already exist (JQueryMobile, JQueryTouch) but higher level drag-and-drop applications are still missing in action. Some systems – like GWT – compile down to JavaScript, and this may be an avenue taken by others (e.g. Dart, CoffeeScript) to give a better runtime.

However, JavaScript isn't a pleasant language to write in. Not from a syntax perspective, but from the quirks that the language gives you; for example, you can't assume that !!a is the same as a, and if you miss out on a var then serious bugs can be introduced.

Adobe already had some HTML5 generation tools available, and I expect that these will take a greater focus in the future. Even though Flash will still be supported for the desktop, mobile devices will need HTML5 and conveniently this will also run on desktops as well.

Security

Outdated software is the key reason for security exploits. Some may argue Flash as being the biggest carrier of such exploits; but in reality, any system may have remotely exploitable bugs if it's not kept up-to-date. Flash doesn't help, in that it requires Administrator privileges on Windows and OSX in order to update itself, which means the 'check for updates' frequently doesn't unless you happen to follow bad practices.

It's also worth remembering that browsers themselves (or the libraries that they depend on) form a huge attack surface. As browsers have got more complicated – especially introducing direct memory manipulation processes such as WebGL – the likelihood is that these runtimes will become the new security concerns in the future. But unlike Flash, these can't be disabled by removing a plug-in; it's there whether you like it or not.

One of the most recent examples is the latest WebKit hack on an iPhone device. By working with the JIT compiler for JavaScript performance improvements, Charlie Miller was able to download and execute rogue code through a specially developed (native) iOS application.

Summary

Mobile flash going away is a win for everyone. It promised much and delivered poor-to-average performance for the platforms it was available on. The fact that it remains a key way of distributing videos will slowly fade away with the adoption of the HTML5 video tag; in any case, many assets are already available in the standard H.264 codec, which is supported in all of the major browsers. (Firefox is no longer considered a major browser; thanks to its recent breaking upgrade policy of throwing new bits over the wall every month, it has become a joke in the browsing world.)

Whilst Adobe may be trimming down the size of its workforce and refocussing efforts, those efforts were probably always better placed in the content creation rather than content runtime tools. Unfortunately, this means an increased likelihood of seeing adverts (which cannot be blocked through plug-in filters alone) on both mobile and desktop browsers.

Git Tip of the Week: Git Notes

2011-11-08T09:00:00.000Z

This week's Git Tip of the Week is about git notes. You can subscribe to the feed if you want to receive new instalments automatically.

At a recent talk for the London Java Community (recorded video is available via the link), I presented Git and Gerrit (based on the successful screencasts I have done previously). One of the things I demonstrated was the use of git notes, so I thought writing about them and explaining what they are made sense.

When files are committed into a Git repository, they are addressed by a hash of the contents. The same is true of trees and commits. One of the benefits of this structure is that the objects cannot be modified after they have been committed (since doing so would change that hash).

However, sometimes it is desirable to be able to add metadata to a commit after it has already been committed. There are three ways of doing this:

Amend the commit message to add in the additional metadata, accepting this will change the branch.
Create a merge node with a more detailed commit, and push that (so that the previous commit is retained and can be fast forwarded).
Add additional metadata in the form of git notes.

Of these three options, only the last one will not change the current branch.

Git Notes

Git Notes are, in effect, a separate ‘branch’ of the repository (stored at .git/refs/notes). They don't show up in the git branch command (that lists .git/refs/heads by default). However, although you could check it out and manually update it, there is a command provided which helps you do that; git notes.


(master) $ git log --oneline
056ca11 More Stuff Again
9defb31 MoreStuff
0c7ff4f Additional
19b6cdf Initial
(master) $ git notes show
(master) $ git notes add -m "ToDo: Fix stuff"
(master) $ git notes show
ToDo: Fix stuff
(master) $ git log
(master) $ git log
commit 056ca11c01b47e2bfe1e51178b65c80bbdeef7b0
…

    More Stuff Again

Notes:
    ToDo: Fix stuff

When you look at the output of git log, it checks to see if there is an associated note, and if so, prints it out as if it were an appendix to the commit. Furthermore, the notes are mutable and can be updated over time:


(master) $ git notes add --force -m "ToDone: Fixed stuff"
Overwriting existing notes for object 056ca11c01b47e2bfe1e51178b65c80bbdeef7b0
(master) $ git notes show
ToDone: Fixed stuff

The advantage of the notes is that they can be updated without changing the commit message (and therefore the hash) of the item that they are referring to. Of course, this can be used for good as well as bad; but bear in mind the mutability if you need to depend on the notes' contents.

Gits all the way down …

Actually, a better title might have been “objects all the way down”, but I liked this one better.

Since Git is a content addressable database, the notes themselves are git objects. You can even view the history of the branch using git log and even check it out. But how are the notes stored?


(master) $ git log --oneline notes/commits
d6ac2b2 Notes added by 'git notes add'
5eb0ee5 Notes added by 'git notes add'
(master) $ git checkout notes/commits
Note: checking out 'notes/commits'.

You are in 'detached HEAD' state. You can look around, make experimental
…
HEAD is now at d6ac2b2... Notes added by 'git notes add'
((d6ac2b2...)) $ ls
056ca11c01b47e2bfe1e51178b65c80bbdeef7b0
((d6ac2b2...)) $ cat 056ca11c01b47e2bfe1e51178b65c80bbdeef7b0 
ToDone: Fixed stuff

The branch contains a list of notes, with file names referenced by the commit (or other object) ID that they correspond to. We can make a change here and update our notes:


((d6ac2b2...)) $ echo Note: Git notes are just objects >> 056ca11c01b47e2bfe1e51178b65c80bbdeef7b0
((d6ac2b2...)) $ git commit -a -m "Note added by me"
[detached HEAD 89e6afa] Note added by me
 1 files changed, 1 insertions(+), 0 deletions(-)
((89e6afa...)) $ git checkout master
Warning: you are leaving 1 commit behind, not connected to
any of your branches:

  89e6afa Note added by me
…
(master) $ git log HEAD^..HEAD
commit 056ca11c01b47e2bfe1e51178b65c80bbdeef7b0
…
    More Stuff Again

Notes:
    ToDone: Fixed stuff

So, we added a new commit and then switched back to master; but as the warning message told us, this has left the commit behind. We really need to update the refs/notes/comits reference if we want to see the new values:


(master) $ git update-ref refs/notes/commits 89e6afa
(master) $ git log HEAD^..HEAD
commit 056ca11c01b47e2bfe1e51178b65c80bbdeef7b0
…
    More Stuff Again

Notes:
    ToDone: Fixed stuff
    Note: Git notes are just objects

Here, the git update-ref is assigning the content of refs/notes/commits the value 89e6afa… (although it's resolving it to a full 40 character hash and checking that it exists first).

Conventions

Just a quick note on conventions; since the notes file is essentially on its own branch, the content doesn't get merged with merges between branches. If you wanted to merge git notes, then following the Key: Value on separate lines is the way to achieve git note merging nirvana. The merging options for git notes allow for appending of notes (i.e. similar to cat noteV1 noteV2) or sorting and uniquifying the data (i.e. cat noteV1 noteV2 | sort | uniq).

However, the notes don't have to be textual, nor do they have to be something which is mergeable. They don't even need to be on the notes/commits ref; you can create notes based on any reference.

In fact, this is how Gerrit works (which I've written about before). Gerrit stores its review information in the Git repository under notes/review. Ordinarily, this doesn't show up (the git log only shows notes in the notes/commits refspace), but you can make it do so if you want:


(BARE:master) $ git show refs/notes/review
commit bb7cba258eaaf4851b20b66c7ef56775f0cb4367
…
    Update notes for submitted changes
    
    * Goodbye world

diff --git a/f7f38314247063271631cfddf560ea99214cd438 b/…
@@ -0,0 +1,7 @@
+Code-Review+2: Alex Blewitt
+Verified+1: Jenkins
+Submitted-by: Alex Blewitt
+Submitted-at: Thu, 20 Oct 2011 20:11:16 +0100
+Reviewed-on: http://localhost:9080/7
+Project: SkillsMatter
+Branch: refs/heads/master
(BARE:master) $ git log HEAD^..HEAD
commit f7f38314247063271631cfddf560ea99214cd438
…
    Goodbye world
    
    Change-Id: I692f8de08938f22da9d6e26005ba44c95a1479d7
(BARE:master) $ git log --show-notes=* HEAD^..HEAD 
commit f7f38314247063271631cfddf560ea99214cd438
…
    Goodbye world
    
    Change-Id: I692f8de08938f22da9d6e26005ba44c95a1479d7

Notes (review):
    Code-Review+2: Alex Blewitt
    Verified+1: Jenkins
    Submitted-by: Alex Blewitt
    Submitted-at: Thu, 20 Oct 2011 20:11:16 +0100
    Reviewed-on: http://localhost:9080/7
    Project: SkillsMatter
    Branch: refs/heads/master

In this case, I reviewed the commit (with a +2 from me, and a +1 from Jenkins) and it's stored in the Git repository, along with everything else. Normally, it's not received by the user when pulling or cloning; but it is a permanent record on the repository (and will be visible if you e.g. do a git clone --mirror). However, if you want to fetch the notes as well you can do so:


[remote "origin"]
	fetch = +refs/notes/*:refs/notes/*
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = ssh://localhost:29418/SkillsMatter.git
	push = refs/heads/master:refs/for/master

The fetch refspec in bold allows me to pull any/all reviews from the repository and make them available in my local clone.

Exercise for the reader …

Since the Git notes can contain any blob, and it's not cloned by default (unless you specifically review it), you can create a distribution and check it into a repository. Instead of storing it in refs/notes/commit, store it in refs/notes/dist and have the binary generated from your compile system export it as a Git Note pointing to the tag. That way, if you want to check out the pre-built bundle for a given tag, you can use refs/notes/dist to point to the tag you want and extract the full binary.

Of course, you don't really need to use git notes to store any blob in the repository in any case; there's no reason why you couldn't have a refs/dists tree, with one file per tag.

Git notes demonstrates the fact that Git is not just a source code control system, like Hg or Bzr. Instead, it's a content-addressable file-system, which just happens to be able to represent trees and files (blobs) in an easy way. As a result, Git will always be capable of being extended with functionality like Gerrit and git notes, because it is not limited to what it can store in a repository – yet, the cloning of the repository can still be efficient since the data you pull from a clone is only the reachable objects from a specific commit. As a result, review notes (and/or binary distributions) need never be part of a cloned repository, even if it is persisted and available in the same Git back-end.

Come back next week for another instalment in the Git Tip of the Week series.

Ten Years of Eclipse

2011-11-07T12:00:00.000Z

I have written up the last Ten Years of Eclipse at InfoQ, including a bit about the pre-history of where Eclipse came from. I was lucky enough to work with both Visual Age for Java (which was written in Smalltalk), the Visual Age Micro edition (which was written in Java, and the forerunner of WebSphere Studio), the WebSphere Studio product itself and ultimately on through Eclipse 1.0's release to the outside world.

Other than my WebSphere roots, it wasn't until late in the development of the Eclipse 2.1 release that I got involved doing testing for the then-upcoming Mac OSX release of Eclipse. Since then, I've been regularly blogging about Eclipse, occasionally going to the EclipseCon conferences (not for a while, sadly) and even writing on EclipseZone and InfoQ about Eclipse and OSGi topics.

Still, it's not often to say you were part of a decade-long process, even if it is one at the periphery. Here's to the next ten years, and Happy Tenth Birthday, Eclipse!

SSHD Server in Java with Kerberos Authentication

2011-11-02T22:51:00.001Z

Having tried (and failed) to get a working SSHD server that supported Kerberos authentication in the past, I was pleasantly surprised by the recently-released Apache Mina SSHD project.

Setting up a project is relatively straightforward; you need to get hold of Apache Mina SSHD 0.6 Jar along with its dependencies (note; since it uses SLF4J for logging, you'll also need an SLF4J implementation such as SLF4J-Log4J12).

You can then create a simple SSH server with the following:


server = SshServer.setUpDefaultServer();
server.setPort(1234);
server.setKeyPairProvider(new SimpleGeneratorHostKeyProvider("/path/to/the/key"));

This sets up a server, running on port 1234, which persists its server key in /path/to/the/key. (It's not mandatory to persist the key; without an argument it regenerates it each time the server starts. But then your clients will complain each time they start that the key has changed if you don't.)

Mind you, as it stands, it's not very useful. It doesn't allow any authentication and doesn't do anything when users are authenticated. First, let's see how we can configure the SSHD server to do something when a connection occurs:


server.setShellFactory(new ProcessShellFactory(new String[] { "ls" });

The ShellFactory is used to instantiate a new process (copying the system input/output) for each new connection that is made. It's possible to write your own as well; in addition, you can take advantage of the SSH channels to do alternative operations.

Kerberos

For the authentication, we're going to look at Kerberos. It's a pain to get right, but the SSH server is capable of supporting it and so it's instructive to know how. This isn't a full guide to Kerberos – one assumes that you have the basics and a working Kerberos environment in the first place. If not, feel free to skip the rest of this section.

Kerberos authenticates users via principals, which are usually of the form me@EXAMPLE.COM for users, and service/canonical.host.name@EXAMPLE.COM for services. For SSH, the normal principal is host/canonical.host.name (the default realm gets added automatically).

Keytabs

To generate a service authenticated by kerberos, the server needs access to a per-host key. This is stored in a Kerberos keytab, which can be exported from the kerberos infrastructure. The details of how to extract a keytab are a part of standard Kerberos administration; in my case, I extracted out a keytab with kadmin.local and the ktadd -k canonical.host.name.keytab -norandkey host/canonical.host.name. Note that this keytab is equivalent to a hashed password, and should be protected as such. Also, the kadmin may result in the key version number being bumped and the key re-randomised; this is to prevent general extraction of the keys. You might find that the server's key is put in /etc/krb5.keytab – although it will be readable by root by default and shouldn't be opened up to the wider world – keytabs are intended only to be permissioned for the individual application(s) that require them.

The default SSH principal will be host/canonical.host.name, which is very often shared with the real SSH server running on the box. Unfortunately there is no easy way of changing this, which means that the Java SSHD server needs access to the same key.

Configuring SSHD for Kerberos authentication

Assuming you have the keytab problem solved, the next step is to hook it up to the server. You can do this with the following:


List<NamedFactory<UserAuth>> userAuthFactories = new ArrayList<NamedFactory<UserAuth>>(1);
userAuthFactories.add(new UserAuthGSS.Factory());
server.setUserAuthFactories(userAuthFactories);

GSSAuthenticator authenticator = new GSSAuthenticator();
authenticator.setKeytabFile(keytab);
// authenticator.setServicePrincipalName(principalName);
server.setGSSAuthenticator(authenticator);

The first part hooks up the server with the GSS authenticator. This is the General Security Service, which is a generic wrapper around Kerberos and other authentication mechanisms. Note that the Apache Mina implementation is specifically crafted towards the Sun implementation, so if you're running in a classloader constrained environment or on a non-Sun/Oracle JVM then there may be problems with this approach. (The CredentialManager can be overridden to be customised to solve this problem.)

The second part configures a GSSAuthenticator to allow a specific keytab to be used. Without this, authentication will fail as the server won't be able to get a kerberos credential to offer to users. (Both the server and client need to have credentials in the Kerberos world; it will often fail because the server has no Kerberos credential sourced from its own keytab.)

Finally, you can optionally specify what the principal is going to be. This is only really of use if you have a custom Java SSH client that can specify its principal. However, it can be useful sometimes if the server is calculating the default principal incorrectly; for example, many hosts have multiple interfaces and therefore the potential for multiple canonical names. The default is to calculate host/ + InetAddress.getLocalHost().getCanonicalName(); so if this is wrong, then you may choose to override it here.

Conclusion

With this code, and a working Kerberos environment, you can now start an SSH server in Java, and connect to it via SSH with Kerberos authentication.

I have put up an example project at GitHub which you can clone via http://github.com/alblue/Examples.git. You can generate a -jar-with-dependencies by running mvn, and specify a port, path to keytab and (optionally) the principal to use for authentication. Note that it includes a log4j.properties file with debug enabled; when you run it you will see a lot of output indicating the stage of the results.


$ java -jar KerberizedSshServer-1.0-jar-with-dependencies.jar 1234 /tmp/canonical.host.name.keytab host/canonical.host.name
2011-11-02 22:25:04,886 [main] INFO  org.apache.sshd.common.util.SecurityUtils - BouncyCastle not registered, using the default JCE provider
2011-11-02 22:25:13,935 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Session created...
2011-11-02 22:25:13,963 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Client version string: SSH-2.0-OpenSSH_5.6
2011-11-02 22:25:13,963 [NioProcessor-2] DEBUG org.apache.sshd.server.session.ServerSession - Received packet SSH_MSG_KEXINIT
⋮
2011-11-02 22:25:14,060 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Authenticating user 'me' with method 'gssapi-with-mic'
2011-11-02 22:25:14,062 [NioProcessor-2] DEBUG org.apache.sshd.server.auth.gss.UserAuthGSS - UserAuthGSS: found Kerberos 5
2011-11-02 22:25:14,083 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Authentication not finished
2011-11-02 22:25:14,091 [NioProcessor-2] DEBUG org.apache.sshd.server.session.ServerSession - Received packet SSH_MSG_USERAUTH_INFO_RESPONSE
2011-11-02 22:25:14,091 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Received SSH_MSG_USERAUTH_INFO_RESPONSE
2011-11-02 22:25:14,091 [NioProcessor-2] DEBUG org.apache.sshd.server.auth.gss.UserAuthGSS - In krb5.next: msg = SSH_MSG_USERAUTH_INFO_RESPONSE
2011-11-02 22:25:14,118 [NioProcessor-2] INFO  org.apache.sshd.server.auth.gss.UserAuthGSS - GSS identity is me@EXAMPLE.COM
2011-11-02 22:25:14,118 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Authentication still not finished
2011-11-02 22:25:14,120 [NioProcessor-2] DEBUG org.apache.sshd.server.session.ServerSession - Received packet SSH_MSG_USERAUTH_GSSAPI_MIC
2011-11-02 22:25:14,120 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Received SSH_MSG_USERAUTH_GSSAPI_MIC
2011-11-02 22:25:14,120 [NioProcessor-2] DEBUG org.apache.sshd.server.auth.gss.UserAuthGSS - In krb5.next: msg = SSH_MSG_USERAUTH_GSSAPI_MIC
2011-11-02 22:25:14,121 [NioProcessor-2] DEBUG org.apache.sshd.server.auth.gss.UserAuthGSS - MIC verified
⋮
2011-11-02 22:25:14,142 [NioProcessor-2] DEBUG org.apache.sshd.server.session.ServerSession - Received packet SSH_MSG_CHANNEL_REQUEST
2011-11-02 22:25:14,142 [NioProcessor-2] INFO  org.apache.sshd.server.channel.ChannelSession - Received SSH_MSG_CHANNEL_REQUEST on channel 0
2011-11-02 22:25:14,142 [NioProcessor-2] INFO  org.apache.sshd.server.channel.ChannelSession - Received channel request: shell

To connect from the same host, you likely can do ssh -p 1234 canonical.host.name, and because the user will have access to the per-user credential cache (if you're running the server as the same userid) then it will Just Work™

However, if you're connecting from a remote host, then you may need to invoke it with ssh -K -p 1234 canonical.host.name instead. The -K says to forward the ticket to the server (which is why it says GSS identity is me@EXAMPLE.COM in the debug trace). Your SSH client can supply per-server or global configurations to always forward Kerberos tickets (GSSAPIAuthentication yes in .ssh/config). Without the forwarded Kerberos ticket, the server can't authenticate you against the kerberos mechanism:


2011-11-02 22:30:28,632 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Authorized authentication methods: gssapi-with-mic
2011-11-02 22:30:28,634 [NioProcessor-2] DEBUG org.apache.sshd.server.session.ServerSession - Received packet SSH_MSG_USERAUTH_REQUEST
2011-11-02 22:30:28,634 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Received SSH_MSG_USERAUTH_REQUEST
2011-11-02 22:30:28,634 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Authenticating user 'me' with method 'none'
2011-11-02 22:30:28,634 [NioProcessor-2] INFO  org.apache.sshd.server.session.ServerSession - Unsupported authentication method 'none'

If you're connecting from a Windows host then you'll need to have a client which is capable of making SSH connections and forwarding the Kerberos tickets. Some versions of Putty have GSSAPI support but check whether a Unix-Unix connection works before trying to debug Windows related problems. Note that some clients (e.g. Eclipse EGit) do not support Kerberos-only authentication, but instead use password-based authentication.

Note that it is possible to configure SSHD to use passwords as well (see the UserAuthPassword.Factory()) but this is outside the scope of this post.

Google Screws Up Reader

2011-11-01T09:39:00.001Z

Not content with the death of Buzz, Google seems intent on killing of one of the only two products that I use. Today, upon logging into Google Reader, I discovered that the UI has completely fallen apart. From the ~~welcome~~ goodbye note:

Like many of Google’s products, Reader has gained a clean new look, with more space and less clutter, making it even faster to navigate your feeds.

More space is the exact opposite of what's required, both in Reader and in Google+. Whereas in Twitter (and the old Reader) it's possible to see many tweets at once, now it's possible to see even less in Google Reader.

As an example, on my Mactop I have 12 tweets showing. On Google Reader, with a window exactly the same size, I have four items showing. And each of those are two lines long, probably fitting into the size of a tweet (give or take).

Yes, it's nice that they're finally standardising on a single Like button (and having all those Likes in one place makes sense, rather than spread around on an app-by-app basis) but let's face it, Google fired the wrong UI team. Google+ has become synonymous with spam notifications and whilst there are three or four people I follow who regularly write many paragraphs (and let's face it, some of them are Googlers themselves) all that happens is I essentially ignore any updates which aren't on the home screen. The fact of the matter is that Google+ is text-heavy and information-light, whereas Twitter is the exact reverse.

But the point of a blog reader is to be able to (a) remember which ones you have read, and which ones you haven't; (b) display that information so you can scan headlines quickly and determine if you want to see more. The abstract is like the twitter message and the body is like the twitlongerer or whatever service you use.

There will always be people who complain because it's different, and because it's changed. I fully expect those three or four people who like ~~Buzz~~ Google+ to like the new unified look of the two. But Reader has just gone several steps backwards in usability thanks to a user interface which appears to be modelled on the East Coast's current snowstorm whiteout, with just a few bits visible here and there.

Can anyone recommend a good Mac (or Windows) client for reading Google Reader and synchronising the read state across multiple devices? Or even a completely new reader UI from a different company? I'm not staying around for this UI disaster.

Git Tip of the Week: Git Flow

2011-11-01T09:00:00.000Z

This week's Git Tip of the Week is about git flow. You can subscribe to the feed if you want to receive new instalments automatically.

Following on from last week's post on merging, and specifically discussing git merge --no-ff, we will be looking at a popular Git strategy known as git flow.

Originally posted as Git Flow: a successful branching model, it has become popular in a number of larger projects which use branching as a way of introducing features.

Feature branches: the key point in git flow is the creation and use of feature branches. Their purpose is to allow parallel (or independent) feature development which can subsequently be merged back into a consolidated branch. Of course, Git is particularly well suited to this task since branches are free to create and merging can be done easily afterwards.

Feature branches are merged back onto the main development branch, known as develop in git flow. The key point with git merge --no-ff is that it gives you a merge node, even if the develop branch can otherwise be fast forwarded. This merge node allows you to determine on which branch you create the fix in the first place.

Although many consider “feature” to be a large term, in fact, it's quite possible to think of a micro feature as well. Some developers will create a new feature branch for each bug that gets filed, which then preserves a link with the bug that was filed in a bug tracker. (You can also use arbitrary commit messages as well to enable this linkage.)

Release branches: once the develop branch is ready for a release, it is branched onto a (pre)-release branch. This is then used for bugfixes for that release only (which are merged back into the develop branch) in preparation for the final tagging. Each branch is named for its version number; so release-1.0 would be the branch used for developing the code ahead of the 1.0 release itself. Once development of the 1.0 release is finalised, it will be tagged and work on the release-1.0 branch will be stopped. (Git Flow suggests the branch is deleted at this point to prevent accidental further work on that branch. Since it is tagged, it's easy to recreate if ever needed.)

Hotfix branches: there may be a need to spin up a new release quickly (sometimes also known as patch releases) for a production issue, but ahead of the next major (or minor) release. Hotfix branches are also named for the release (e.g. hotfix-1.0.1) and only contain the specific bugfixes for that release. Changes are merged back into the develop branch to ensure that the bug is also fixed in subsequent releases as well. Typically only a few commits exist on hotfix branches at a time, since they often merge back into the release.

What's in a name?

None of these names really change what you can achieve with Git; after all, one can just as easily create a branch from a tag as well as a branch. Indeed, the git flow model suggests work is done against the develop branch instead of the master branch (which is conventionally the main branch for Git based development) – in git flow, the master branch is reserved for the released tags only.

However, one key point is that the merge nodes store the name of the branch that they merged from. This means if the branch names are used, you can trace the flow of a feature from the feature branch through to the develop branch and ultimately a release branch. There's a great image (which I strongly encourage you to look at in the context of the original post).

To demonstrate how it works in practice, here's a shortened sequence of development in Git Flow and the resulting merge tree.


(master) $ git log --oneline
4f5da46 Initial Commit
(master) $ git checkout -b develop
Switched to a new branch 'develop'
(develop) $ …
(develop) $ git checkout -b feature1 develop
Switched to a new branch 'feature1'
(feature1) $ …
(feature1) $ git checkout -b feature2 develop
Switched to a new branch 'feature2'
(feature2) $ …
(feature2) $ git checkout develop; git merge --no-ff feature2
Merge made by recursive.
…
(develop) $ git checkout develop; git merge --no-ff feature1
Merge made by recursive.
…
(develop) $ git checkout -b release-1.0 develop # ready for 1.0
Switched to a new branch 'release-1.0'
(release-1.0) $ echo 1.0 > version; git add version; git commit -m "Version 1.0" version 
[release-1.0 c51b802] Version 1.0
(release-1.0) $ git checkout master
Switched to branch 'master'
(master) $ git merge --no-ff release-1.0
Merge made by recursive.
(master) $ git tag -a 1.0 # tag the release once it's finished
(master) $ git checkout -b hotfix-1.0.1 master # create a hotfix for 1.0
Switched to a new branch 'hotfix-1.0.1'
(hotfix-1.0.1) $ echo 1.0.1 > version; git add version; git commit -m "Version 1.0.1"
…
(hotfix-1.0.1) $ git checkout master
Switched to branch 'master'
(master) $ git merge --no-ff hotfix-1.0.1
Merge made by recursive.
(master) $ git checkout develop # merge the hotfix back into 'develop'
Switched to branch 'develop'
(develop) $ git merge --no-ff hotfix-1.0.1
Merge made by recursive.
(develop) $ git branch -d hotfix-1.0.1
Deleted branch hotfix-1.0.1 (was f43bb33).

Although the edits above don't have any real work contained in them, the point of the merges allows us to generate this psuedo real-world looking merge tree:


(develop) $ git log --decorate --graph --oneline
*   17e4c5f (HEAD, develop) Merge branch 'hotfix-1.0.1' into develop
|\  
| * f43bb33 (hotfix-1.0.1) Hotfix2
| * 50a102d Hotifx1
| * d2fe7d6 Version 1.0.1
| *   d534111 (tag: 1.0) Merge branch 'release-1.0'
| |\  
* | \   77e7eac Merge branch 'release-1.0' into develop
|\ \ \  
| | |/  
| |/|   
| * | 9261fad (release-1.0) Release10Fix2
| * | 2562cb3 Release10Fix1
| * | c51b802 Version 1.0
* | | fbdd638 Work6
* | | 7353285 Work5
|/ /  
* | 538bb9a Work4
* |   cdef254 Merge branch 'feature1' into develop
|\ \  
| * | 793b1bc (feature1) Feature1Work4
| * | 3f2be07 Feature1Work3
| * | 7879593 Feature1Work2
| * | 8e01f4d Feature1Work1
* | |   9239c56 Merge branch 'feature2' into develop
|\ \ \  
| |_|/  
|/| |   
| * | 6752543 (feature2) Feature2Work3
| * | d065311 Feature2Work2
| * | 040fcaf Feature2Work1
| |/  
| * a9a97be Work3
| * a4253c8 Work2
| * 163c835 Work1
|/  
* 4f5da46 Initial Commit

This might seem noisy, but we can condense it down by using the --merges flag of git log, which restricts us to just the merge nodes:


(develop) $ git log --decorate --graph --oneline --merges 
* 17e4c5f (HEAD, develop) Merge branch 'hotfix-1.0.1' into develop
| * d534111 (tag: 1.0) Merge branch 'release-1.0'
* 77e7eac Merge branch 'release-1.0' into develop
* cdef254 Merge branch 'feature1' into develop
* 9239c56 Merge branch 'feature2' into develop

At this point, the benefit of using the --no-ff becomes clear; it is a way of documenting the merges such that they can be filtered by the git log command. In addition, the naming conventions of Git Flow enable a well described set of feature and release branches that is identifiable from the graph.

Whether or not Git Flow is right for you now, it's worth being aware of what it is (and why it follows the path that it does). Even if it's not something that works for you now, as your repositories scale up and your use of git grows, it may be worth coming back to in the future. There's also a set of scripts available at the GitHub repository, which makes managing the individual branches easier.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Merging Revisited

2011-10-25T09:00:00.000+01:00

This week's Git Tip of the Week is about merging. You can subscribe to the feed if you want to receive new instalments automatically.

We previously looked at merging back in April, but we touched on the the index last week, and the index (or stage) numbers, which play a part in merges. This week, we'll look at what that means with different kinds of merges. If you're not familliar with merges, take a look at the previous post first.

Merging allows you to bring together two (or more) commits into a single merge commit, whilst at the same time bringing together all of the different files combined in the two. If there are merge conflicts, these need to be resolved individually. Merges where one of the commits is an ancestor of the current commit are referred to as fast-forward merges, and do not result in a merge commit being created. When you pull from a remote source, you're actually doing a fetch and a merge in one step – though usually, you can just fast-forward on when you do.

For the purposes of this post, I'll create three branches called ‘Alex’, ‘Bob’ and ‘master’, in homage to Alice and Bob. Each branch will have a file called Hello which will contain the text “Hello, my name is Alex” (c.f. Bob) as well as an AlexsFile, BobsFile etc.


(master) $ git ls-tree Alex
100644 blob ada4c4c4f33cd190fe40769d5ca9826adb9fb7ce	AlexsFile
100644 blob ca4eef2f4e3f1fe92028176cb547b590a08c2259	Hello
(master) $ git ls-tree Bob
100644 blob eea826732acee08a8cf83445e3b98cf58f11ce5c	BobsFile
100644 blob ca4eef2f4e3f1fe92028176cb547b590a08c2259	Hello
(master) $ git ls-tree master
100644 blob ca4eef2f4e3f1fe92028176cb547b590a08c2259	Hello

Strategies

When you create a merge, you have the option of saying how the merge will be processed. The usual strategy is recursive. This means Git will walk each directory (tree) and find out which files have differences compared to the base revision, and then use the one with changes. (If both have changes, the new file's contents are merged textually, and if there's a problem with that, a conflict ensues.) This is why you see the message “Merge made by recursive” after you do the operation:


(master) $ git checkout Alex
Switched to branch 'Alex'
(Alex) $ git merge Bob
Merge made by recursive.
 BobsFile |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 BobsFile
(Alex) $ git log --oneline --graph
*   d612aab Merge branch 'Bob' into Alex
|\  
| * 8afb2d3 Bob's File
* | 4abf59e Alex's File
|/  
* 5cba624 Hello
(Alex) $ git show d612aa
commit d612aab9858289ed027230d3b9a7b2a7a5e75945
Merge: 4abf59e 8afb2d3
…
    Merge branch 'Bob' into Alex

The commit d612aa… is a merge node in that it brings two separate branches together. Since neither is an ancestor of the other, they can't be fast-forwarded, and as such, the merge node is created. We can determine the commits' parents with HEAD^1 and HEAD^2:


(Alex) $ git rev-parse HEAD^1
4abf59ef73c186e93db25e8b7bc4423fbd11bbd0
(Alex) $ git rev-parse HEAD^2
8afb2d368ce26ca71cec539c31400c7001a18efc

It's even easier when there are no changes to be merged:


(Alex) $ git checkout master
Switched to branch 'master'
(master) $ git merge Bob
Updating 5cba624..8afb2d3
Fast-forward
 BobsFile |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 BobsFile

The fast-forward here indicates that master is behind Bob, and so we can simply move the pointer forwards – as a result, we don't need to create a merge node.

So, two strategies that Git uses are to do a fast-forward or merge made by recursive. This covers 99% of the merges that you'll need to do; but it's worth noting that Git has a few more tricks up it's sleeve which can help in certain situations.

Octopus Merge

The examples above all have one or two parents. However, a Git merge node is capable of representing more than two heads in a merge, and it uses a strategy called octopus. This is selected by default when you merge more than two branches:


(master) $ git reset --hard 5cba624b94a7a622183c960697867c8bba73aa91
HEAD is now at 5cba624 Hello
(master) $ date > NewFile
(master) $ git add NewFile
(master) $ git commit -m "New File"
[master 598ad85] New File
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 NewFile
(master) $ git merge Alex Bob
Trying simple merge with Alex
Trying simple merge with Bob
Merge made by octopus.
 AlexsFile |    1 +
 BobsFile  |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)
 create mode 100644 AlexsFile
 create mode 100644 BobsFile
(master) $ git log --oneline --graph
*-.   5a2aa0d Merge branches 'Alex' and 'Bob'
|\ \  
| | * 8afb2d3 Bob's File
| * | 4abf59e Alex's File
| |/  
* | 598ad85 New File
|/  
* 5cba624 Hello
(master) $ git show
commit 5a2aa0da3d3b3365703d710dad8aeebc0770b8ef
Merge: 598ad85 4abf59e 8afb2d3
…
    Merge branches 'Alex' and 'Bob'
(master) $ git rev-parse HEAD^1 HEAD^2 HEAD^3
598ad850114e1f7445ee8b02e93ee23060439560
4abf59ef73c186e93db25e8b7bc4423fbd11bbd0
8afb2d368ce26ca71cec539c31400c7001a18efc

In this case, we have three commits merging together into the merge node; the master (which had diverged), and the Alex and Bob branches from before. Since our merge node now has three parents, we can use git rev-parse to convert the HEAD^1/2/3 into the first, second and third heads.

What if the files are conflicted?


(master) $ git checkout Alex
Switched to branch 'Alex'
(Alex) $ echo Hello, my name is Alex > Hello 
(Alex) $ git commit -a -m "My name is Alex"
[Alex c2cb955] My name is Alex
 1 files changed, 1 insertions(+), 1 deletions(-)
(Alex) $ git checkout Bob
Switched to branch 'Bob'
(Bob) $ echo Hello, my name is Bob > Hello  
(Bob) $ git commit -a -m "My name is Bob"
[Bob 7cb6225] My name is Bob
 1 files changed, 1 insertions(+), 1 deletions(-)
(Bob) $ git checkout master
(master) $ git merge Alex Bob
Trying simple merge with Alex
Trying simple merge with Bob
Simple merge did not work, trying automatic merge.
Auto-merging Hello
ERROR: content conflict in Hello
fatal: merge program failed
Automatic merge failed; fix conflicts and then commit the result.
(master|MERGING) $ git ls-files --stage
100644 ada4c4c4f33cd190fe40769d5ca9826adb9fb7ce 0	AlexsFile
100644 eea826732acee08a8cf83445e3b98cf58f11ce5c 0	BobsFile
100644 ca4eef2f4e3f1fe92028176cb547b590a08c2259 1	Hello
100644 a5a820416bae2c7b77340e5b2120aab9595d2bfc 2	Hello
100644 98b16693fe64acb9d002af1fe5f5162d58bd40b4 3	Hello
100644 09d774502f97ba9a46f25f8f11601b653c376828 0	NewFile
(master|MERGING) $

We can launch a three-way diff tool with git mergetool:


(master|MERGING) $ git mergetool
(master|MERGING) $ git mergetool
merge tool candidates: opendiff kdiff3 tkdiff xxdiff meld tortoisemerge gvimdiff diffuse ecmerge p4merge araxis bc3 emerge vimdiff
Merging:
Hello

Normal merge conflict for 'Hello':
  {local}: modified file
  {remote}: modified file
Hit return to start merge resolution tool (opendiff): 
…
(master|MERGING) $ git commit -a -m "Merged"
[master 999a938] Merged

Note that the Octopus merge does not handle conflicts of more than 2 files. If you do, you end up with a different error mesage:


(master) $ echo Hello World > Hello 
(master) $ git commit -a -m "Hello World"
[master fe79e59] Hello World
 1 files changed, 1 insertions(+), 1 deletions(-)
(master) $ git merge Alex Bob
Trying simple merge with Alex
Simple merge did not work, trying automatic merge.
Auto-merging Hello
ERROR: content conflict in Hello
fatal: merge program failed
Automated merge did not work.
Should not be doing an Octopus.
Merge with strategy octopus failed.

Ours

Finally, the last strategy that's worth knowing about is the ours strategy. This takes any number of heads, and creates a merge node, but without actually doing any changes. In other words, a git diff HEAD^1 will always return empty for an ours strategy:


(master) $ git diff HEAD^1
(master) $ git log --oneline --graph --decorate
*-.   c5f84cc (HEAD, master) Merge branches 'Alex' and 'Bob'
|\ \  
| | * 7cb6225 (Bob) My name is Bob
| | * 8afb2d3 Bob's File
| * | c2cb955 (Alex) My name is Alex
| * | 4abf59e Alex's File
| |/  
* | 598ad85 New File
|/  
* 5cba624 Hello
(master) $ git rev-parse HEAD^{tree}
78784bb4dea678c157d8711bc56c5478a74588c3
(master) $ git rev-parse HEAD^1^{tree}
78784bb4dea678c157d8711bc56c5478a74588c3

We can verify that these are identical, since tree pointed to by HEAD is the same as the tree pointed to by HEAD^1 (i.e. the parent). The suffix ^{tree} is used to show the tree associated with the commit.

The --decorate argument to git log adds the branch names to the output, which can be useful in showing where merges have come from. The --graph argument is mostly of use with the --oneline argument; although you can run a git --graph the full commit messages tend to hide the structure of the graph.

The ours strategy is only really useful if you want to encode a set of previous commits but not have them affect the current master (say, because you've cherry picked some of the contents and don't want to take other parts, but preserve them in the history as-is somehow).

Merge Message and Fast Forwards

The merge message will be created automatically, based on the names of the branches you are merging. However, it's possible to pass a -m option, like with git commit, to supply an additional message. This can be useful if the merge message needs additional information encoded (such as which bug(s) were fixed).

It's also possible to force a merge, even if one isn't necessary. If you have topic-based branches, it can be useful to denote that the work was carried out on a separate branch before being merged back into master. Running git merge --no-ff will create a merge node, whether or not the branch can be fast-forwarded it or not. Since the merge commit has the branch name you're merging from as part of the commit, you can end up with descriptive names to show the feature having been completed:


(master) $ git checkout -b "bug12345"
Switched to a new branch 'bug12345'
(bug12345) $ echo BugFix >> Hello 
(bug12345) $ git commit -a -m "Fixing bug 12345"
[bug12345 e2bd64e] Fixing bug 12345
 1 files changed, 2 insertions(+), 0 deletions(-)
(bug12345) $ git checkout master
Switched to branch 'master'
(master) $ git merge bug12345 # without --no-ff
Updating 999a938..e2bd64e
Fast-forward
 Hello |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
(master) $ git reset --hard HEAD^
HEAD is now at 999a938 Merged
(master) $ git merge --no-ff bug12345 # with --no-ff
Merge made by recursive.
 Hello |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
(master) $ git log --oneline --graph --decorate
*   4a905c8 (HEAD, master) Merge branch 'bug12345'
|\  
| * e2bd64e (bug12345) Fixing bug 12345
|/  
*-.   999a938 Merged

Being able to merge with a --no-ff can be useful when using some kinds of commit workflows, where many branches are used to develop individual features and then brought into a master branch subsequently. It's also worth noting the --merges argument allows you to filter just the merges in a repository:


(master) apple[merge] $ git log --oneline --merges --decorate
4a905c8 (HEAD, master) Merge branch 'bug12345'
999a938 Merged

Next time, we'll look at what we can do with the different Git workflows using merge --no-ff.

Come back next week for another instalment in the Git Tip of the Week series.

Buzz Buzzes Off

2011-10-18T10:33:00.001+01:00

As announced last week, Google have finally done the inevitable, shutting down Google Buzz. Good riddance, says I. When it launched, I was sorely unimpressed, both with the style of its launched (remember how it advertised every contact you ever mailed?) and of the quality of the implementation itself.

For example, Google Buzz sent me a mail message whenever I posted something on it. Good grief – the fact that I've posted it should be enough of a clue that perhaps I knew what I said? I, like many others, ended up setting e-mail rules just to delete any and all mail that came from Buzz that had my name in it, just so I wouldn't get notified of my own buzzes.

When it launched, the API was pretty sparse (and hey, look – they repeated the same mistake that was Google+). Unfortunately, whilst Google may know a lot about systems they know nothing of platforms, and Buzz was always sitting in a wasteland whilst others, such as Facebook and Twitter, took off.

Google Buzz 2.0: Also known as Google+, though slightly less googleable. I have absolutely no idea why they ran these for different periods of time. It would be like Apple continuing to run iTools after MobileMe was announced, or continuing to run MobileMe after iCloud was announced. Apple has a switchover date – with a period of testing for developers and other interested users – but when the Big Switch happens, it's on with the new and off with the old. Yes, things can go wrong (Exhibit A: Mobile Mess) but at least you know there's a transition, and the data is migrated automatically.

The real question is why Google didn't kill off Buzz earlier. Perhaps it's a case of different teams fighting for fiefdom in the Google realm, or maybe it's just that the Buzz team turned into the Plus team and didn't think about turning the lights off as they left the room.

Google selcriC

The real problem, of course, is that Google+ is exactly the same as Google Buzz. Yes, there's a snazzier UI for letting you follow, but as with politics, it's out with the old and in with the same old new. The key issue is that it is exactly backwards from what it needs to be. Even the name suggests this to be the case; you follow someone, you don't ask them to follow you. But that's exactly the UI that you have in Google+, other than the Public scope.

I have a lot of different interests. I hack code on the OSX Kernel (specifically ZFS), I write a lot about Git, I am an advocate of OSGi and occasionally I a have been known to go flying.

The chance of someone being interested in this exact set (or a superset, let's be fair) is pretty close to zero. If I were to publish everything on Insert Social Network Here, I would kill off many followers that I have. I'm guessing that a fair proportion of people who follow my blog pick up on one of the specific feeds – such as the gtotw or eclipse or osgi feeds.

What Circles really needs is a way of saying “Here are a set of things that I write about; add yourself to them” That way, if someone wants to learn more about Git but doesn't want to hear about OSGi, then they can do so. I can then target posts I make to that group to specifically address the Git posts, instead of the wide ranging ones.

That doesn't diminish the need for a Public group, nor does it stop the requirement for having private groups, though the private ones can accidentally go public, as Steve Yegge recently found.

Google+ Tags

What we really need is an upgrade to Google to support arbitrary tags. These can be the names of existing circles, or yet-to-be-created circles. A circle can either be public or private; in the former case, anyone can opt in/out but in the latter case, you have control over who will opt in.

When I post an item, I just give it a list of tags. I may choose to use ZFS or Eclipse or OSGi, and followers of those circles will see that post. If the post isn't also tagged as public, then only those will see it.

I can thus manage my groups to the set of private circles I want (i.e. mostly 'Friends' and 'Family) and any other circle gets automatically created and membership managed by those who want to receive notifications.

Unfortunately, it looks like too much emphasis was given to UI and not enough to usability. There is an entirely different aspect between UI and usability which seems to have been missed by the designers of the Google+ infrastructure. For example, when I post something, it takes a good few seconds (depending on network connection speed) from clicking on the 'Post an update' button to actually writing something. Why it needs to do any kind of network connectivity at this point is beyond me. There should be a way of putting flat tags in, with perhaps a background notification indicating if a tag represented a new circle or not – but it shouldn't let a post be prohibited just because it needs to find a list.

Finally: Notifications

The notifications in Google+ drive me batshit insane. The red box number at the top of the screen has become a pointless red box – regardless of whether there's something interesting to see or not, it always highlights a non-zero number. This is often because people have started to follow me, but there is absolutely no setting which says 'STFU' and only let me focus on the things that I want to see. If anyone knows how to get rid of that damn red box, please leave a message in the comments. To quote an anonymous Twitter comment, “Google+ has gone from a way of sharing information to a way of being annoyed by notifications”

Git Tip of the Week: Index Revisited

2011-10-18T09:00:00.000+01:00

This week's Git Tip of the Week is about indexes. You can subscribe to the feed if you want to receive new instalments automatically.

In last week's tip we visited the purpose of the index. But what actually is it?

It's not actually a tree object, as I alluded to last time. That is, you can't iterate the contents with git ls-tree. It does point to blobs in the object database, however. So why do we need a different type of object to refer to the index?

Some of the reasons are performance oriented. Whenever you do a diff (or other repository-wide operation), Git needs to quickly and efficiently compute whether the state of the working tree has changed since the last index. Some tools, including the bash shell prompt, need to be able to determine if the working tree is dirty or not quickly:


(master) $ git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>…" to update what will be committed)
#   (use "git checkout -- <file>…" to discard changes in working directory)
#
#	modified:   example
#
no changes added to commit (use "git add" and/or "git commit -a")
(master) $ export GIT_PS1_SHOWDIRTYSTATE=true
(master *) $

If Git only knows whether the tree is dirty by doing a full walk of the contents (and calculating their SHA1 hashes), this operation would be prohibitively expensive. Fortunately, Git has a number of optimisations that allow it to avoid this case.

The index stores not only the file names, but also the last modification time of those files. As a result, Git knows whether there have been any changes to timestamps, by iterating through the files' metadata and comparing the timestamps with those in the index. If a file is missing from the index, it's represented as an addition. If a file is missing from disk, it's represented as a deletion. If a file's modification time is different then this is represented as a modification.

As well as storing the timestamps, the index also stores the SHA1 hashes of each blob. This allows the index to update itself, should the file be reverted to a previous state but with a later timestamp.

Finally, the index is also used for processing merges. In the index, there is a concept of having multiple index numbers (or stage numbers). Normally, only 0 is used since this represents the state of the current working tree. However, if a merge conflict arises, then the index is used to disambiguate the state of the files at each level. If you have a conflict, then stage 0 is used to represent the current working tree, stage 1 is used for your change, then stage 2 and 3 for the other differences. You can see the stage number by running git ls-files --stage (or -s):


(master) $ git status
(master) $ git ls-files -s
100644 ce013625030ba8dba906f756967f9e9ca394464a 0	example
(master) $ git pull # with known conflict
Auto-merging example
CONFLICT (content): Merge conflict in example
Automatic merge failed; fix conflicts and then commit the result.
(master|MERGING) $ git ls-files -s
100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 1	example
100644 a895c0db0a627cb9451ae390a2a0922495dbb161 2	example
100644 13940d48c3d3693113f7543d4fc5423a916ef55d 3	example

The stage 1 file here is the place the two files derived from, so we can do 1..2 and 1..3 diffs to find out what each side of the tree has changed since. (The more astute of you will recognise e69…391 as the empty file.) We can show the contents of these versions of the files, or load them into a 3-way diff tool if you have such a thing:


(master) $ git show e69de #empty file
(master) $ git show a895c 
Left tree
(master) $ git show 13940
Right tree

Of course, normally Git will handle the diffs for you and you don't need to worry about the specific changes, nor extracting the contents out of the index. But it does highlight the fact that when you have finished doing a merge of a single file, running git add on the file puts a copy in stage 0 of the index, removing the 1,2,3 indexes:


(master|MERGING) $ git add example
(master|MERGING) $ git ls-files -s
100644 319c128291474d30f48e721ca87bd10425e8e296 0	example

This is why merging large conflicting changes with Git is easy. Each file can be addressed on a file-by-file basis; when you have finished merging a file, you can add it to the index, which records both its contents as well as removing the other files from the merge status. Merging many files then becomes an exercise in merging them one-by-one, and adding them as you go. And since they're all transiently stored in the index, you can keep adding them until you are ready to perform a git commit.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Understanding the Index

2011-10-11T09:00:00.000+01:00

This week's Git Tip of the Week is about indexes. You can subscribe to the feed if you want to receive new instalments automatically.

In the previous Git Tip of the Week, we looked at interactive adding; that is, the ability to add just parts of a file instead of the whole file as part of the commit.

It's about time we focussed on the staging area of Git, which we implicitly used when we added parts of the file last time. This is one of the main differences between Git and other version control systems, and often people are confused about its purpose.

The staging area of Git allows you to freeze a state of your working tree, such that the subsequent git commit takes that frozen state and uses it as the point of contact. Many version control systems only let you freeze at the point of commit and so don't have this intermediary stage; and when you are using Git for the first time, you will often have a habit of using the git add to be immediately followed by a git commit, or even setting up an alias to do everything.

So why does Git have the concept of an index, anyway? Well, remember that Git uses content-addressable files; in other words, when you have a specific piece of content (like the empty file) it always has the same identity – e69..391 – whatever the file is called. What happens when you run git add is that the object is added to the object database. As well as the object being added, it needs a pointer to point to it, so there's a virtual tree called the index which contains a tree, which points to the blobs contained therein.

When you add files (or rm files), you really end up modifying this tree which represents what you'd like to do next. When you run commit, it takes that tree, builds a valid tree object, and commits that to the database (as well as updating the branch, if any).

Although it may not immediately seem useful to have this feature (and some argue that this is an example of Git's complexity over other systems), it can be very beneficial for doing specific operations; for example:

Staging parts of a file to break it up into different commits (as last time)
Working with large merges, where many files may have conflicts (you can record which ones don't have conflicts, and ones that you've already worked with, but running git add; you're then left with a decreasing number of differences to process)

When you run git status, it tells you all you need to know about how the index corresponds to your current working tree, giving you different messages about the files that it finds. To speed up processing, Git usually uses timestamps to determine if a file has been changed, but in doing a full processing sweep will calculate the SHA1 hash of the contents of the files (and thus, the directories) to determine differences against the index:


(master) $ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	deleted:    deleteme
#	renamed:    same -> renamed
#
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	modified:   changed
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	addme

The contents of the index are really the snapshot of the files that have been modified. For example, the renamed and deleted above show changes which have been staged (i.e. added to the index) whilst the not staged changes have been modified but are not yet committed. The index also allows for quick identification of changes in the local repository which have yet to be added.

So, for a given file, there are possibly three separate copies whilst working on it. There's the previous version that was committed (i.e. HEAD), there's the current version on disk (the working tree) and a third copy, which is a cached version in the index. That's why, when you have a local change combined with one that already exists you might see the same file twice in the status message:


(master) apple[example] $ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	modified:   three
#
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	modified:   three
#

You can use git diff to show you the differences between the two files:


(master) $ git diff
diff --git a/three b/three
index bb87574..3179466 100644
--- a/three
+++ b/three
@@ -1 +1 @@
-This is the version in the index
+This is the version in the working tree
(master) $ git diff --cached
diff --git a/three b/three
index 48d0444..bb87574 100644
--- a/three
+++ b/three
@@ -1 +1 @@
-This is the previously committed version
+This is the version in the index

In the second example, the --cached says to compare between the index and the previous commit (otherwise it's comparing the index and the working tree). You can, if you want, get the full contents of each of those files as long as you know the hashes (shown above in the diffs):


(master) $ git show 48d0444
This is the previously committed version
(master) $ git show bb87574
This is the version in the index
(master) $ cat three
This is the version in the working tree

The last one, of course, doesn't have a Git object yet by virtue of the fact we've not added it yet. If we were to add it, we'd replace the previous version in the index. We'll take a deeper dive into the index next time.

Come back next week for another instalment in the Git Tip of the Week series.

One Last Thing

2011-10-06T09:54:00.002+01:00

As many of you have no doubt by now read, Steve Jobs passed away on 5th October 2011. As a mark of respect, the landing page for Apple has been replaced with a simple iconic image.

Steve has been in my technical career most of my adult life. The first computer I bought was a Nextstation pizza box, where I was introduced to Nextstep 3, and my first real coding language was Objective-C. (When Java came out, three years afterwards, being able to pick up object orientation was trivial since the ground work had already been done.) Although I was never an 'original' Apple user, I did move over and pick up my first PowerBook G4 running at a steaming 500MHz to run OSX 10.0, and I never looked back. These days, I'm still sitting in front of the same OS – with cosmetic changes, of course – and I'm carrying around an always-on device, running the same OS and far more powerful than my original Nextstation.

Back when Steve retired from Apple, it was clear that it was the beginning of the end. Although he kept going a couple of months, the downward slide had begun and his retirement didn't last long. At least he lived long enough to see Tim Cook and friends launch the iPhone 4S, which would have been a distinctly different affair had events transpired the other way around.

However, we're all going to die sometime. The point of life is to enjoy and achieve greatness in the brief period of time we have on this world. Without a doubt Steve changed the world for the better. (The original packaging of my Nextstation has the line “At NeXT we believe a few people can change the world”, and I know of no better tagline that can describe Steve's outlook.) Quite apart from being a showman and lead presenter for Apple's product announcements and presentations, his vision resulted in significant changes in the way that technical and entertainment industries operate today. Here are just some of them:

Original Apple PC. Back before PC was called PC, Apple arrived on the desktop computer scene. IBM PCs would later join in the fray, and eventually overtake them, but Apple stood out as part of that initial phase from home computers to work comptuers.
Nextstep operating system. Five years before Linux 1.0 happened, Nextstep already had a fully featured operating system, and could play videos, record sound and embed objects. In comparison, Windows hadn't even reached version 3 by the time of Nextstep's debut.
NeXT hardware. Although it didn't catch on commercially, the systems had high-quality defaults. Ethernet as standard, 24-bit colour, 16-bit sound, built in audio mic and speakers (and this in the day when SoundBlaster was a recently launched product rather than a generic API). True, it was overpriced for what it was – but the first version of HTTP was written on a NeXT box.
iMac and general iLine. Bringing technology to the masses, rather than to the geeks. The all-in-one form factor of the iMac remains to this day (although the factor has shifted with the transition from CRTs to flat screens).
Aggressive phase-out of old technology and support of new technology. Floppy drives, PS/2 connectors (or ADB connectors) all went away with the advent of the iMac. USB and FireWire were their replacements, which still exist today. These days, with the MacBook Air even the DVD has gone, as has minor inconveniences like the removable battery.
Computer Animated Movies. Pixar significantly altered what was possible with the release of Toy Story, which wouldn't have been possible without Renderman, a Nextstep based graphical animation tool. And of course, Steve's involvement with Pixar helped it to go on to many further movies, including an involvement with Disney for mass adoption.
Music. Every MP3 player is generically called an iPod, even if it's not an Apple system. They weren't the first (“No wireless. Less space than a nomad. Lame.”) but they did make it easy to get music on a portable device. The original one only had space for 1,000 songs – which most people rarely exceed their regular playlists in any case – but it went on to define a generation of devices. In particular, the success of the iPod led to the creation of the iTunes store and then, an opportunity for the music cartel to get a profit from buying songs instead of pirating them through the likes of Napster. Since then, Amazon have gone on to create a successful MP3 store and DRM has gradually been removed, but without Apple's bargaining power with the media cartels this wouldn't have happened.
Movies. Similar with music, although we've not seen as many copy-cat stores to date. But digital media consumption is clearly the path of the future and Apple is helping to lead the way.
Smart touch-based phones. Again, Apple wasn't the first – the Sony P800 probably was, though it had a flip-over cover if you wanted 'real buttons' for calls and the like. As with other devices they were stylus rather than finger based. But when the iPhone launched (and was derided by competitors such as Nokia and RIM, who now are far back in the shadow) it changed the face of smart phones overnight. Instead of having hundreds of different models, all with slightly incompatible operating systems and bugs which were never fixed, Apple pioneered the device with a user-updatable OS and no carrier lock-in. Since then, all of the smart phones have been based on the original design of the iPhone.
Tablet computers. The MacBook Air was the first tablet-esque computer, with solid state memory, a non-removable battery – although that still had a keyboard and a fully-featured operating system on board. (These days, the Air is the Pro without a DVD though it remains to be seen how long the DVD will last in the Pro line up.)
Tablet devices. As if the Air wasn't enough, the launch of the iPad – based on the iPhone (and iPod Touch) but with a bigger screen estate – again changed the tech world overnight. New tablet devices are always marketed as the 'iPad killer' in the same way that smart phones are always 'iPhone killers', but in truth, none has really come close.

It's difficult to think of other ways that Steve changed the world. A less tangible, but still none the less important, is the focus on putting the user, rather than the computer, first. All of the Apple products are designed around making it easy for the end user to operate and use, regardless of the programming trickery or delays in getting it Just Right. Whole new conferences on UX exist, and many of them are inspired by Steve's attention to detail and getting it right.

Whatever your personal thoughts are of Apple (and there's usually those at each ends of the spectrum), Steve Jobs certainly caused a major impact over his lifetime and changed the destiny of millions, perhaps billions, of people. His legacy will live on. And, at the end, all of us would like to go knowing that we had made a positive difference on the world.

Git Tip of the Week: Interactive Adding

2011-10-04T09:00:00.000+01:00

This week's Git Tip of the Week is about interactive adding. You can subscribe to the feed if you want to receive new instalments automatically.

This week's tip of the week is about interactive adding. Up until now, whenever you've added a file with git add, it has taken the entire file and added that into the index.

However, it is possible to add only parts of a file to the index; in Git terminology, these are called hunks. A hunk is merely a set of changes to a file, involving lines added (those prefixed with +) and lines removed (those prefixed with -). As well as being a way of textually showing the differences, when it comes to packfiles (which we've covered previously), it uses these hunks to store multiple versions of the same content whilst using significantly less space.

There is an interactive menu that can be brought up with git add --interactive, or git add -i for short. However, most of the options here are not very useful; the one that you'll find yourself using most frequently is the [p]atch command. There is a shorter way of solving this, with the git add --patch command, or just git add -p.

What does this do? Well, it allows you to selectively choose which diffs get added into the index. Although this might not sound particularly useful, it lets you break down changes to one file into multiple smaller changes, provided that you commit them each time. Let's look at an example, with a Person class that we're adding a first name and last name to:


(master) $ git show Person.java | tail -4
@@ -0,0 +1,3 @@
+public class Person {
+
+}
(master) $ git diff
@@ -1,2 +1,17 @@
 public class Person {
+  private String firstName;
+  public String getFirstName() {
+    return firstName;
+  }
+  public void setFirstName(String firstName) {
+    this.firstName = firstName;
+  }

+  private String lastName;
+  public String getLastName() {
+    return lastName;
+  }
+  public void setLastName(String lastName) {
+    this.lastName = lastName;
+  }
 }

Normally, if we do git add, it will put the change for both the firstName and lastName into the index. What if we wanted to segregate these out? Well, we could just take a copy of the file, edit out one change, add it, copy the file back, and then add the second change. But we can use Git to help us here with git add -p:


@@ -1,2 +1,17 @@
 public class Person {
+  private String firstName;
+  public String getFirstName() {
+    return firstName;
+  }
+  public void setFirstName(String firstName) {
+    this.firstName = firstName;
+  }

+  private String lastName;
+  public String getLastName() {
+    return lastName;
+  }
+  public void setLastName(String lastName) {
+    this.lastName = lastName;
+  }
 }
Stage this hunk [y,n,q,a,d,/,e,?]?

The 'stage this hunk' is asking us what we want to do. If we wanted to add it, we'd say 'y' or 'a'. If we didn't, we could say 'n' or 'd'. (The former is 'just this one' whilst the latter is 'and all the rest'.)

What if we wanted to add it piecemeal? Well, there's a [s]plit command we can use, which breaks this hunk down into smaller hunks:


Stage this hunk [y,n,q,a,d,/,e,?]? s
Split into 2 hunks.
@@ -1,2 +1,9 @@
 public class Person {
+  private String firstName;
+  public String getFirstName() {
+    return firstName;
+  }
+  public void setFirstName(String firstName) {
+    this.firstName = firstName;
+  }
Stage this hunk [y,n,q,a,d,/,j,J,g,e,?]? y

@@ -2,2 +9,9 @@
 
+  private String lastName;
+  public String getLastName() {
+    return lastName;
+  }
+  public void setLastName(String lastName) {
+    this.lastName = lastName;
+  }
 }
Stage this hunk [y,n,q,a,d,/,K,g,e,?]? n

What we've done is to add the first 7 lines of the change into the git index, whilst leaving the last 7 outside. If we do a git diff, we'll just see the unstaged difference:


(master) $ git diff
diff --git a/Person.java b/Person.java
index a6d00fd..23dd325 100644
--- a/Person.java
+++ b/Person.java
@@ -7,4 +7,11 @@ public class Person {
     this.firstName = firstName;
   }
 
+  private String lastName;
+  public String getLastName() {
+    return lastName;
+  }
+  public void setLastName(String lastName) {
+    this.lastName = lastName;
+  }
 }

We can see in the Git status that we have a staged change and an unstaged change:


(master) $ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#	modified:   Person.java
#
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#	modified:   Person.java
#

Normally you'd be concerned if you saw this – as if you had forgotten to add something. However, it's exactly the right effect in this case; we've added part of our change, and we have part of the change still to go. From here, we can commit as usual:


(master) $ git commit -m "Added firstname"
[master 2e0d5f8] Added firstname
 1 files changed, 7 insertions(+), 0 deletions(-)
(master) $ git add Person.java
(master) $ git commit -m "Added lastname"
[master 1d09387] Added lastname
 1 files changed, 7 insertions(+), 0 deletions(-)

If we look at the blame for the file, we can see we've committed the changes as separate commits:


(master) $ git blame Person.java | cut -c 1-9,50-80
391f64ef  1) public class Person {
2e0d5f8b  2)   private String firstName;
2e0d5f8b  3)   public String getFirstNam
2e0d5f8b  4)     return firstName;
2e0d5f8b  5)   }
2e0d5f8b  6)   public void setFirstName(
2e0d5f8b  7)     this.firstName = firstN
2e0d5f8b  8)   }
391f64ef  9) 
1d093875 10)   private String lastName;
1d093875 11)   public String getLastName
1d093875 12)     return lastName;
1d093875 13)   }
1d093875 14)   public void setLastName(S
1d093875 15)     this.lastName = lastNam
1d093875 16)   }
391f64ef 17) }

Being able to add changes in parts, rather than in their entirety, is a useful technique when you have subsets of changes that have been made to a file without interening commits.

Come back next week for another instalment in the Git Tip of the Week series.

Removing CVS from the Eclipse Platform

2011-09-27T09:54:00.001+01:00

Wayne has proposed that the Eclipse Foundation makes all the CVS Repositories read-only from December 2012. It was always the intention, once Git was accepted by the Eclipse community, to stop supporting at least one of the version control systems, and CVS has certainly had its day.

This leads to an interesting question; if CVS is no longer used by the Eclipse Foundation, should it even ship CVS by default? CVS has been present in the Eclipse Platform build since its inception, and additionally with the EPP packages for every release train since.

Whilst there have been Subversion projects (Subversive at Eclipse and Subclipse at CollabNet/Tigris), they've never shipped with the default platform; in some cases, due to additional (non-EPL) dependencies that are required.

That's not to say CVS wouldn't be available – it would still be available as an installable feature in Eclipse update sites, much like Mylyn (or Subversive). But perhaps it is time to remove it from the default downloads.

I argue this should be done before Juno is released. There were several projects who wanted to stay on CVS until after Indigo shipped, and to maintain their CVS repositories to support Indigo SR1 and SR2. But we will always have the case where immediately after Eclipse Release Xxx ships, there will be a need to support Eclipse Release XXX SR1 and Eclipse Release XXX SR2.

CVS is an archaic technology and all Eclipse projects must move from it at some point. Making the CVS feature an optional install across the board is a signal to those that still need CVS that their time is drawing near. It wouldn't prevent the build scripts from working, nor would it prevent people from using CVS if they needed to. But it would mean that a signal has been raised.

Delaying this default removal until Juno will bring us into exactly the same discussion, twelve months from now. The only way to remove it will be early in the milestones, and leaving it until after Eclispe Juno SR2 will be too late in the Eclipse Lightyear^* release schedule for that one as well.

I say, remove CVS from the default downloads by the next milestone and replace it with an optionally installable feature.

* No, I have no idea what it will be called. Lightyear would be cool though.

Git Tip of the Week: Git Archive

2011-09-27T09:00:00.000+01:00

This week's Git Tip of the Week is about archives. You can subscribe to the feed if you want to receive new instalments automatically.

If you want to extract the contents of a Git repository, perhaps to make it available for a source download somewhere, then you can of course zip (or tar) up the contents of the repository with a command line tool.

However, there's another way of doing this with a Git repository, using the git archive command. This takes the contents of the current working tree and generates a zip (or tar) file.

One key advantage of using Git to perform the archive rather than a command line tool is to avoid accidentally capturing the (large) .git directory, or any work-in-progress content. For example, if you have just run a build, then zip (tar) will include the content of the build output as well.

Another advantage is that you can extract the content of the repository at an arbitrary revision. Whilst HEAD is used by default, you can put in any tree or tag in the extraction, which makes it useful for being able to generate a source tar ball from a given tag (even if that tree doesn't happen to be the default). For example, let's say we wanted to generate a source bundle from the EGit repository

(master) $ git archive --format tar v1.0.0.201106090707-r | gzip -9 > /tmp/egit-v1.0.0.tgz (master) $ tar tzf /tmp/egit-v1.0.0 | head .eclipse_iplog .gitattributes EGIT_INSTALL LICENSE README SUBMITTING_PATCHES org.eclipse.egit-feature/ org.eclipse.egit-feature/.gitignore org.eclipse.egit-feature/.project org.eclipse.egit-feature/.settings/

This feature is used when browsing the contents of a repository via cgit. It's possible to click on any link (commit or branch) and download a tgz of the repository at the time. All of this is powered by git archive. In fact, you can create an archive from a remote repository, without needing an explicit clone – though it's worth noting that most http repositories don't support this.

(master) $ git archive --format tar -9 --remote ssh://server.org/path/to/git > /tmp/remotearchive.tgz

Finally, it's possible to extract only a subset of files rather than the whole repository. If you wanted to generate only the docs for a project, and they were all present in the docs/ folder, then you could create an archive just containing that with:

(master) $ git archive --format tar -9 HEAD docs > /tmp/docs.tgz

It's fairly common that git describe will be used in conjunction with git archive in creating the name of the output file, and optionally, the global prefix to put in the compressed archive output as well:

(master) $ NAME=project-`git describe` (master) $ git archive --format tar -9 HEAD docs > ${NAME}-docs.tgz

Come back next week for another instalment in the Git Tip of the Week series.

OSGi Community Event 2011 - OSGi, Past Present and Future Keynote

2011-09-26T09:00:00.003+01:00

As I mentioned a couple of weeks ago, last week I gave a keynote at the OSGi Community Event. During the event I unveiled the Omega problem, which is the condition whereby projects are given inexplicable and unmemorable Greek letters, which unnecessarily interferes with getting started with OSGi.

However, that wasn't he only focus of the event. I also highlighted P2 p2's inefficient repository mechanism. Using figures derived from the Eclipse 3.7.0 platform update site, I presented the following chart:

These figures are calculated from the p2 content Jar located at http://download.eclipse.org/eclipse/updates/3.7/R-3.7-201106131736/content.jar. It's worth noting that this file alone is 354818 bytes large (347k) and that now that 3.7.1 has been released, that adds an additional http://download.eclipse.org/eclipse/updates/3.7/R-3.7.1-201109091335/content.jar or, at 361789 bytes (353K), of extra content that has to be downloaded each time you update Eclipse.

Of those files, the majority of the data is worthless. Approximately ³₅ of the content (or 60%) is appendix; present but entirely useless. Only ²₅ of the data contains useful information.

The problems can be boiled down to:

Multiple redundant copies of the full text of the EPL license; in the 3.7.0 release, 101 copies alone (and thus, the 3.7.1 release adds another 101 copies)

Pretty printing of the XML file, when it's supposed to be parsed programmatically

Unnecessary data describing how many child nodes a tree element has, when the tree already has that property

None of these are new problems; it was first reported back in September 2010 (and again in June 2011). So when you're updating to Eclipse 3.7.1, the reason you have to download 700k is because of the way the update mechanism was designed.

Raw data

The data was calculated by stripping the file of unnecessary whitespace (c.f. license/copyright, size), and recompressing the JAR file. Differences between the compressed (JAR) file sizes were reported in bytes. For comparison, the same content files were compressed with GZip to compare against corresponding JAR file waste, not shown above.

Bundle data: 148666 bytes (JAR) – 143672 (GZip)

License/Copyright text: 127388 bytes (JAR) – 127799 (GZip)

Whitespace: 81303 bytes (JAR) – 59611 (GZip)

Size: 4432 bytes (JAR) – 4146 (GZip)

Help me OBR, you're my only hope

The main reason for mentioning these limitations in p2 is the fact that OBR is due out as part of next year's Enterprise OSGi release. It is my firm hope that these issues get fixed, and that the spec mandates decent compression (i.e. using GZip instead of JAR format) combined with mandating the generation of XML files without excessive whitespace.

Anyway, you can read all about it yourself; the slides are available on SlideShare.net, or if you want, you can watch a video of the slides and me talking over them on YouTube. All of the other presentations are available, and you can find my write-up on InfoQ if you want to read more.

The Omega Problem

2011-09-21T09:00:00.000+01:00

What is the Ω problem? It's a problem caused by the use of greek letters and astrological signs to refer to OSGi projects (or ΩSGi projects), thereby hiding the purpose of the project itself behind a name that is likely to be both unfamiliar and eminently forgettable.

There are certainly Java projects which have a good naming convention, such as the Spring project set – where the name of the project gives immediate reference but also is easy to discuss in community forums, project mailing lists and on CVs.

Spring Data

Spring Web Services

Spring LDAP

Spring Social

Spring Batch

Apache Commons Lang
Apache Commons IO

Apache Directory Server

Apache RegExp

Then there are projects whose name you're not even sure how to spell, let alone remember when you're trying to advertise the benefits to muggles. And since they're all so similar, you often end up confusing one for another:

Eclipse Epsilon

Eclipse Gemini

Eclipse Libra

Eclipse Virgo

Eclipse Orion

Eclipse Orbit

Apache Aries

Apache Gogo

Apache Karaf

Apache Sigil

Apache Tuscany

About the only thing you can take away from this list is that both Apache and Eclipse make some weird sounding project names.

The problem is that in order to progress with OSGi in an Enterprise world, you need implementations of some of the Enterprise services. Whilst the Core services are all part of the standard release (e.g. Equinox, Felix), the Enterprise services are extras that are required in order to do anything but are not shipped by default. So, to talk to a database you need the Enterprise JDBC service, the Enterprise JDNI service and quite probably the Enterprise JPA and JTA services as well.

If you buy into Big OSGi Servers – like WebSphere – then it's quite likely that all of these will be present and Just Work™. But if you're trying to kick start a project in Felix or Equinox (or Knopflerfish or ProSys or …) then knowing which the magic set of bundles required is the key.

In an ideal world, these bundles would already be grouped together and work in an out-of-the-box package. The fact that we have so many bundles to pull together – assuming you can remember the names – is one of the reasons why Enterprise OSGi isn't as used as it could be.

For Enterprise OSGi to be successful, it needs to be as easy as when the likes of Tomcat servers provide a JNDI mapping for your database, and can surface that to a Servlet looking up a jdbc/DataSource. Ideally, this wouldn't need code changes to work for non-OSGi code, in order to allow the migration of existing JPA-enabled persistence units and databases configured externally to the bundles that ultimately get wired to them. Anything more complex and developers are likely to shy away from using OSGi and be stuck with Hibernate and Tomcat through ease of use alone.

The Omega Problem was coined at the OSGi Community Event 2011. It uses a greek letter for irony, but also because it fits with the ΩSGi name as well. Finally, is a nod to the footnote in The Last Continent, Page 74, which can be seen here, and is a parody of the Mad Lib Thriller Title genre. There's even a site which helps you make you own.

Git Tip of the Week: Packfiles redux

2011-09-20T09:00:00.000+01:00

This week's Git Tip of the Week is about packfiles. You can subscribe to the feed if you want to receive new instalments automatically.

When Git compresses files, it does so using pack files, which are collections of objects compressed into a single file.

One key difference between Mercurial and Git is the use of the on-disk storage. Mercurial uses a per-file storage model, where all of the history about that file is stored within that one unit. This is similar to CVS' use of ,v files to store versioned information about a specific file, although Mercurial's format is much better tuned. Git on the other hand has a logical object model, and whether those objects are in a compressed pack file or 'loose' on disk makes no difference to Git.

This enables Git to repack objects, and to generate new pack files, subsequent to initially creating them. As long as the object is available, it doesn't really matter where it came from – after all, the unique hash will always point to the same content in the object regardless of where it is loaded from. Of course, since a pack file is immutable after creation, any changes are implemented by the creation of new pack files and then disposing of the old ones.

Since the pack files contain many objects, it is possible to perform delta compression against the objects. In other words, if there are five versions of a file, all largely the same but with minor differences, then an initial version of the file can be stored as is, and just the minor modifications stored afterwards. (These typically include an insert of a range of bytes, a delete of a range or a transpose of a range of bytes.) So although logically the pack file can contain multiple objects, it actually only stores one set of each change.

There are several ways of storing files in the pack file as well. For example, consider a file with the contents 'A' and a subsequent version 'AB'. This can be stored as an A and then +B, or it can be stored as AB and -B. The latter is more performant, since the set of changes involves a deletion of an existing range, so only needs to store the start and length of the deletion occurring. As files grow over time, it also means that the largest file is often stored as-is (typically, the most recent) which gives faster access for that than for previous versions.

Clones and fetches

The other time pack files get created are when you clone or fetch/pull from a Git repository. The 'smart http' protocol actually uses the Git protocol over an HTTP wrapped connection; what happens is there is a bit of back-and-forth in deciding what hashes you have, and what you want. Ultimately, this conversation results in the server knowing what sets of objects to send to the client.

The most efficient mechanism for sending results back to the client is in the form of a pack file. The server knows what to send back, so it builds a pack file containing just those objects. This will involve compressing (both delta compression as well as deflate compression subsequently) and then sending the objects back. You even see this in the chatter that is sent in the message channel when you clone or update the repository:

(master) $ git pull remote: Counting objects: 1177, done. remote: Compressing objects: 100% (352/352), done. remote: Total 1018 (delta 609), reused 787 (delta 390) Receiving objects: 100% (1018/1018), 378.46 KiB | 206 KiB/s, done. Resolving deltas: 100% (609/609), completed with 110 local objects.

The first line is the server counting the number of objects (blobs, trees, commits etc.) that are needed as part of the object set. Roughly this is doing the same kind of work as a git rev-list might do; in other words, it's figuring out the set of objects necessary to send.

The second and third lines is the server saying it's now compressing those objects into a pack file. This is done on the server side before it can start sending any data, as the pack file itself isn't written to in a streaming fashion. In addition, it can write out either full objects or delta objects. (Normally a pack file is self-contained; i.e. it will only create deltas against other objects that are in the same pack file. For clones, the pack file may store references to objects that the client already has, but without copying the object in the pack file itself.)

The fourth line is the client receiving the pack file from the server, once the server has created it. Metadata in the pack file gives an indication of how far it's gone through, which is why you can get an update occurring as the file is received.

The final fifth line is the client assembling the data and closing any of the loops that might be present (i.e. local references to other objects). It will also generate an index file of the objects, since they can be calculated based on the SHA1 hash of the objects themselves.

All of this put together means that when you clone a project for the first time, you end up with a large pack file that contains everything. Subsequent updates can bring down smaller pack files which can build up on the files before. At any time (or when you run git gc manually) these pack files can be reorganised into more appropriate storage units; for example, condensing a number of pack files into one or a few pack files. Since this operation is persistent (i.e. the large pack file will retain its compression behaviour) it may be beneficial to compress a repository periodically with git gc --aggressive.

Summary

The Git pack file is the unit which makes object storage efficient, and also supports the efficient transfer of data between client and server during either an initial clone or a subsequent fetch/pull operation. Although pack files are immutable once created, they can be re-created with more compressed content on an as needed basis.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Objects and Packfiles

2011-09-13T09:00:00.000+01:00

This week's Git Tip of the Week is about objects and packs. You can subscribe to the feed if you want to receive new instalments automatically.

So far, we've talked about commits, trees and objects. We've seen how they bind to the logical object model as well as being represented on disk in the .git/objects directory.

But storing every version of every file in separate files (albeit compressed) is going to be a huge waste of space, right? Yes, there's some sharing of identical content between commits, but Git would hardly be the efficient store that it's known for with storage structure like that.

Pack files

Fortunately, Git has the ability to merge together multiple objects into single files, known as pack files. These are, in essence, multiple objects stored with an efficient delta compression scheme as a single compressed file. You can think of it as akin to a Zip file of multiple objects, which Git can extract efficiently when needed.

Pack files are stored in the .git/objects/pack/ directory. For new projects, this is likely to be empty; what happens is that Git starts off adding all files as non-packed objects, or loose objects. One of the reasons it does this is because as you're working through changes, you're quite likely to re-write various files (blobs) and directories (trees) before you commit. In fact, each time you do a git add to stage a file, you're creating a new object in the loose objects structure.

What happens is that periodically (or on user demand), Git will run a compression on the loose objects. This is triggered either by a git gc request, or automatically after various thresholds have been met. Git will then create the pack file and remove the loose object files.

(master) $ touch empty (master) $ git add empty (master) $ git commit -m "Empty" [master (root-commit) cab1545] Empty 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 empty (master) $ ls .git/objects/ 41 ca e6 info pack (master) $ ls .git/objects/pack/

You may recognise the 'e6' directory as being the prefix of the empty file in Git, which we covered earlier and is identified by e69de29bb2d1d6434b8b29ae775ad8c2e48c5391. However, at this stage, there's no content in the pack directory. What happens if we pack it?

(master) $ git gc Counting objects: 3, done. Writing objects: 100% (3/3), done. Total 3 (delta 0), reused 0 (delta 0) (master) $ ls .git/objects/ info pack (master) $ ls .git/objects/pack/ pack-0c1ff4e31096ebcdb390b30ebe763ae15de650eb.idx pack-0c1ff4e31096ebcdb390b30ebe763ae15de650eb.pack

Where did the objects go? Well, they've been compressed into a single read-only pack file. We can still address them using their hash, even if they're not loose files any more:

(master) $ git cat-file -t e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 blob (master) $ git cat-file -s e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0

The pack file's contents on disk is smaller than the set of files on their own (though in trivial examples like this, there isn't that much difference between them). The pack file is actually made up of two entries; the index (.idx) and the pack (.pack) files. Whilst the latter stores data, the former stores a table-of-contents list of objects contained within the pack itself:

(master) $ hexdump .git/objects/pack/pack-0c1ff4e31096ebcdb390b30ebe763ae15de650eb.idx 0000000 ff 74 4f 63 00 00 00 02 00 00 00 00 00 00 00 00 0000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * 0000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 0000110 00 00 00 01 00 00 00 01 00 00 00 01 00 00 00 01 0000330 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00 02 00003a0 00 00 00 03 00 00 00 03 00 00 00 03 00 00 00 03 0000400 00 00 00 03 00 00 00 03 41 7c 01 c8 79 5a 35 b8 0000410 e8 35 11 3a 85 a5 c0 c1 c7 7f 67 fb ca b1 54 54 0000420 f6 75 88 fb 81 27 96 75 09 38 77 09 7a 75 21 de 0000430 e6 9d e2 9b b2 d1 d6 43 4b 8b 29 ae 77 5a d8 c2 0000440 e4 8c 53 91 7d 73 67 fc 61 d0 d2 e8 6e 76 00 29 0000450 00 00 00 85 00 00 00 0c 00 00 00 b1 f5 5c c2 b5 0000460 8e 21 45 55 02 06 88 64 e9 1b 8b 52 75 c4 46 3d 0000470 57 df 0b ca 6a 8f f3 57 6c d4 97 78 df 30 1d bc 0000480 4d 24 1e a4 0000484

You'll recognise in the hex dump of the index the 'empty object' stored in Git (e69d..5391), along with the tree containing the empty file (417c…67fb).

The purpose of the index file is really a marker to tell Git that the corresponding object is in this pack file. In this case, we've only got one pack file but large repositories will have many such files. The index allows Git to load many small files to determine the answer to “Where are these objects?” so that it can extract them in the most efficient manner.

Summary

Whilst Git stores objects in loose form whilst you work on new changes, it will compress them into pack files to take greater advantage of delta compressions. This happens when you run a git gc or when various thresholds are met automatically. It also explains why Git's storage requirements follow a sawtooth like structure; each time the ramp goes up, it's because new objets are being created, and each time it goes down, it's because a pack has been run and new pack files have been created (along with the corresponding objects being deleted).

Come back next week for another instalment in the Git Tip of the Week series.

Speaking at OSGi Community Event

2011-09-12T09:43:00.001+01:00

I'll be giving a keynote at the OSGi Community Event in Darmstadt, Germany next week, entitled OSGi: Past, Present and Future. In the talk, I'll do a retrospective on how we got where we are today, a view of what is happening in today's modular environment, and a peer into the future of the kind of challenges that OSGi will face in the future.

The full agenda is available, which brings many from the OSGi community together to look at upcoming specifications, such as subsystems, presence in the cloud and even modular EJBs. There's also a lot of information on tools, such as BndTools, The BundleMaker and Eclipse Virgo – as well as best practices such as μServices and OSGi anti-patterns. There are too many to mention individually, so I encourage you to view the full agenda for more details.

The OSGi Community Event is 20^th and 21^st September (i.e., next week) and electronic registration is open until Friday. Hope to see you there!

Git Tip of the Week: Commits

2011-09-06T09:00:00.000+01:00

This week's Git Tip of the Week is about git commit storage. You can subscribe to the feed if you want to receive new instalments automatically.

Last week we looked at the way trees are stored in Git (and the week before how objects are stored in Git). We're now going to see how those are hooked up to commits, which are the basis of branches, tags and the like. Here's an example commit:

(master) $ git cat-file -p HEAD tree 2b61e34a91ca9780ea2f943e72f1a4a022cdd206 parent f44c95384463187acd83ff418ddd9c48659db8dd author Alex Blewitt <alex.blewitt@gmail.com> 1314178977 +0100 committer Alex Blewitt <alex.blewitt@gmail.com> 1314178977 +0100 Another empty (master) $ git rev-parse HEAD ca5fc4f022595972639331adcab40d810b9882a0

It's not going to come as a surprise that a commit is a hashed object, stored in exactly the same mechanisms as blobs and trees are. A commit is a hash of the commit message, with an identifying type and length (as for blobs and trees). In this case, the commit message is 236 bytes long, so we write out commit 236\0 followed by the content, and show the hash:

(master) $ (echo -en "commit 236\0"; git cat-file -p HEAD) | shasum ca5fc4f022595972639331adcab40d810b9882a0 - (master) $ # Or, we can use this to find the size automatically: (master) $ (echo -en "commit $((`git cat-file -p HEAD | wc -c`))\0"; → git cat-file -p HEAD) | shasum ca5fc4f022595972639331adcab40d810b9882a0 -

So, given this knowledge, we can create a new commit all of our own. All we need to do is to refer to a tree (such as d2d6bbd1c25c154fcbb045d66e8a6f9b83587a68 from last time), refer to the HEAD as the parent, and add in some timestamp information.

(master) $ # TIMENOW=`date +%s` (master) $ TIMENOW=1314385772 (master) $ echo -en "tree d2d6bbd1c25c154fcbb045d66e8a6f9b83587a68\n→ parent ca5fc4f022595972639331adcab40d810b9882a0\n→ author Alex Blewitt <alex.blewitt@gmail.com> $TIMENOW +0100\n→ committer Alex Blewitt <alex.blewitt@gmail.com> $TIMENOW +0100\n→ \n→ Manually generated commit" | git hash-object -w --stdin -t commit 195751d8f0822325eb3f234de9c0e720ae53d8ff

We've created our first (manually generated) commit, and it points to the tree from last time. Since all is now well, we should be able to check this commit out:

(master) $ git checkout 195751d8f0822325eb3f234de9c0e720ae53d8ff Note: checking out '195751d8f0822325eb3f234de9c0e720ae53d8ff'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b new_branch_name HEAD is now at 195751d... Manually generated commit ((195751d...)) $ ls anotherEmpty empty void ((195751d...)) apple[bar] $ git diff HEAD^ diff --git a/void b/void new file mode 100644 index 0000000..e69de29

That represents the committed tree which we wrote last time. We can even do diffs between the previous version to find out that the new file is indeed the void that we added previously.

Now we've got the ability to create our own commits, we can take a deeper look into Git's storage structure next time.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Trees

2011-08-30T09:00:00.002+01:00

This week's Git Tip of the Week is about git tree storage. You can subscribe to the feed if you want to receive new instalments automatically.

Last week, we looked at how Git stores objects in the local repository. This week, we're going to look in to how they correspond to directories, or trees.

Git uses a uniform storage model for all of its objects. Each object is identified with its hash, but the type of the object is stored in metadata along with the object. Thus, it's possible to find out from an ID what its type is, as well as its content:

(master) $ # Note: objects from previous (master) $ git cat-file -t e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 blob (master) $ git cat-file -p e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 (master) $ git cat-file -t 8ab686eafeb1f44702738c8b0f24f2567c36da6d blob (master) $ git cat-file -p 8ab686eafeb1f44702738c8b0f24f2567c36da6d Hello, World!

How do these objects get packaged up, so that you can get them in your working directory? Well, blobs are arranged in trees, which corresponds to directories in a directory structure. If we have a directory with a file called empty, we can print out its contents:

(master) $ git ls-tree master . 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 empty

Note that this isn't listing the contents on disk; rather, it's showing you Git's view of the folder. This allows us to list items on different branches (or tags) without needing to check them out first, and in fact, is how hosting sites like GitHub and tools like GitWeb work. The master is simply asking to show us the branch with the same name.

What happens if we add another file, with the same contents?

(master) $ cp empty anotherEmpty (master) $ git add anotherEmpty (master) $ git commit -a [master ca5fc4f] Another empty 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 anotherEmpty (master) $ git ls-tree master . 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 anotherEmpty 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 empty

We have a new entry in the tree, but the blob pointer is to exactly the same object, like a hard link on a UNIX filesystem. As with a hard link, if we change one of the objects, we don't change the contents; rather, we create a new copy (since it has a different hash) and the tree is updated to point to that instead.

How is this tree stored in the repository, though? Well, it turns out that it's another object type, stored in the same mechanism as blobs. You can find out the tree from a commit (or branch) with the ^{tree} suffix:

(master) $ git cat-file -t HEAD^{tree} tree (master) $ git cat-file -p HEAD^{tree} 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 anotherEmpty 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 empty (master) $ git rev-parse HEAD^{tree} 2b61e34a91ca9780ea2f943e72f1a4a022cdd206

The tree represents a directory, containing a mixture of blobs and trees. We can find out what it resolves to using git rev-parse, to determine that this tree is an object which hashes to 2b61e34....

How is this tree created? Well, again, it's a well-formatted object which is hashed through its sha1 value. The object type is tree, and instead of having simple values like the blob, the tree is a set of index values pointing to the objects, along with a mode (typically 100644 for files and 100755 for directories). However, we know the size of the SHA hash, so it doesn't need to be in human-readable numbers; we can serialize it out as bytes. The length works out at 28 bytes per row, plus however many bytes there are in the file name. In our case, we have 28 + "anotherEmpty".length() + 28 + "empty".length(), or 73 bytes in total:

(master) $ echo -en "tree 73\x00→ 100644 anotherEmpty→ \x00→ \xe6\x9d\xe2\x9b\xb2\xd1\xd6\x43\x4b\x8b→ \x29\xae\x77\x5a\xd8\xc2\xe4\x8c\x53\x91→ 100644 empty→ \x00→ \xe6\x9d\xe2\x9b\xb2\xd1\xd6\x43\x4b\x8b→ \x29\xae\x77\x5a\xd8\xc2\xe4\x8c\x53\x91→ " | shasum 2b61e34a91ca9780ea2f943e72f1a4a022cdd206 -

Creating a tree, on the other hand, is a little more tricky. To solve this problem, the git mktree command exists, which can take a git ls-tree formatted stream, and generates a tree object for you. It's a little like the git hash-object from above, but without having to convert the references from the string hash to a sequence of hex characters. In addition, it also ensures that the tree's contents are appropriately sorted, which is a mandatory pre-requisite (in order to support fast retrieval).

(master) $ git ls-tree master . 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 anotherEmpty 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 empty (master) $ git ls-tree master . | git mktree 2b61e34a91ca9780ea2f943e72f1a4a022cdd206

This allows us to easily create a new tree, with a new file in it:

(master) $ echo -en→ " 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\tanotherEmpty\n→ 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\tempty\n→ 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\tvoid\n" | git mktree d2d6bbd1c25c154fcbb045d66e8a6f9b83587a68

We've now got three files in a tree (all the same contents; all empty), but now we can referr to the new tree directly. We can even list it again:

(master) $ git ls-tree d2d6bbd1c25c154fcbb045d66e8a6f9b83587a68 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 anotherEmpty 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 empty 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 void

Although we haven't shown it here, if you wanted to create a tree with other trees (instead of blobs) then they work in exactly the same way; the difference is the word 'blob' is replaced with 'tree', and of course, the object has to point to the right hash.

We've now seen blobs and trees; next week, we'll have a look at how they turn up on branches in the form of commits.

Come back next week for another instalment in the Git Tip of the Week series.

Steve Jobs retires from Apple

2011-08-25T09:37:00.001+01:00

It had to happen sooner or later. Steve Jobs, the founder and most successful CEO that Apple ever had, has stepped down from the firm, with a letter that simply suggested executing the succession plan, and putting Tim Cook into the driving seat.

It's worth noting that he would like to remain as Chairman of the Board and continuing as an Apple employee, a request that was immediately granted. This quote from the board summarises Steve's importance to Apple over the years:

“Steve’s extraordinary vision and leadership saved Apple and guided it to its position as the world’s most innovative and valuable technology company,” said Art Levinson, Chairman of Genentech, on behalf of Apple's Board. “Steve has made countless contributions to Apple’s success, and he has attracted and inspired Apple’s immensely creative employees and world class executive team. In his new role as Chairman of the Board, Steve will continue to serve Apple with his unique insights, creativity and inspiration.”

Not bad from someone who grouped together a few hackers in a garage, and then took it to become the most valuable company in the world. Let's also not forget his role in Pixar and Next on the way, too.

It's easy to be accused of being an Apple Fanboi, but having started my programming career on a NeXT pizza box and learning Objective-C on Nextstep, it has been a career (and a set of products) that I have followed for decades. When OSX first came out, my company bought its first PowerBook G4, and from then, I've had a succession of Macs and MacBooks, from the Cube through to an original iMac in Bondi blue, and the original G5 cheese grater, all of which still work today. Meanwhile, in my garage, I have the remnants of various beige boxes in various states of disconnection whose sole purpose is to offer spares for my printer's memory and if I need to access a floppy for some reason.

I think this is the start of a gradual fade away from Steve; I suspect he might appear "One more time" at the next iPhone event, but thereafter he will be involved less and less. Whether that matters remains unclear; the design genius behind most of the recent Apple products has been Jonny Ive, and many of the hardware advances have been made by dedicated teams across Apple, which I'm sure will continue. But as iOS and OSX draw ever nearer, whether Steve's autocratic design filter will be replaced or whether it will even be needed remains to be seen.

Here's to the crazy ones. Here's to Steve Jobs.

Git Tip of the Week: Objects

2011-08-23T09:00:00.000+01:00

This week's Git Tip of the Week is about git object storage. You can subscribe to the feed if you want to receive new instalments automatically.

This week we'll be taking a bit of a deeper dive into the way that Git stores its objects. We'll look at how they're identified, how they're related, and see why Git handles moves better than other version control systems.

By now, you're familiar with the concept of a commit hash (or just commit) – a 40-character hexadecimal sequence, which can uniquely identify a change log, such as d16085b3b913e5bc5e351c0a7461051e9973629a. But where does this come from?

A git repository is actually just a collection of objects, each identified with their own hash. Whenever you add a file, you get a hash generated on its contents, and this hash is used to uniquely point to that version of a file. For example, if you create an empty file, it will have the hash e69de29bb2d1d6434b8b29ae775ad8c2e48c5391. You can confirm this by adding an empty file to a repository and using git ls-tree to see the contents:

(master) $ touch empty (master) $ git add empty (master) $ git commit -a -m "Empty" [master (root-commit) 4145429] empty 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 empty (master) $ git ls-tree master . 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 empty

What git ls-tree is saying is that the master branch contains a file called empty whose permissions are 100644 (owner read/write, group+other read), and whose hash is e69de29bb2d1d6434b8b29ae775ad8c2e48c5391.

Similarly, if you look in the repository's object store, you'll find that a file .git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 has been created. The directory is split such that there are 256 different top-level names (00-ff) and the name of the hash is concatenated with the parent's directory.

So, how does Git compute this value? Well, it uses SHA1 hash; but the SHA1 of an empty input isn't this value. In fact, Git prefixes the object with "blob ", followed by the length (as a human-readable integer), followed by a NUL character, followed by the contents. So for our case, we have:

$ echo -en "blob 0\0" | shasum $ echo -en "blob 0\0" | openssl dgst -sha1 $ printf "blob 0\0" | shasum $ printf "blob 0\0" | openssl dgst -sha1

All of these print out the same value, e69de29bb2d1d6434b8b29ae775ad8c2e48c5391. Note that the \0 is the escape code for the NUL character; the -e to echo stipulates it should obey the escape. If you get fef5d… then it is interpreting the \0 as two characters, the \ and the 0. (And if you get be21… or b825… then it's adding newline at the end.)

Instead of calculating this format ourselves, we can use git hash-object to calculate a hash – or, with -w, insert an object in our local repository:

(master) $ echo 'Hello, World!' | git hash-object -w --stdin 8ab686eafeb1f44702738c8b0f24f2567c36da6d (master) $ ls .git/objects/8a b686eafeb1f44702738c8b0f24f2567c36da6d (master) $ echo -e 'blob 14\0Hello, World!' | shasum 8ab686eafeb1f44702738c8b0f24f2567c36da6d

This has created a hash of our object (blob 14\0Hello World!\n) and written it into the objects directory under the same name. The contents are compressed with the DEFLATE algorithm; but at the moment, it's not used or referred to anywhere in our tree. Although we don't see it in the working directory, we can see it in the repository itself:

(master) $ git show 8ab686eafeb1f44702738c8b0f24f2567c36da6d Hello, World!

Next time, we'll look at how Git organises objects into directories, and ultimately, commits.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Detached Heads

2011-08-16T09:00:00.000+01:00

This week's Git Tip of the Week is about detached heads. You can subscribe to the feed if you want to receive new instalments automatically.

Today's topic is the subject of detached heads. They're not as bad as they sound, and they don't involve a guillotine in any way.

Heads

Since a git repository is a tree-of-commits, with each commit pointing to its ancestor(s), it is possible to directly address a single commit by means of its commit hash. Not only that, but it's possible to record this hash in a variety of different systems; Twitter, E-mail, Bugzilla etc.

Both Git branches and Git tags are merely links to an item, by commit hash, in the repository. Creating a hundred tags (or branches) is tantamount to creating a hundred pointers, and is one of the reasons why Git is so blazingly fast.

Whilst tags are (generally) immutable, branches are not. Each time a commit is made on a branch, the pointer (reference) is updated to point to the newest commit. Thus, three commits on branch involves three modifications to the branch pointer (as well as the corresponding entries being added into the repository for the content).

These pointers are stored in the .git/refs subdirectories. Tags are stored in .git/refs/tags and branches are stored in .git/refs/heads. If you look at any of the files, you'll find each tag corresponds to a single file, with a 40-character commit hash.

The point for this post is that branches are also known as heads. When you have a master branch, there's a file refs/heads/master which is a pointer to where the current branch is at that point.

Detached Heads

So, if a head is synonymous with a branch, what does that make a detached head? Well, it's simply a commit hash which isn't pointed to by a tag or a branch. So, whenever you have checked out a non-referenced head, you end up with a detached head. Perhaps an example is called for:

$ git init example Initialized empty Git repository in example/.git (master) $ touch file (master) $ git add file (master) $ git commit -m "Initial" [master (root-commit) 123be6a] Initial 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 file (master) $ touch other (master) $ git add other (master) $ git commit -m "Second" [master 5a11d1c] Second 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 other (master) $ git log --oneline 5a11d1c Second 123be6a Initial (master) $ git checkout HEAD^ Note: checking out 'HEAD^'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b new_branch_name HEAD is now at 123be6a... Initial ((123be6a...)) $ git log --oneline 123be6a Initial

What we've done here is to create a repository with two commits, then back out one version. (The HEAD^ checks out HEAD's parent.) The master branch is still pointing to the second commit (5a11d1c) but we're looking at effectively an unnamed commit.

The Git checkout contains a .git/HEAD, which normally points (indirectly) to a ref. In the case of 'detached head' mode, the .git/HEAD file contains the commit hash itself:

((123be6a...)) $ cat .git/HEAD 123be6a76168aca712aea16076e971c23835f8ca ((123be6a...)) git checkout master Previous HEAD position was 123be6a... Initial Switched to branch 'master' (master) $ cat .git/HEAD ref: refs/heads/master

Working with (or on) a detached head isn't a problem. This occurs when you are dealing with bisects, or if you want to simply check out a specific version of a previous commit. There's also nothing to stop you working on this unnamed branch, either; you can keep going and committing as long as you want.

Bear in mind, however, that finding a commit is dependent on what you do with that hash. Typically, you will create a new branch (say, it's a bug fix for a previously released bit of code), or you end up tagging it with a hot fix identifier. There's always the reflag which you can use to get back to your commit if you end up changing out of a detached head. Just remember that the periodic git gc will run, and will clean up commits that aren't referenced (directly or indirectly) by a tag or a branch.

Come back next week for another instalment in the Git Tip of the Week series.

IntelliJ Plugin Development

2011-08-12T09:52:00.002+01:00

My attention recently turned towards IntelliJ plugin development, and since information on the subject is incredibly scarce I thought I'd jot down a few notes here. I'll also try and relate it to plugin development with more familiar environments (Eclipse) for those that are interested in making the switch over.

Firstly, it's worth noting that I'm writing this with IDEA 10.5 as a base. Things can change over time, so if you find this on a Google Search ten years for now you can use common sense to see if it still applies.

IntelliJ IDEA has only (relatively) recently enabled plugin development at all, so things have changed over time. One of the most notable issues is that whilst other products (Eclipse, NetBeans) have focussed on a modular architecture from the beginning, IDEA has always been shipped as a Big Bag of ~~Hurt~~ Jars. So both the plugin development – and the available plugins – reflect the youthfulness of the application, often with contradictory results. For example, updating IDEA itself is handled in a completely different way from updating the plugins.

Installing plugins

When IDEA installs, it creates a plugins directory at the root of the application install, which is used to store the plugins themselves. Each plugin gets its own directory, which uses the id of the plugin (and if the id is not available, the name) by default. Underneath that is a mandated lib directory, and inside that are the plugin JARs themselves.

IDEA\ plugins\ MyPlugin\ lib\ MyPlugin.jar MyPluginExtra.jar AnotherPlugin\ lib\ AnotherPlugin.jar

One common way of installing plugins is to just unzip the folder into the plugins directory. The IDEA build process generates a zip file which contains the name of the project as part of the internal folder structure.

Each plugin has a single class loader, so whether it is shipped as a single JAR or as multiple JARs are a matter of convenience rather than anything else. If you consume downstream libraries then you can simply embed a copy of it in the plugin's lib directory.

The existence of lib is somewhat mystifying; it's as if JetBrains had other thought regarding resources which they could then put to use; or perhaps they were just following web app development structure. Either way, any other content appears to be ignored in the plugins directory.

Update sites

You can't have update sites in IntelliJ IDEA. Well, you can't have sites, but you can have site – hardcoded into each IntelliJ install is a constant called DEFAULT_PLUGINS_HOST, which resolves to http://plugins.intellij.net. This is similar to Eclipse's MarketPlace and is usually the de-facto method of installing plugins into IntelliJ.

If you want to have your own update site, you're out of luck. There's no way to host an IntelliJ update site outside of this mechanism.

There is laughably something called “Enterprise Repository support” in which you can have a list of plugins, identified with a file called updatePlugins.xml. This is nothing of the sort.

Yes, you can start IDEA with a -Didea.host.url= http://path/to/updatePlugins.xml property, and yes, it will download that file. However, when you put any plugin in this file, it will show up a dialog asking you to install all of plugins in that file.

It doesn't integrate with the Plugin model (where people will be expecting it to be) and it forces you to install everything. Oh, and it doesn't work either.

Instead, the update site has two URLs which it uses to convey information back to clients who ask. The first is the list of plugins that are available, which it accesses from http://plugins.intellij.net/plugins/list/. This is an XML file (not even compressed!) which contains a list of every plugin known to man (or to JetBrains, at least). The format looks like:

<plugin-repository> <ff>"Build"</ff> <category name="Build"> <idea-plugin downloads="5483" size="46833" date="1258402969000" url="http://handyedit.com/antdebugger.html"> <name>Ant Debugger</name> <id>com.handyedit.AntDebugger</id> <description><![CDATA[...]]></description> <version>1.0</version> <vendor email="antdebugger@handyedit.com" url="http://handyedit.com/">Alexei Orischenko</vendor> <idea-version min="n/a" max="n/a" until-build="3999"/> <change-notes><![CDATA[...]]></change-notes> </idea-plugin> … </category> … </plugin-repository>

Rather than telling you where you could get all of these plugins from as part of a url attribute, JetBrains makes you go back and get them via another URL, which is http://plugins.intellij.net/pluginManager. This seems to act as a redirector for plugins based on their (plugin) id, and it redirects you to a URL where you can download the file from. For example, the first entry in the IDEA repository above is the Ant Debugger, whose id is com.handyedit.AntDebugger. Add this to the URL, and you get http://plugins.intellij.net/pluginManager?action=download&id=com.handyedit.AntDebugger&build=IC-107.322 , which redirects you to the JAR, and which subsequently installs the content. (The name of this file is derived from the plugin's ID and a unique number which corresponds to the download id in the plugins.intellij.net site – it's not related to the repository at all. Note the without the build parameter, the redirection doesn't work – a primitive form of server-side detection of compatible version numbers.

The one ray of sunshine is that there is a way of overriding this default host; if you run IDEA with -Didea.plugins.host=http://somewhere/else then you can use a different update site to the one compiled within IntelliJ. This can do something slightly more intelligent (like; allowing you to use multiple update sites through careful redirections) but it's a significant piece of missing functionality in IntelliJ IDEA that you have to do this.

It's no wonder that IntelliJ users often just extract the contents of a plugin into their local install, or acquire it through the central update site. It's almost impossible to do otherwise.

Plugin Development

In order to do any kind of plugin development in IntelliJ, you have to configure an SDK. This is non-obvious and almost all of the Google searches turn up with nothing of any use whatsoever. Fresh out of the box, IDEA doesn't even have a Java SDK defined.

You need to set up the Java SDK first, followed by the IDEA SDK (which it can use as itself). Go into the Project Settings and find the Platform Settings, which includes SDKs. There's a small [+] icon at the top of the second column; click it and select Java SDK first; it should auto-detect which JDK you've launched it from, but you can select any JDK on your system. Once that's done, click on the [+] icon again and this time select IDEA SDK. It will prompt you for the JDK you've just defined and select itself as the host. Whilst it looks like it's set everything up, unless you hit 'OK' down at the bottom these changes won't be remembered.

Once you have your SDKs defined, you can create a Plugin Module project. This creates a file called META-INF/plugin.xml which defines the name, id, and version of the plugin. There are also several entries for hooking into IntelliJ:

Plugin metadata (name, version, id)

Application components – global content

Project components – things which are specific to a single project

Actions – things that show up in menus

Extensions – things that plug into extension points

The plugin also has a reference to something which can initialise the component (like a Plugin constructor, or an activator in OSGi/Eclipse). This appears to be called during IDEA startup, but it can block startup until it returns; so if there's long-running operations that you need to do, consider running them in a background thread.

Swing development

My eyes! My eyes! I had really forgotten how bad Swing development can be, until I was forced back into it. Still, it's not as bad as on some platforms but if you're used to IntelliJ you probably see past the UI widget mess in any case. (I'm sure that people will have similar diverging opinions of Eclipse and Xcode; at some level, there's an aspect of familiarity which means you look over warts in your own IDE of choice.)

One advantage of developing IDE plugins for IDEA over that of Eclipse is you get to use Swing. Whilst not a UI win, it is a programming win, since you don't have to worry about dispose() or leaking resources. It's also generally easier to find tutorials on the subject – Eclipse has always been somewhat cryptic with the way that menus and actions are contributed (not helped by the fact that the recommended way has changed every couple of releases).

In any case, it's relatively easy to create tool windows (similar to Eclipse Views, except that they are always present in the window as either minimised entries or showing in the screen somewhere). You get handed a Container in the createToolWindowContent() method (which turns out to be a Swing container) and you can throw what you like in there, wiring it up to the mouse events to trigger actions.

Deploying the plugin is a case of doing “Build → Prepare Plugin for Deployment”. This generates a ZIP in the same folder as your plugin (no option to install it elsewhere, it seems) which contains the above folder structure with your module in. Modularity in Java has actively been harmed by IntelliJ's awful project structure, and is the main cause for pain in leaky implementations for code primarily developed in the IDEA.

If you want to package dependent libraries as well, you can do so by going to the Project Settings and creating a new library, which you then put the JAR into (confusingly, with the 'classes' button). It will then get exported with your plugin when it gets built.

Hello World

With that out of the way, the process involved for a Hello World toolWindow is as follows:

--- 8< --- META-INF/plugin.xml <extensions defaultExtensionNs="com.intellij"> <toolWindow id="HelloWorld" icon="/helloworld.png" anchor="right" factoryClass="com.example.HelloWorld"/> </extensions> --- 8< --- src/com/example/HelloWorld.java public class HelloWorld implements ToolWindowFactory { public void createToolWindowContent(Project project, ToolWindow toolWindow) { Component component = toolWindow.getComponent(); component.getParent().add(new JLabel("Hello, World!")); } }

The only thing I found significantly painful was that the runtime on OSX immediately crashed with a lack of PermGenSpace. Fortunately, "Run → Edit Configurations" whereupon you can add additional Java VM arguments (in this case, -XX:MaxPermSize=256m) which solved that problem. One other annoying feature; each time I added a key press in the -VM parameters field, I got a dialog box flash up asking whether I wanted to accept incoming connections or not. I think this is likely to be an issue with the recent Lion build (along with the error message "2011-08-12 08:34:56.286 java[10515:407] -[NSOpenPanel _setIncludeNewFolderButton:] is deprecated. Please stop calling it." when invoking the Open dialog.

Summary

Once you get past the ugliness that is Swing, and resign yourself to manual installation of plugins, developing for IntelliJ isn't that bad. Most of it uses vanilla Swing operations, though it does helpfully suggest some improved IntelliJ Swing classes in place of the standard Swing ones (though in my uses, it actually performed worse than the standard Swing ones did so I ignored that suggestion).

Once you have a displayable component, it's relatively easy to pick up mouse events and respond to actions. As yet, I have not integrated with the ADT or have processed any source code which is a challenge for another day.

Git Tip of the Week: Searching for Commits and Changes

2011-08-09T09:00:00.000+01:00

This week's Git Tip of the Week is about grepping to find logs. You can subscribe to the feed if you want to receive new instalments automatically.

In last week's post, I talked about how to search a git repositories content with git grep. In this second part, we'll look at how you can search the commit logs alone.

Sometimes it's useful to be able to search through the commit log to find out when a change occurred. Some projects like to embed a bug identifier in the commit message; for example, 1ac0a2 in the EGit project contains a reference to Bug: 324736.

To find this quickly (and without having to refer to an external bug tracking system to get the answer), we can use git log --grep to find the commit message that corresponds to that particular bug.

[EGit] (master)$ git log --oneline --grep="Bug: 324736" 1ac0a29 [historyView] Reveal selected commit on filter change 20c9560 [historyView] Preserve commit selection on filter change

This is obviously useful where you have specific commit messages that include the associated bug tracker, but not so useful if you don't record that information with the commit itself.

Another use-case is when you want to find out when a particular change was introduced. For example, there might be a change in a file with the text table.reveal(c);. Normally, you could use git blame to find out the change; but if the change was added and then subsequently removed, it might not be in the current file to change. It might also have been added elsewhere initially, then copied and pasted elsewhere.

git log has another feature which can be used to search for patterns in deltas. These effectively search the patches rather than the content of the files themselves. For example, if we wanted to search EGit for changes that introduced the above code change, then we could run:

[EGit] (master)$ git log --oneline -Gtable.reveal 1ac0a29 [historyView] Reveal selected commit on filter change 8635f79 Show commit corresponding to selection in commit graph table 390b6b1 Branches and Tags links in commit message viewer dfbdc45 Initial EGit contribution to eclipse.org

Even though table.reveal no longer exists in the current codebase (and as such, the git grep no longer shows this phrase), using git log -G allows us to find when the change was introduced (and when it was removed).

Using git log -G in this way can be used to search history for potentially sensitive information which may have been accidentally committed into the repository. One can even set up period jobs to ensure that the history does not contain information that should not have been committed.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Searching for Patterns

2011-08-02T09:00:00.000+01:00

This week's Git Tip of the Week is about grepping to find content. You can subscribe to the feed if you want to receive new instalments automatically.

Sometimes, when investigating the contents of a repository, it's not always obvious where to find the definition of a function (or method). Clearly, Unix tools such as grep allow you to find the content easily enough, but if there's a large amount of generated data (such as compiled code) simply looking through all files can be time-consuming.

To find a file with a specific content element, you could use something like find . -exec grep pattern '{}' ';'. This will run the grep command on all files in the working directory. However, there's a faster way of achieving this with git grep instead. Let's say we wanted to find the contents of occurrences in the EGit repository of the variable newPushURI. We could use find to achieve this:

EGit (master)$ find . -exec grep newPushURI '{}' ';' URIish newPushURI = uri; newPushURI = newPushURI.setPort(GERRIT_DEFAULT_SSH_PORT); newPushURI = newPushURI.setScheme(Protocol.SSH.getDefaultScheme()); newPushURI = newPushURI.setPort(GERRIT_DEFAULT_SSH_PORT); newPushURI = prependGerritHttpPathPrefix(newPushURI); uriText.setText(newPushURI.toString()); scheme.select(scheme.indexOf(newPushURI.getScheme()));

OK, we've found some occurrences but it doesn't print the names of the files, which isn't too helpful. We could print the file out afterwards if we wanted to but this wouldn't help for files which don't have the match. Or, you could write some kind of script or alias to handle the scan-and-test. It's also not particularly fast:

EGit (master)$ time find . -exec grep newPushURI '{}' ';' > /dev/null real 0m1.605s user 0m0.541s sys 0m0.849s

An alternative is to use git grep to scan the contents of the current working tree:

EGit (master)$ git grep newPushURI org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: URIish newPushURI = uri; org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: newPushURI = newPushURI.setPort(GERRIT_DEFA org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: newPushURI = newPushURI.setScheme(Protocol. org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: newPushURI = newPushURI.setPort(GERRIT_DEFA org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: newPushURI = prependGerritHttpPathPrefix(ne org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: uriText.setText(newPushURI.toString()); org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: scheme.select(scheme.indexOf(newPushURI.getScheme()

Not only does it show which files they are located, it's also an order of magnitude faster:

EGit (master)$ time git grep newPushURI > /dev/null real 0m0.024s user 0m0.014s sys 0m0.033s

The arguments that git grep takes are much the same as grep itself; for example, -l lists files with matches (and -L is files without), -E allows an extended regexp, -i is ignore case and -w is word regexp.

There are also some options which are specific to git. The --no-index example scans the files in the directories, whilst --cached searches blobs in the index.

As well as the current working directory, git grep can also be used to specify a treeish (tag, branch, commit) and subfolders within a repository. If we wanted to look for the regex extension.point in the stable-1.0 branch for matches located in the org.eclipse.egit.core folder, we could do:

EGit] (master)$ git grep extension.point stable-1.0 -- org.eclipse.egit.core stable-1.0:org.eclipse.egit.core/plugin.xml: <extension point="org.eclipse.core.runtime.preferences"> stable-1.0:org.eclipse.egit.core/plugin.xml: <extension point="org.eclipse.team.core.repository">

Finally, it's possible to use -p to print out the name of a function in which a match occurs.

EGit (master)$ git grep -p newPushURI org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java= private void setDefaults(RepositorySelection selection) { org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: URIish newPushURI = uri; org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: newPushURI = newPushURI.setPort(GERRIT_DEFA org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: newPushURI = newPushURI.setScheme(Protocol. org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: newPushURI = newPushURI.setPort(GERRIT_DEFA org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: newPushURI = prependGerritHttpPathPrefix(ne org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: uriText.setText(newPushURI.toString()); org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java: scheme.select(scheme.indexOf(newPushURI.getScheme()

Note that this shows the function annotated by = instead of a : at the end of the name. This can be used to quickly find which functions contain a reference to a given pattern:

EGit] (master)$ git grep -p newPushURI | grep java= org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/clone/GerritConfigurationPage.java= private void setDefaults(RepositorySelection selection) {

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Git Bisect

2011-07-26T09:00:00.000+01:00

This week's Git Tip of the Week is about bisecting to find where a failure was introduced. You can subscribe to the feed if you want to receive new instalments automatically.

When a problem is discovered in an existing system, it's not always clear what change caused the regression. Sometimes it's obvious, but in some cases the only way to find out is to go back and test each stage of the history to find out when the code went from 'good' to 'bad'.

Since this can take a significant amount of time, git bisect can be used to help automate the discovery of the problem. It relies on the ability to determine whether a problem is 'good' or 'bad', and uses a binary sort to discover through the history when the error was introduced. To do so, you use the git bisect start, followed by the git bisect good and git bisect bad commands. Let's use it to find out when "Line two" was added:

# Sample file, adding one line per commit (master) $ git blame test ^b06000a (Alex Blewitt 2011-07-15 08:44:49 +0100 1) Line one 3c30996f (Alex Blewitt 2011-07-15 08:45:04 +0100 2) Line two 4b2a81b8 (Alex Blewitt 2011-07-15 08:45:24 +0100 3) Line three 67664b1e (Alex Blewitt 2011-07-15 08:45:32 +0100 4) Line four 50cc6273 (Alex Blewitt 2011-07-15 08:45:40 +0100 5) Line five (master) $ git bisect start (master|BISECTING) $ git bisect good b06000a (master|BISECTING) $ grep two test Line two (master|BISECTING) $ git bisect bad Bisecting: 1 revision left to test after this (roughly 1 step) [4b2a81b86fcdb0e59981dc5d0ecdef5e6567c71b] Third ((4b2a81b...)|BISECTING) $ grep two test Line two ((4b2a81b...)|BISECTING) $ git bisect bad Bisecting: 0 revisions left to test after this (roughly 0 steps) [3c30996f285905b16f1d1d8fb293372615272ce4] Second ((3c30996...)|BISECTING) $ git diff HEAD^ diff --git a/test b/test index 67c8eb8..b8b933b 100644 --- a/test +++ b/test @@ -1,2 +1,2 @@ Line one - +Line two

So we specified a good revision (b06000a) and a bad revision (50cc6273) and then picked a midpoint between them. This then recurses until only that commit is left, and in this case, it's the one that introduced the 'bug' (Line two). We can even see what we've done after the fact:

((3c30996...)|BISECTING) $ git bisect log git bisect start # good: [b06000a98bd9d88f71356746680404de14b68a89] First git bisect good b06000a98bd9d88f71356746680404de14b68a89 # bad: [50cc627394f5647e16921c30e4a984b4fd76577c] Fifth git bisect bad 50cc627394f5647e16921c30e4a984b4fd76577c # bad: [4b2a81b86fcdb0e59981dc5d0ecdef5e6567c71b] Third git bisect bad 4b2a81b86fcdb0e59981dc5d0ecdef5e6567c71b

Clearly in this case we could have just identified the line with git blame, but it highlights the principle of solving a problem. In the case of doing these steps, we were manually running grep two to find out if there was a problem; this would typically be replaced with a specific test command, such as make or equivalent. But wait! We can do it faster! Firstly, we can specify the starting good/bad endpoints:

((3c30996...)|BISECTING) $ git bisect reset Previous HEAD position was 3c30996... Second Switched to branch 'master' (master) $ git bisect start HEAD HEAD~5 Bisecting: 1 revision left to test after this (roughly 1 step) [4b2a81b86fcdb0e59981dc5d0ecdef5e6567c71b] Third

Secondly, we can run a script which can perform some kind of automated tests. With git bisect run we can execute a script per commit which detects whether the condition is good or bad. This could be some kind of make script, or it could be a custom script to test for a particular condition:

((4b2a81b...)|BISECTING) $ echo 'grep two test && exit 1' > ~/test.sh ((4b2a81b...)|BISECTING) $ echo 'exit 0' >> ~/test.sh ((4b2a81b...)|BISECTING) $ chmod a+x ~/test.sh ((4b2a81b...)|BISECTING) $ git bisect run ~/test.sh running /Users/alex/test.sh Line two Bisecting: 0 revisions left to test after this (roughly 0 steps) [3c30996f285905b16f1d1d8fb293372615272ce4] Second running /Users/alex/test.sh 3c30996f285905b16f1d1d8fb293372615272ce4 is the first bad commit … bisect run success

If the script to test needs to be the same for all runs, it's best to put it outside the repository (which will be reset per bisect operation). It's also possible to have a script which merges in simple fixes or builds with additional parameters (e.g. -DDEBUG).

Come back next week for another instalment in the Git Tip of the Week series.

Full-screen support for Eclipse on OSX Lion

2011-07-24T12:45:00.001+01:00

If you've updated to Lion, you know that the full-screen apps are likely to be a significant impact to applications. Unfortunately, the release of Eclipse Indigo was just before OSX Lion was released, so it wasn't possible to implement full-screen handling for Eclipse applications.

Fortunately, the change is relatively easy. the code is already in place in SWT to make it happen, although not invoked by Eclipse. If you have an SWT window and you want to make it full-screen on OSX, you can do:

NSWindow nswindow = shell.view.window(); nswindow.setCollectionBehavior(1 << 7);

If you want to apply this to your current Eclipse instance, point an update site to http://github.bandlem.com/ where I have put a plugin together which applies this change to existing Eclipse windows. You can then hit the arrow at the top-right of the screen in order to put Eclipse into full-screen mode; to bring it back, move the mouse to the top-right of the screen and the menu will pop back into action – you can click on the fullscreen icon to bring it back as a regular window.

Hopefully this will be added by default in future versions (bug 349148) of Eclipse rather than needing this plugin as a workaround.

Update: This seems to fail on some versions of OSX (or on some JVMs on OSX) with missing osgi.os properties; in addition, the code is compiled against the 64-bit SWT and so fails when running with a 32-bit SWT. These problems will be fixed in the near future.

Update 2: I have pushed a new version which supports both 32-bit and 64-bit Eclipse. It also fixes the empty toolbar at the top of the screen in full-screen mode.

Update 3: This is now available via the Eclipse Marketplace

Update 4: The plugin now supports coming out of fullscreen mode with Escape as well as adding a Window menu item to Toggle Full Screen. Work continues on bug 349148 which will extend the SWT API to do this natively, probably from Eclipse 3.8 onwards as it's adding new API.

Git Tip of the Week: Autocompletion in Shells

2011-07-19T09:00:00.000+01:00

This week's Git Tip of the Week is about autocompleting shell environments. You can subscribe to the feed if you want to receive new instalments automatically.

Although many GUI clients exist for Git, sometimes it's useful to drop down to the shell in order to work with repositories. The examples used in this blog all show the shell equivalents, since it's more to do with concepts than specific implementations.

This post is different, however, in that it's explicitly about working with Git in shells. If you never use shells for developing with Git repositories – or you're using Windows – then you can safely come back next week for something more general.

Most shells are fully featured programming environments whose talent is frequently underused as a means of inspecting the file system. You can build functions, loops, file processing and so on. And most shells have a means of displaying a prompt which in the examples I'm using has just been $. But this can be changed to any other value by suppling a different value for a variable called PS1:

$ export PS1='+++ Redo From Start +++ ' +++ Redo From Start +++ echo Hello World Hello World +++ Redo From Start +++

This isn't particularly exciting; it's just changed the $ for +++ Redo From Start +++. Note that it was specified in quotes to ensure that the trailing space is present in PS1, as otherwise it would blur into the command given.

It turns out that PS1 is evaluated each time, which means we can use a function instead of static text each time we want to use it:

$ d() { date; } $ d Thu 14 Jul 2011 08:34:31 BST $ PS1='$(d) \$ ' Thu 14 Jul 2011 08:34:50 BST $ echo Hello Hello Thu 14 Jul 2011 08:34:51 BST $ echo World World Thu 14 Jul 2011 08:34:52 BST $

Note that each time we hit enter after a command, the function is called again, and you get a different time. (If you see the same time, check you remembered the quotes around the variable as otherwise the function will be evaluated at PS1 assignment time; check it's what you expect with echo $PS1.)

How does that fit in with Git? Well, it turns out there's a great script called git-completion.bash in the contrib directory of your Git installation; if it's not there, you can look take a copy from contrib/completion/git-completion.bash instead.

You can make a copy of the script in your local directory (or just symlink it to the contrib location) – in the examples below, I've copied it as .git-completion.sh in my home directory. You need to source it when your shell starts; the common way (for bash shells) is to add this to your .bashrc script:

$ echo source ~/.git-completion.sh >> .bashrc

The .bashrc is read upon each (non-login) bash shell, though there's a .bash_profile which is used for login shells. To ensure a consistent experience, ensure that your .bash_profile sources your .bashrc and then put changes solely in the .bashrc files:

$ echo source ~/.bashrc >> .bash_profile

Note that on Mac OS X, new Terminal windows and tabs are always considered login shells so this is a necessary step for OS X users.

Showing Git status in the prompt

Having covered the basics, how can we use this to our advantage in Git repositories? Well, the git-completion.sh script defines a function, __git_ps1, which gives you information about what branch you're on, if you're in a Git repository. (Note that you have to start a new shell to ensure you source the completion script):

$ __git_ps1 (master) $ cd /tmp $ __git_ps1 $

This gives us the branch that we're on if we're in a Git repository, and nothing if we're not in a repository. We can use this to set up a prompt accordingly:

$ export PS1='$(__git_ps1) \$ ' (master) $ cd /tmp $

This shows us when we're on the master branch, and doesn't when we're not, but it gives us extra spaces. We can format that by passing a %s (standard printf formatting for a string) to add a space only when necessary:

$ export PS1='$(__git_ps1 "%s ")\$ ' master $ git checkout -b example example $

As well as showing the branch, you can also show whether there are uncommitted changes or whether you are behind/ahead of a remote branch. There are various variables you can set:

GIT_PS1_SHOWDIRTYSTATE=true will show a * if there are unstaged and + if there are staged (but uncommitted) changes

GIT_PS1_SHOWSTASHSTATE=true will show a $ if there are stashed files

GIT_PS1_SHOWTRACKEDFILES=true will show a % if there are tracked files

GIT_PS1_SHOWUPSTREAM=auto will show a = if you are at the same commit, < if you are behind, > if you are ahead and <> if you have diverged from the upstream branch.

Autocompletion

So far, we've talked about the PS1 environment variable and tracking the branch name (along with the shells). However, the completion script also adds in the ability to auto-complete git commands (including arguments). Simply by typing a few characters, you can complete argument names and branch/tag names by hitting the TAB key. This can be useful if you don't know the full options that a command can take (type -- and then hit TAB) or when you're referring to a branch/tag frequently and don't want to type it in all the time.

There's a lot more that you can do to customise your shell prompt; since Git runs quickly, and you can set up a function to do what you want, it's possible to print out the current commit id or even use colours to denote different stages. Similar extensions exist for different shells, such as Steve Losh's extravagant ZSH prompt.

Come back next week for another instalment in the Git Tip of the Week series.

Setting up Google Code with Git

2011-07-18T09:45:00.001+01:00

Since Google Code has started to support Git, I wanted to take the opportunity to reboot the ObjectivEClipse project to consume the Eclipse CDT mirror on GitHub so that I could keep up to date with the main mirror whilst adding support for Objective-C. This post shows you how you can migrate your existing Google Code repositories to Git.

Firstly, you'll need your account details from https://code.google.com/hosting/settings. Since Google Code uses https instead of ssh for its code pushes, we need to know what the password is to use.

For security reasons, I recommend against using your Google Account password. Although it's possible to configure, in essence the password is transmitted in plain text, albeit over an https connection. But if this password gets out, you've lost control to all of your Google Account estate.

Instead, Google Code shows you your GoogleCode password when you visit the URL above:

alex.blewitt@example.com's googlecode.com password: Tm90TXlQVyEK

We can use this password to authenticate instead, and if it gets let out accidentally, like Doctor Who, we can click on the "Regenerate" button and get a new one.

Ordinarily, we wouldn't want to have to remember this each time we use clients. You might be using a GUI tool (like EGit*) which remembers the password for you, but if you want to do things on the command line then using SSH keys to login are widely known but not applicable for https connections.

Instead, we can put the entries in the ~/.netrc file (on Unixes, at least) which will be used for the interop with the server. We can configure it like so:

echo machine code.google.com >> ~/.netrc echo login alex.blewitt@example.com >> ~/.netrc echo password Tm90TXlQVyEK >> ~/.netrc chmod go= ~/.netrc

Note that this applies only to machine connections going to code.google.com, but you might want to configure it for your project's name directly:

echo machine project.googlecode.com >> ~/.netrc echo login alex.blewitt@example.com >> ~/.netrc echo password Tm90TXlQVyEK >> ~/.netrc echo machine wiki.project.googlecode.com >> ~/.netrc echo login alex.blewitt@example.com >> ~/.netrc echo password Tm90TXlQVyEK >> ~/.netrc

Note that the project's wiki is in a different location than the main code for Git and Hg repositories (in SVN, it's part of the same repository as the code).

As an alternative, you can access using the code.google.com host for both the project and the wiki – just suffix the project name with .wiki instead, e.g. for https://code.google.com/p/objectiveclipse/ use https://code.google.com/p/objectiveclipse.wiki/

Note that when you convert your project over from SVN to Git, it will appear that the Wiki content gets deleted. That's because the Wiki server starts serving content from your new Wiki location, which won't be populated with the old content.

Fortunately, migrating the wiki content is easy enough. You can access it via the old SVN site you have, which though it's not visible via the main project's URL, is visible via the direct project repository site:

$ git clone https://wiki.project.googlecode.com/git projectWiki $ svn checkout http://project.googlecode.com/svn/wiki wiki $ cp wiki/*.wiki projectWiki $ cd projectWiki $ git add . $ git commit -a -m "Imported from old SVN wiki" $ git push

You now have your wiki pushed to the repository. Of course, if you wanted to maintain history, then you could instead do a git svn clone instead.

Migrating the main repository (from GitHub to GoogleCode) should be as simple as adding a new remote, and hitting push:

$ cd /path/to/existing/github/clone $ git remote add googlecode https://project.googlecode.com/git $ git push googlecode master:master

Congratulations, you are now using Google Code in conjunction with Git.

* Note to EGit users: there's currently a bug which prevents EGit from using Google Code since the latter reports an invalid value.

Google Code and Git Wikis

2011-07-16T08:25:00.001+01:00

Just an addendum from yesterday's post on Git at Google Code – the wiki pages are backed by a version controlled repository. As a result, if you migrate from one repository format to another, your wiki pages will need to be migrated as well.

The locations are shown on the wiki support page:

If Subversion is used, all wiki pages are stored in the /wiki/ directory of your project's Subversion repository, as files ending in .wiki (other files and subdirectories will be ignored). That means you can use a Subversion client and your favorite text editor to add and edit wiki pages.
If Mercurial is used,the wiki content is stored http://wiki.[projectname].googlecode.com/hg/ where [projectname] is the name of the specific project. For Git, http://wiki.[projectname].googlecode.com/git.

If you have a Google Code project primarily for issue and wiki support, and host your Git content elsewhere, then you need to ensure a seamless transition of your wiki pages to the new Git repository after conversion, as otherwise it appears that all your wiki pages are just deleted.

Git at Google Code

2011-07-15T19:07:00.000+01:00

It's great news that Google Code has finally come into this century and now supports Git as a native repository format. Previously, it just supported Mercurial and SVN as two repository formats.

I've written it up at InfoQ, and it is yet one more reason to switch to Git from archaic repositories such as Subversion and CVS.

Note that the Support FAQ lists a minimum version of Git 1.6.6, which was when the 'smart http' protocol was introduced. It's likely that Google Code, together with GitHub, have abandoned the 'dumb http' protocol for performance reasons. (GitHub switched off dumb http support last month.)

Note that as well as support for Git on Google Code, it also applies to new (and presumably, existing) projects on Eclipse Labs as well.

Time to re-open the ObjectivEClipse project!

Git Tip of the Week: Assigning Blame

2011-07-12T09:00:00.000+01:00

This week's Git Tip of the Week is about playing the blame game. You can subscribe to the feed if you want to receive new instalments automatically.

Most version control systems have the concept of blame, but in a good way. This allows you to see who has made changes to a file, or when the file was last changed by someone. Git has the same feature as well. This can be used to find out what feature(s) were added in a release in a process known as blamestorming.

To find out who changed a file, you can run git blame against a single file, and you get a breakdown of the file, line-by-line, with the change that last affected that line. It also prints out the timestamp and author information as well:

$ git blame file 566a0863 (Alex Blewitt 2011-07-12 09:43:39 +0100 1) First line ed0a7c55 (Alex Blewitt 2011-07-12 09:43:51 +0100 2) Second line 8372b725 (Alex Blewitt 2011-07-12 09:44:06 +0100 3) Third line ed0a7c55 (Alex Blewitt 2011-07-12 09:43:51 +0100 4)

The timestamps and abbreviated commit hashes show the changes were introduced sequentially, but in this contrived example it's easy to see. Since Git has full information about the committer, it can show you the person's name, or the persons e-mail address:

$ git blame -e file 566a0863 ( 2011-07-12 09:43:39 +0100 1) First line ed0a7c55 ( 2011-07-12 09:43:51 +0100 2) Second line 8372b725 ( 2011-07-12 09:44:06 +0100 3) Third line ed0a7c55 ( 2011-07-12 09:43:51 +0100 4)

Sometimes you get extra noise when whitespace differences are introduced into the file. You can use -w to suppress reporting on changes that only affected whitespace.

There are also changes you can show on a subset of ranges as well (like git diff, but with lines annotated with the commit hash). For example:

$ git blame ed0a..566a -- file ^566a086 (Alex Blewitt 2011-07-12 09:43:39 +0100 1) First line

Finally, if you have large files, it can often generate more output than is necessary. You can post-filter the results, but it's more efficient to tell Git which lines you want to see so that it doesn't need to do more work than is necessary finding out. You can specify lines with an explicit line number, (start/end), an offset from the start line (valid for end only) or a regular expression. This can be useful to blame a specific function, if you are following a C-like formatting where the function begins in column zero. This allows us to see the blame of a specific function (that starting with ^bar) and to the next closing brace (ending with ^} after the ^bar):

$ git blame tst.c 6bee4066 (Alex Blewitt 2011-07-12 09:57:54 +0100 1) foo() { 6bee4066 (Alex Blewitt 2011-07-12 09:57:54 +0100 2) // the foo function 6bee4066 (Alex Blewitt 2011-07-12 09:57:54 +0100 3) } 6bee4066 (Alex Blewitt 2011-07-12 09:57:54 +0100 4) 0cdc3645 (Alex Blewitt 2011-07-12 09:58:15 +0100 5) bar() { 0cdc3645 (Alex Blewitt 2011-07-12 09:58:15 +0100 6) // the bar function 0cdc3645 (Alex Blewitt 2011-07-12 09:58:15 +0100 7) } 6bee4066 (Alex Blewitt 2011-07-12 09:57:54 +0100 8) 6bee4066 (Alex Blewitt 2011-07-12 09:57:54 +0100 9) main() { 6bee4066 (Alex Blewitt 2011-07-12 09:57:54 +0100 10) // the main function 6bee4066 (Alex Blewitt 2011-07-12 09:57:54 +0100 11) } $ git blame -L/^bar/,/^}/ tst.c 0cdc3645 (Alex Blewitt 2011-07-12 09:58:15 +0100 5) bar() { 0cdc3645 (Alex Blewitt 2011-07-12 09:58:15 +0100 6) // the bar function 0cdc3645 (Alex Blewitt 2011-07-12 09:58:15 +0100 7) }

Come back next week for another instalment in the Git Tip of the Week series.

Real-time Text for Jabber

2011-07-08T19:57:00.001+01:00

Here's something that might be interesting to those who remember talk (and friends such as ytalk and ntalk); the Real-time Text for Jabber, also known as XEP-0301.

These days, chat systems work by sending a block of text at a time, usually when the person on the other end presses the Enter key. In Google Talk, you sometimes get a notification when someone has started typing; but you don't see anything until they've concluded with what they were trying to say.

Although this lacks immediacy, previous systems had the capability to show people's text as they were typing. These were usually inefficient, sending a network packet for each character pressed – but realistically no more different to a modern remote SSH terminal onto a remote host, which does the same thing.

What seeing someone type does is gives you an idea of a conversation flowing, much like a spoken conversation would. You can interrupt someone (to say that you've heard it, and they needn't spend time going in detail) or ask them to expand on the point that they're making.

The real-time Jabber extension to XMPP has been published as XEP-0301, which means any Jabber client could implement it if it wanted. The way Jabber extensions work is that any Jabber client has a list of supported extensions that it knows about, and if two clients both support it then they are free to use it. This permits (for example) clients to agree on what features they support and so to send real time data (or not). Needless to say, it is also backward compatible and will send a full body after the message has been sent, so in group chats (for example) those clients that can support the real time text can display it, whilst those that don't will simply see the final message.

If the protocol was merely batching up changes and firing them out periodically, then you'd see characters turn up in batches. That wouldn't look natural, and it wouldn't solve the problem of trying to get a conversation to evolve either. Instead, the real time text protocol includes timing information for the characters typed, so that the client on the remote end may receive a single packet of data but then drip feed the characters to the screen at an appropriate rate.

To get an idea of how this looks in practice, take a look at the real-time text demo at the creator's website. This shows the same conversation, shown as it is at the moment (with messages appearing en masse as they are finished), one where the characters are merely batched and flushed each second, and one where the characters are drip fed out. (The example uses an animated GIF to show how it would look, and is fairly effective at getting the point across.)

The status of the real-time text protocol is marked as “Experimental” so it may be a while before it is picked up by other major players such as Google and Apple. However, the XMPP protocol used means that Google Talk could be used to act as the intermediary for two supporting clients. It will be interesting to see if Google is interested in using this for Google Talk on the Web, or whether it is an idea in waiting for the future.

Google selcriC

2011-07-04T09:42:00.001+01:00

Having used Google+ for a few days now, I'm not entirely convinced that Google Circles are tremendously useful. Yes, they're animated in a snazzy fashion, and yes, it's all JavaScript goodness – but from a practical perspective, it's all pretty useless as a way of categorising people.

To be honest, Google Circles is isomorphic to groups in your local address book, with the exception that you aren't holding onto an e-mail address but rather an identity. That permits, for example others to update their contents whilst keeping the same identity throughout; a kind of distributed address book, if you will. Except that unlike Buzz, it's not integrated into your mail contacts.

Buzz screwed up not because it was integrated with your contacts, but because it auto-exposed your contacts to the wide world, based on frequency of mail exchanges. Had Buzz not exposed contacts automatically, but rather used an 'opt-in', like Google Circles, it would have been much better (and essentially what you've got now with Buzz). But ultimately, where we are now with Circles is the same as we are now with Buzz; the circles exist for your categorisation purposes only.

That's useful when (for example) you have a small group of people that you define, such as close family members or members of a local club. But unfortunately for Google+ social relationships aren't reflexive; if you follow someone, it doesn't imply they follow you (or vice versa).

I post about varied topics on my blog. Eclipse used to be a mainstay, but has dropped off over the years. ZFS used to be another before it forked off. These days, I'm most likely to write about Git – but there are always other topics (like personal health updates or IPv6) that I write about occasionally.

I'm pretty sure my blog as it stands correlates exactly with one person's interests – mine. (There can't be that many people who are bothered about my health, and I know my wife has no interest in Eclipse.) That's why the various feeds I offer are useful; it allows someone to take a feed of just the Eclipse related items (which then gets fed to Planet Eclipse). Those interested in my Git Tip of the Week series can follow their own feed, without having to read things that aren't interesting to them.

All of this works because which feeds to consume are a self service. I don't define a list of people and say “Person X is only interested in Topic Y; I'll ensure they get all those items.” Instead, Person X can choose whether to subscribe to Topic Y (or X or Z) on their own terms and time. People's interests change over time; whilst they might be interested in Topic Y to start with, they may gain an interest in Topic X (and cease to be interested in Topic Y) and so adjust subscriptions accordingly. It shouldn't be necessary for me to be in the loop to make that decision happen.

Finally, the followers and following are asymmetric; or to put it another way, the two sets are unlikely to be a complete overlap. There are people I follow who don't follow me; there are people who follow me whom I don't follow, and there are those that follow each other. The same is true of blogs; this post is likely to be read by some of the bloggers that I follow, but equally, it's likely to be read by other bloggers that I don't (or even who don't blog at all).

Asking me to categorise all my followers is a waste both of my time and also of the opportunity for my followers to categorise themselves. I tag each of my posts so that people know what they are related to; this in turn fills the Eclipse feed or the Git Tip of the Week Feed or the OSGi Feed. And those that are (just) interested in OSGi can consume a tailor-made feed, just for them.

Google Circles, on the other hand, is only good for publishing to publicly, or to a small infrequently changing set such as family (family members don't come and go that frequently). And in all honesty, unless a significant proportion of your family members are on Google+ already, there's probably not a lot of point in even that.

What Google Circles needs is a way of defining publicly subscribable circles. I can create a few circles that are of interest to me (Eclipse, OSGi, iOS) but membership of those circles should be public. That way, I can publish something to just the Eclipse circle, and just those interested in Eclipse will see the information, just like they can today with my blog posts. It's no good allowing anyone to follow me but then expect me to have to put them in a circle in order to get a customised feed for them; right now, it's all or nothing.

Git Tip of the Week: Tracking Branches

2011-07-04T09:00:00.000+01:00

This week's Git Tip of the Week is about configuring what happens when you pull. You can subscribe to the feed if you want to receive new instalments automatically.

Last week I wrote about the behaviour of pulling tracked branches; this week, it's worth taking a dive in to find out what a tracked branch is.

When you initially use Git, you learn that to update items from master involves a git pull (or git fetch). Both of these reach out to the remote repository and get content that you're interested in, with the git pull variant doing either a merge or a rebase as appropriate.

But how does Git know what to pull when you invoke git pull? Where should it pull it from? What makes a branch you have checked out locally differ from one you have pulled from a remote repository?

Remotes and Refspecs

Firstly, a (local) git repository can have many remotes. Each remote is a name of a repository on a remote end, which corresponds to a URL and a refspec. (In fact, remotes can have a second URL; one is used for fetching, whilst the other is used for pushing – this is used to permit anonymous fetches but authenticated pushes.) You need to specify, when fetching and pulling, what repository you're talking about. For remote repositories, this will default to origin if not specified

You can specify what the refspec is when interacting with a remote repository. This is the set of branches that will be updated if you interact with that repository. This is normally of the form refs/heads/master:refs/remotes/origin/master, where refs/heads are the pointers to your local branches, and refs/remotes are the remote branches.

An optional + prefix on fetch refspecs indicates whether or not to fetch non fast-forward commits automatically. And whilst you can't have partial wildcards (like refs/for/qa*) you can have sub paths (like refs/for/qa/*). You can also use the reference HEAD to refer to the commit that the current branch is on as a source for the refspec, which can be useful for pushes.

Tracked branches

However, each branch also has the concept of what it is tracking. As well as the branche(es) that will be affected by a fetch/pull/push, tracking says which branch is upstream of which.

Normally, branches checked out of a remote repository are automatically set up as tracking branches. If you check out EGit, you'll get a master branch that tracks refs/remotes/egit/master (or origin if you didn't specify a default repository identifier). Any changes you pull into your master come (by default) from EGit's master.

However, what if you wanted to spin off another branch for experimental purposes, and keep that updated? If you do git checkout -b experimental, it diverges from your local master at that point in time. You either need to pull changes through master and then rebase, or remember where your merge point was.

Instead, you can set up your experimental branch to track another one. This means you can fetch and pull, as if you were pulling from a remote repository, and consume changes from the ongoing branch moves. This is useful if you have a long-running UAT branch which needs to be refreshed periodically from a moving target; setting it up as a tracked branch means that the only thing you need to do is git pull, and you're up to date.

So, how do you set up a branch for tracking? Well, when you check out a branch from a remote master, it gets set up automatically. In fact, all a tracked branch is is one that's explicitly mentioned in the .git/config file, since it lists what its remote is and where to merge from:

$ git clone upstream clone Cloning into clone... done. $ cd clone $ tail .git/config [branch "master"] remote = origin merge = refs/heads/master

The way to read this is that master is a local branch, which tracks refs/heads/master on the remote origin. Any pulls that happen on master will result in a merge (or rebase) from refs/heads/master.

What if we wanted to set up our experimental branch? If we just do git checkout -b experimental, it won't be tracked:

$ git checkout -b experimental Switched to a new branch 'experimental' $ grep branch .git/config [branch "master"]

We can flag it as tracked using the --track option of git checkout (or its shorter -t alias):

$ git checkout master Switched to branch 'master' $ git branch -d experimental Deleted branch experimental (was 4a3fa88). $ git checkout --track -b experimental $ tail -3 .git/config [branch "experimental"] remote = . merge = refs/heads/master $ git pull From . * branch master -> FETCH_HEAD Already up-to-date.

Hang on, what's the remote = . doing in here? Well, that's a special short hand meaning this repository, much like it means this directory in filesystem access. What we have here is experimental tracking master, and not origin/master; in other words, it's a local branch tracking another local branch. There are times when this is useful, but what if you want to track the remote one directly instead of having to pull through a local copy?

$ git checkout master Switched to branch 'master' $ git branch -d experimental Deleted branch experimental (was 4a3fa88). $ git checkout -b experimental origin/master Branch experimental set up to track remote branch master from origin. Switched to a new branch 'experimental' $ tail -3 .git/config [branch "experimental"] remote = origin merge = refs/heads/master

Now we have a branch experimental which is tracking origin/master. When we have an update in the upstream repository, and do a pull, we see it updating both master, and also experimental:

$ git pull remote: Counting objects: 3, done. remote: Compressing objects: 100% (2/2), done. remote: Total 2 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (2/2), done. From upstream 4a3fa88..55eb534 master -> origin/master Updating 4a3fa88..55eb534 Fast-forward 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 third

The update shows origin/master being updated to the new value. The subsequent step is the updating and fast-forward of the local experimental branch. But what of the local master branch?

$ git log --oneline experimental 55eb534 Third 4a3fa88 Second ff8536c Start $ git log --oneline master 4a3fa88 Second ff8536c Start

So although both experimental and master are tracking the same upstream branch, they can be updated and processed independently. This is useful when you want to advance the state of one branch (perhaps for experimentation purposes) but don't want to change the local state of a branch.

It's possible to add upstream tracking information to an existing local branch after the fact, in recent versions of git. If we'd checked out the experimental branch as in the first step, and didn't want to delete/re-create it (perhaps because we'd made some local changes) then you can add it afterwards:

$ git checkout master Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. $ git checkout -b experimental2 $ git checkout -b experimental2 Switched to a new branch 'experimental2' $ git branch --set-upstream experimental2 origin/master Branch experimental2 set up to track remote branch master from origin. $ tail -3 .git/config [branch "experimental2"] remote = origin merge = refs/heads/master

So even if you have existing branches, it's possible to wire them up to be tracking branches after the fact. You can also use this if you want to change which branch you're tracking (say, swapping a local branch for a remote one or vice versa) by re-running the command.

It's also worth mentioning that there is a --no-track option of git checkout, which can be used to prevent the tracking of branches upon checkout if that's desired. This is sometimes useful if you are consuming a feature or bugfix branch and you don't want/need to pull from it in the future.

Lastly, all of this is configured with the branch.autosetupmerge config option. If this option is false, then branches are never tracked by default. If the option is true, then branches are tracked if they are remote, and not tracked if they are local. If the option is always, then branches are always set up as tracked branches, regardless of whether they are local or remote. These effectively specify the defaults, but they can be overridden on a branch-by-branch basis using the --no-track and --track command line flags of the git checkout or git branch commands.

Come back next week for another instalment in the Git Tip of the Week series.

Google Plus

2011-06-30T09:50:00.001+01:00

Now that Plus has launched, it's interesting to compare to what I wrote about my predictions for what circles could be in the light of (at the time) Twitter's only mistake – the dreaded dickbar. At the time, I wrote:

So, what could Google innovate on? Well, to be a success (over and above Buzz), it has to have:

An open API, probably backed with OpenID/OAuth/OAuth2 to permit additional clients being developed

A way of *not* having any and all messages delivered to your e-mail, especially when it's from yourself

Have a way of associating groups of people and defining groups by role, e.g. “public”, “friends”, “work colleagues”, “family” and have a way of switching visibility on a per-message basis

A way of uploading video/pictures along with text messages, probably from portable mobile devices with cameras

On top of all this, Google almost certainly needs to have a couple of native clients, not just a web client, to demonstrate that anyone can join in.

So far, it's early days to expect there to be a public API so the first point is still out for the jury. The second point has been a facepalm episode – every time someone comments on your post, you get an e-mail notifying you that something has happened. It's like Douglas Adam's button on Disaster Area's stunt ship, which when you press it lights up a light saying “Do not press this button again.” Fortunately, theres various settings that you can change, including email delivery, which you can adjust to your liking afterwards (read: none).

On the plus side, they do appear to have created the circles aspect I alluded to, with friends and family (although they call friends ‘acquaintances’, which might be a sensible rewording of the term). And it looks like it may be integrated with mobile devices to the extent where picture uploads will become possible – there's an Android client already, with an iOS client in the works.

Limited invites

One of the things that limited invites does is increase scarceness, which in turn, pushes up the value of something. At the moment, Google+ is new, and people are wanting to get on board.

But the power of a social network is for it to grow at its own speed. Indeed, the more people are in your social network, the more likely you are to use it (and conversely, the less people you know, the less you are likely to use it). Wave died a death because people though “What is this for?” when there was no-one to share it with. And with waves of people (wanting to) join Google+, but no-one there, it means you end up with a much more restricted set of people to talk with. As Martin Gratzer said on twitter: “Ok, this is how Google+ looks like. First impression, nice look & feel. Facebook clone with Buzz integration, video chat minus 800 friends.”

This may just be growing pains. According to Google, they opened then closed invites (source). But it's likely the early adopters who are interested in joining in the first place; similarly, those early adopters are often the most influential (technically) with their friends and often have connected networks in the first place. By creating artificial scarceness, the net effect may be to limit the number of fully connected networks that grow in the early stages, leaving the early adopters to walk away and thus impact the growth.

One thing's for sure. Both Buzz and Wave failed in the adoption stakes. By addressing the privacy issues of Facebook, coupled with the ease of use for Twitter and native clients, Google+ may be a success hit – even if you can't search for it by name. And, if you want to follow me I'm alblue on Google+.

Git Tip of the Week: Pulling and Rebasing

2011-06-23T09:00:00.000+01:00

This week's Git Tip of the Week is about configuring what happens when you pull. You can subscribe to the feed if you want to receive new instalments automatically.

Pulling

Back in March, I wrote about pushing and pulling as an introduction to getting data from a remote Git server. Now that we've talked about rebasing (twice), we can talk about the different pull strategies.

Recall that git fetch merely makes the changes available in your local repository; it doesn't affect the branch(es) that you are on. On the other hand, git pull does affect the branches that you are on as it tries to include changes from upstream.

By default, Git will attempt to do a merge whenever you pull changes. Here's what it looks like when you do a merge:

$ git log --oneline master e1c1744 Third 36c3a20 Second c66b73b First # Take a branch from a previous point in history $ git checkout -b feature-merge 36c3a20 Switched to a new branch 'feature-merge' # Set it up so I can pull from master $ git branch --set-upstream feature-merge master Branch feature-merge set up to track local branch master. $ git log --oneline feature-merge 36c3a20 Second c66b73b First # Add another file so we diverge $ touch feature-merge.txt $ git add feature-merge.txt $ git commit -m "Feature-Merge" [feature-merge a16a3db] Feature-Merge 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 feature-merge.txt # Now, let's see what happens when we pull $ git pull From . * branch master -> FETCH_HEAD Merge made by recursive. 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 c $ git log --oneline --graph * 23b50a0 Merge branch 'master' into feature-merge |\ | * e1c1744 Third * | a16a3db Feature-Merge |/ * 36c3a20 Second * c66b73b First

Here, we've invoked the default operation which is to do a merge. When we're behind master with local changes, and we do a pull, it automatically sets up a merge node, as shown by the graph above. (Normally the branch would be a remote one; however, I'm showing a tracked local branch for convenience.)

However, this is configurable to do a rebase instead. This configuration is done on a branch-by-branch basis. Let's create a new branch to experiment with this feature:

$ git checkout -b feature-rebase 36c3a20 Switched to a new branch 'feature-rebase' $ git branch --set-upstream feature-rebase master Branch feature-rebase set up to track local branch master. $ git log --oneline feature-rebase 36c3a20 Second c66b73b First # Add another file so we diverge $ touch feature-rebase.txt $ git add feature-rebase.txt $ git commit -m "Feature-Rebase" [feature-rebase 62855ee] Feature-Rebase 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 feature-rebase.txt $ git log --oneline 62855ee Feature-Rebase 36c3a20 Second c66b73b First # Configure the branch for rebase operations $ git config branch.feature-rebase.rebase true # Now, let's see what happens when we pull $ git pull From . * branch master -> FETCH_HEAD First, rewinding head to replay your work on top of it... Applying: Feature-Rebase $ git log --oneline * ff0ecc2 Feature-Rebase * e1c1744 Third * 36c3a20 Second * c66b73b First

What's happened here is that instead of creating a merge node, the pull operation resulted in a rebase of the underlying branch against the current master. This is dangerous (in the sense that a rebase is dangerous and changes history) but provided that you're doing this on local branches (those not pushed to the repository yet) then you may find this acceptable.

This configuration has to be done on a branch-by-branch basis. If your preferred way of working is to always enable this option, then there is a configuration item which can help:

$ git config branch.autosetuprebase always

If this is configured (either in a repository or globally with the --global setting) then whenever you create a new branch, it will automatically add branch.name.rebase=true for you.

Come back next week for another instalment in the Git Tip of the Week series.

Eclipse Indigo Released

2011-06-22T14:00:00.000+01:00

Today marks the release of Eclipse Indigo, the combined set of 62 projects and 46 million lines of code. I've written up the details at InfoQ so I won't bother going into them again here.

What is worth taking away, though, is not just the fact that Eclipse is a successful Foundation (which it is), or the fact that Eclipse projects generally make great software (which they do), but the fact that co-ordination in open source projects is not only possible, but it's also predictable.

The Releases on Wikipedia lists Eclipse releases by the Eclipse Foundation, which was created back in 2004, but in fact the Eclipse project goes back to a public release in November 2001. Before then, Eclipse was the embryonic foundation of WebSphere Studio, an HTML editor for WebSphere's servlet engine (at the time, called just WebSphere) to complement Visual Age for Java's IDE.

Since VAJ wasn't sensibly able to handle non-Java files, WebSphere Studio grew out of a desire to have first plain HTML, then later mixed HTML, JSP and Servlet content. VAJ didn't have a large following at the time (though, like Visual Age for Smalltalk users, it did have its loyal userbase). At the time, products like Symantic Café and Borland's JBuilder had larger followings, in part due to VAJ's organisation at the logical (class/method) instead of physical (file), and in part due to other products being cheaper.

Eclipse today still has the “Java Browsing” perspective, modelled on VAJ's logical view of the world; it is telling that most Eclipse users automatically use the “Java” perspective, which provides a tree-view onto the physical representation and thus needs tools like Mylyn to reduce the clutter. Perhaps this predisposition comes from the fact that Windows and Linux file managers often only have a tree-based view of the world, whereas other operating systems like Nextstep and OSX have column views which can show more information in a more condensed way.

NetBeans was open-sourced in 2000, and IntelliJ was released in early 2001. Eclipse 1.0 was released in November 2001, making it the newcomer in the IDE scene. At the time, it was buggy (and had a tendency to not notice when files were changed outside of the IDE, requiring manual refreshes in order to open them) but none the less, its approach to modular extension was ahead of IntelliJ (which wouldn't be open-sourced until much later) and really kick-started the thoughts of modular software development. Eclipse 2.0 followed seven months later in June 2002.

Since then, the Eclipse project – and later, the simultaneous releases which congregate around it – have followed a yearly release pattern. Whilst 2.1 was released in March (in time for EclipseCon), all the other releases have been in June and have kept to a fixed schedule, including milestone builds and release candidates (often with an M5a or M6a for good measure). Eclipse releases have been clockwork:

Eclipse 1.0 – 7 November 2001 (Win32/Linux32 Motif)

Eclipse 2.0 – 27 June 2002 (Linux32 Motif + GTK, and Solaris/QNX/AIX)

Eclipse 2.1 – 27 March 2003 (OSX first version)

Eclipse 3.0 – 25 June 2004 (first OSGi version)

Eclipse 3.1 – 27 June 2005

Eclipse 3.2 – 29 June 2006 (Callisto)

Eclipse 3.3 – 25 June 2007 (Europa)

Eclipse 3.4 – 17 June 2008 (Ganymede)

Eclipse 3.5 – 11 June 2009 (Galileo)

Eclipse 3.6 – 8 June 2010 (Helios)

Eclipse 3.7 – 22 June 2011 (Indigo)

Whilst we're not quite at the 10 year anniversary of Eclipse (which will be in November this year), the fact is that over the last ten years we have had one release per year, and that since Eclipse 3.0 – when the Eclipse runtime transitioned to OSGi – that release has been predictable in mid to late June, often known about a year in advance. We'll probably be having the same conversation on the 20th June 2012.

Incidentally, the switch to OSGi back in 2004 is worth calling out. A brilliant decision, often not understood or liked at the time, OSGi has come to be a bedrock of not just Eclipse but many other servers and systems in the years since. The wider use of the OSGi service model has been percolating ever since, leading to tools like E4 and a gradual drift away from the Eclipse registry. Not only that, it has made OSGi popular as a standalone technology, with developers moving on from just building Eclipse plugins to designing fully-fledged OSGi systems. At least one of the lasting legacies of the Eclipse platform has been the certification that OSGi is a trusted module system.

My involvement with Eclipse has spanned over the decade as well, with experiences in pre-release WebSphere software following through to testing for the OSX release in 2.1 and beyond. My oldest public bug report dates back to Eclipse 2.0 in November 2002, although that's just one of the 570 bugs I've filed over the past 7½ years, a rate of around 75/year – or to put it another way, three bugs every two weeks for the last 380 weeks.

Hopefully by reporting, and in some cases patching, I've been able to help make the Eclipse platform and ecosystem a better place. I certainly hope my blogging and articles on sites like InfoQ and EclipseZone have come in useful to others over the years. I know that there are those who think my bug reporting style (and ties) are brash; but none of it has been driven by “this bug is affecting me, fix it!” but rather “this could impact other people.”

In fact, for most of the last five years, I've not been working in Java or using Eclipse in a work environment; it's all been just a hobby. The only times I've been to EclipseCon were self funded; and the last two EclipseCons I've covered remotely.

The last ten years have had their ups and downs, both technologically and personally. But I'm sure that if I'm still around ten years hence, I'll be writing about my continued involvement in the Eclipse platform for the last two decades.

Git Tip of the Week: EGit

2011-06-21T09:00:00.001+01:00

This week's Git Tip of the Week is about using Eclipse with EGit. You can subscribe to the feed if you want to receive new instalments automatically.

(E)Git at Eclipse - a history

This post is about using EGit in Eclipse; if you are uninterested in Eclipse as a platform, feel free to come back next week.

EGit, and its library layer JGit, have been in development for a long time and this week will be shipped as version 1.0 with the Eclipse Indigo simultaneous release. This is the first time anything other than CVS has been shipped by default out of the box for an Eclipse download.

As well as being good news for Git, it's also great news for Eclipse. The transition towards DVCS has been a long road and yet EGit is just the beginning.

Originally, Eclipse shipped with just CVS, but whilst additional plugins were able to be used to access Subversion, none of them were shipped with the default package due to licensing issues. It also didn't help that there were two competing Subversion projects, Subclipse and Subversive, both of which needed additional binaries in order to work. (It really didn't help Subclipse's case that it didn't ship a pre-packaged OSX client, just as OSX was taking off as the de-facto development and conference-touting laptop.) Even now, Subclipse doesn't ship with drivers for 64-bit Windows as this becomes a more common platform.

The nail in the coffin for Eclipse's subversion usage ultimately was Git and GitHub. Whatever your personal preferences of (D)VCS are, there can be no doubt that GitHub has transformed the industry in adopting DVCS, and more specifically, Git. Not only that, but with Eclipse being swayed by Git, and Apache's read-only Git mirrors, it's clear that the majority of foundations are leaning towards Git support. (Google Code remains the outlier with Hg; in part, due to its implementation in Python – but in future, Google Code will support Git as well).

EGit is the set of Eclipse UI libraries that integrate with the Team providers, whilst it relies on a re-implementation of the core Git libraries in Java, JGit. As well as powering EGit, JGit also powers the runtime inside Gerrit, a popular review tool (which I've written about before), as well as a port to Android in the form of Agit.

One of JGit's advantages is that the Git on-disk format is both well documented and well understood. In fact, it's this on-disk format that has resulted in most of the additional libraries and tools being made available; instead of having to call out to a specific blessed library (like SVN and Hg do), a Git client is capable of creating its own tree from content in an existing Git repository. Pretty much every Git tool reads, processes and generates trees of objects and references to those.

Using EGit

Most of the Eclipse Indigo release packages already have EGit in place; but if not, it's a simple operation to go to the EGit entry on Eclipse Marketplace to download it into your client. Whilst Helios shipped with a 0.12 version in service release 2, Indigo ships with version 1.0 – although that can also be installed in a Helios runtime if you want to add the update site to your runtime.

There's a lot of good documentation on the EGit wiki that includes my Git for Eclipse Users, which gives a good background in Git for those who aren't aware. But there's also lots of screenshots to show you how to get things done as well.

Unlike CVS/SVN repositories, a Git repository will exist on your machine and your project will be hosted out of that. It's not recommended to create Git repositories under the workspace directory – Eclipse doesn't tend to like that. When you create a new Git repository (from the Git Repositories view, or from a newly shared project), it will default to putting it in ~/git. you can then create projects underneath that location, or share a project and choose that Git repository.

The other aspect to note is all projects in a Git repository share the same branch. If you have two projects, both on master, then if you switch branch on one project (say, to release37) then both projects branches will be changed. If you don't want that, you can create a clone of the Git repository locally, and remap the project to the local clone. However, this is likely to cause confusion if you do this normally.

Once you've shared your project (or imported it from a previously created git repository) then using it is much like any other team provider in Eclipse – you can commit, merge, branch, compare etc. as normal.

The only significant difference is that commit operations are local (i.e. affect your own local repository) rather than remote. So others won't see your changes unless you push them up. Similarly, if you want to get changes from others, you need to pull them down.

If you're using a Gerrit workflow, then it's worth enabling the (undocumented) gerrit.createchangeid flag, which enables the automatic creation of the Change-Id field. This is set if you clone from a Gerrit repository in the first place, but doesn't have an option to set it up afterwards.

If you're not using Gerrit, then having a clone with a pull policy set to 'rebase' is the most common one you'll find for emulation of the traditional workflows. You can configure this when cloning a project initially, but if not, you can set git config branch.autosetuprebase always, followed by configuring it for the branch(es) you have checked out with git config branch.name.rebase true.

The migration begins ...

Many projects have already moved to Git at Eclipse – the full list is at http://git.eclipse.org/c/, and major players like CDT and EclipseRT are in the process of joining other projects, like Virgo and ECF, who have already made the transition.

There are still some rough edges, both in EGit and also in the Git contributions policy, which are likely to be overcome by the end of the year. But having support out of the box for Git within all Eclipse runtimes, and the fact that projects are stepping up to move over to Git, means that the tooling will be under close scrutiny and improve over the coming months and years.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Rebasing Revisited

2011-06-14T09:00:00.000+01:00

This week's Git Tip of the Week is about using rebase to move between branches. You can subscribe to the feed if you want to receive new instalments automatically.

Interactive Rebasing

In my last rebasing post, I discussed the concept of rebasing, the ability to create new history by replaying past commits in a different order. In fact, most of that post turned out to be discussing interactive rebasing, which allows you to change the order of commits, squash two (or more) commits into one and even remove commits from the history.

However, rebasing also plays another key part in the way developers frequently interact with git, by moving a branch in its entirety forward to a new point.

Rebasing branches

Let's say you've been working on a feature branched off a known point, and you now want to commit it to the repository. Let's assume the history looks like:

A → B → C → 1 → 2 → 3

where C was the point of master at the time of starting the feature branch. If the remote branch has moved on since then (say, to “F”), we have the option of creating a merge node “G”, or moving our feature branch forwards, based on where the remote is now:

A → B → C → D → E → F → G // Merge node \→ 1 → 2 → 3 →/ A → B → C &rarr D → E → F ⇒ 1 → 2 → 3

Some developers or large development teams have a preference to create merge nodes, not only as a way of avoiding problems (the merge node can be tested) but also as a way of documenting where it came from in the first place. Some teams even create merge nodes when they're not needed (such as the git pull --no-ff option).

Other developers like trying to keep the number of merge nodes to a minimum, to try (as far as is possible) to have a linear history in the repository.

Either way, although there's not a right answer, Git allows you to do both depending on what the right answer is for you, at that particular point.

Rebase example

If we wanted to achieve the second history, we could write an interactive rebase script which picked first changes D, E and F, followed by 1, 2, and 3. However, doing this manually would be error prone, particularly if the branch has moved on more than a few commits.

That's where git rebase onto comes into play.

If the above branches were named master (for A..G) and feature (for 1..3), we can transplant feature forwards with:

$ git rebase master feature First, rewinding head to replay your work on top of it... Applying: 1 Applying: 2 Applying: 3

This takes the set of changes in feature that aren't in master, applies them to where master is now, and then calls that the new feature branch.

The second argument can be optimised away if we're already on the feature branch:

$ git branch * feature master $ git rebase master First, rewinding head to replay your work on top of it... Applying: 1 Applying: 2 Applying: 3 72366d5 3 8da923e 2 a4fe060 1 65521f9 F 62f8b88 E e210d31 D 94c037d C 668b955 B a34bd14 A

One nice property of git rebase is that it won't duplicate deltas that are the same. So if we need a particular change for a bugfix, it won't re-apply that:

$ git checkout master Switched to branch 'master' $ git cherry-pick 8da923e [master cf6d845] 2 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 2 $ git log --oneline cf6d845 2 65521f9 F 62f8b88 E ... $ git rebase master feature First, rewinding head to replay your work on top of it... Applying: 1 Applying: 3 $ git log --oneline ba0ce72 3 6979b71 1 cf6d845 2 65521f9 F 62f8b88 E ...

In this case, we cherry-picked 2 from the branch, and then applied the remaining feature changes afterwards. This is a quick way of doing an interactive rebase if you know you just want to pull a single change but don't want to fire up an editor.

Rebasing onto

So where does --onto come into play? This is used for some serious tree surgery; it basically allows you to take a set of commits, and transplant them onto a different node.

In our example, let's say we had created two feature branches,feature1 and feature2, which were logically independent, but we'd ended up developing it so that feature2 branched off feature1. We'd have something that looked like:

Ma → Mb → Mc ← Master \ → F1a → F1b ← Feature1 \ → F2a → F2b ← Feature2

Now let's say that we want to push Feature2 to Master, but we don't want to push Feature1 to Master yet (perhaps because it's not ready). We can perform the transplant with a git rebase --onto as follows:

$ git log --oneline feature2 986d8ac F2b 625bcde F2a fb24802 F1b d8f7e48 F1a 3cd0af7 Mc addd99a Mb 7ad7ead Ma $ git rebase --onto master feature1 feature2 First, rewinding head to replay your work on top of it... Applying: F2a Applying: F2b $ git log --oneline feature2 b3e017a F2b ac65d2e F2a 3cd0af7 Mc addd99a Mb 7ad7ead Ma $ git log --oneline feature1 fb24802 F1b d8f7e48 F1a 3cd0af7 Mc addd99a Mb 7ad7ead Ma $ git log --oneline master 3cd0af7 Mc addd99a Mb 7ad7ead Ma

We've now transplanted the bit between feature1 and feature2 onto the current master branch. In effect, we're doing a git cherry-pick of all the changes between feature1 and feature2 onto the current point of master, then reseting the feature2 branch (that we're currently on) to point to the new location. Our git repository now looks like:

Ma → Mb → Mc ← Master \ → F1a → F1b ← Feature1 \ → F2a → F2b ← Feature2

Summary

Rebasing is an incredibly convenient way of updating your local repository to the current version of the branch on a remote server, and is used frequently. It can also be used to perform multiple cherry-pick operations and reorder (local) history in order to create new versions of that history.

As with any power tool, care must be taken not to reorder changes which have previously been made available (e.g. via GitHub) as creating new history causes a divergence in the timeline which causes eddies in the space time continuum. Or at least annoys people.

Come back next week for another instalment in the Git Tip of the Week series.

Running Gerrit with Jenkins/Hudson

2011-06-10T12:00:00.001+01:00

I've been writing about Gerrit and Jenkins/Hudson for some time now, and I've posted a couple of screencasts for Java developers and for iOS developers as well. I've also been promising to write about how to set this up, so you can replicate the goodness of the screencast in your own work location and help spread the word.

That day has finally come, and the article is available at InfoQ under http://www.infoq.com/articles/Gerrit-jenkins-hudson, which should take you through all the steps you need to in order to replicate the experiment.

I've deliberately used the term Jenkins/Hudson throughout, because the technique works fine, regardless of which system you're using. I've also tested it on the latest of both Jenkins and Hudson, as well as the recently-released Gerrit.

A couple of changes are worth pointing out:

Gerrit is migrating data from an SQL database to using Git notes for storing review information. The 2.1.x and 2.2.x differ in this regard; and as such, so do the permissioning screens.

Gerrit 2.2.1 (and 2.1.7.2) have renamed the “All Projects” project from “--+All+Projects+--” to “All-Projects”. The upgrade script will do this rename for you, but if you have any systems which refer to this by name (or URLs in an article... :-) then you have to change them to reflect that.

There was an issue with Hudson 2.0.0 with the latest Gerrit-Trigger, which is fixed with to 2.0.1

Gerrit's JSON API changed between 2.1.6 → 2.1.7, and has now been reverted in 2.1.7.2 (and similarly, 2.2.0 → 2.2.1). This may cause some minor knock-on consequences with the Gerrit Trigger and/or Gerrit Mylyn review.

I hope that the article is useful. Any feedback/comments, feel free to add them at the InfoQ article, comments on here, or tweeting me about it.

World IPv6 day

2011-06-08T08:37:00.001+01:00

Today is World IPv6 day, where internet large organisations and small (including my own Bandlem Limited as well as this blog) enable IPv6 connectivity to encourage others to do so.

By enabling IPv6, anyone with a dual-stack will try to connect over IPv6 first and fall back to IPv4 afterwards. So even if your IPv6 connectivity is broken, you should still be able to access the current sites. If you only have an IPv4 address then you will not notice any difference.

I've written over on InfoQ what World IPv6 day is all about, but chances are there are a handful of IPv6 enabled sites that you will visit today, including the popular search engines Google and Bing. It also includes internet social giant Facebook, although at this time Twitter does not appear to be participating in the IPv6 experiment.

You can find out more about your connectivity via http://ipv6-test.com/ if you're interested in knowing what your computer supports.

Git Tip of the Week: Cherry Picking

2011-06-07T09:00:00.000+01:00

This week's Git Tip of the Week is about pulling changes from one branch to another, called cherry picking. You can subscribe to the feed if you want to receive new instalments automatically.

Picking changes

Each commit in a repository corresponds to a full tree of files. Usually, these files have been created over a number of commits. But sometimes, it's necessary to take the delta between two commits, and apply it to a different branch.

One common case where this occurs is when an issue has been identified, and subsequently fixed, but needs to be backported to a previous release branch.

In this case, you don't want to take the current state of the tree (which might have unfinished or untested changes); you just want to take the delta associated with that change.

In other version control systems, you would just create a diff based on the most recent change, and then patch the change on to your release branch. Instead, with Git, we can use the cherry-pick command to do the work for us:

$ git checkout master $ echo Working >> file.txt $ git commit -m "Working" file.txt $ echo BugFix >> bugfix.txt $ git commit -m "BugFix" bugfix.txt $ echo More Working >> file.txt $ git commit -m "More working" file.txt # We want to apply 'bugfix' to release $ git checkout release10 $ git cherry-pick master~1 [release10 41037ab] BugFix 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 bugfix.txt

This has allowed us to take a single change – described here with master~1 – and copy the delta into the release branch.

Sets of changes

We can pick commit sets (ranges of revisions) to pick if we wanted to. Had we had several changes, we could have master~3..master~1. Unlike just generating a diff and then patching the current tree, this will copy the commits (and their relationship) over to the new branch.

We've actually seen pick in use already; when we covered rebasing last week. When you create a series of commands for rebasing, you're actually giving it instructions to pick or edit existing changes:

@ edit 7bf9271 Typo @ pick 756281e First @ fixup 07e9061 First @ pick 13aba60 Second

The “pick” here means the same as “git cherry-pick” for the single change.

In fact, “edit” is really a short-hand for “git cherry-pick -e”, and “fixup” and “squash” are short-hands for git cherry-pick -n.

Recording source

Finally, it's worth noting that when you copy a change using this mechanism, the commit hash will change (notably because it will have a different parent hierarchy).

Sometimes that doesn't matter, but if you want to record where the original change came from, you can run git cherry-pick -x. This inserts a commit message indicating where the original change came from:

# From example above $ git checkout release10 $ git cherry-pick -x master~1 $ git cherry-pick master~1 [release10 41037ab] BugFix (cherry picked from commit 938a4c0bbb3985524192aa8a926ea6757263e94b) 0 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 bugfix.txt

However, note that this usually only makes sense if the change that's being cherry-picked is from a public branch (so that the referenced change is visible). Another way of representing this change is to create a merge node between the release branch and the ongoing development branch as a way of showing that the merge has occurred.

Creating new history

Whenever you are cherry-picking, especially if you are reordering commits, you're creating new history. However, you've never really thrown away old history; it's all available from the reflogs. You're not destroying history, you're creating alternate histories. All cherry-picking gives you is the ability to apply patches from other branches in a safe and error-less manner.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Rebasing

2011-06-01T19:56:00.001+01:00

This week's Git Tip of the Week is about rewriting history. You can subscribe to the feed if you want to receive new instalments automatically.

Rewriting history

One of the philosophical differences between Git and Mercurial is whether history should be allowed to be re-written or not. When a commit is made, the commit hash represents a point on that history – and subsequent commits then rely on that hash for integrity and representation of parental links.

Rewriting history can thus be dangerous; if you change a commit in the past, it invalidates the current commit's hash. Instead, you need to re-commit the current change against the new commit to get a new hash value.

However, dangerous is relative. It's not always the case that changing the history is bad – if that history is local, and hasn't been seen by anyone else, then the only person it's affecting is you. As long as you know what you're doing (and who you will affect) then changing the local history is no different than undoing your local changes in a text editor and re-saving.

(Side note; Mercurial has the concept of 'patch queues' which are the equivalent of local mutable history – but you end up with two separate repository concepts instead of a single concept of history as in Git.)

So, the question is not so much as whether rewriting history is dangerous as to understanding the effects if these changes are exposed to others (e.g. via pushing to GitHub). Sometimes, it's necessary to publicly break the commit hashes – for example, someone accidentally committed a large binary, or copyrighted code which shouldn't be present (or even a password which shouldn't have been committed) – but in these cases, making the change often involves a public notification to warn others.

Rebasing

So, what is rebasing? Well, rebasing is Git's concept of changing (recent) local history. In essence, it is an undo/replay option that you can use to make changes as if they were done in the past.

A Git rebase unwinds history to a particular point (typically specified as HEAD~n where n is the small number of previous commits in the past), and then replays the same changes on top of the code. If the changes are unmodified, then the resulting commit will be the same as before.

However, it's more normal to want to adjust the commit(s) in some way, for example:

Reword – change the commit message to something else (e.g. to add a bug reference)

Edit – to make changes to the commit itself (e.g. fix a typo in the code)

Pick – to include that commit in the history

Squash – to condense that commit with the previous and make them one (and concatenate log entry)

Fixup – to condense that commit with the previous and make them one (and discard log entry)

As well as these options, it is also possible to re-order them simply by re-ordering the list of changes.

Example

Let's build up a repository with some changes we'd like to make:

$ git init example Initialized empty Git repository in example/.git $ cd example $ git commit --allow-empty -m "Initial Commit" $ echo Helo World > README.txt $ git add README.txt $ git commit -m "Typo" README.txt $ echo Second > Second.txt $ git add Second.txt $ git commit -m "Second" Second.txt $ echo Frst > First.txt $ git add First.txt $ git commit -m "First" First.txt $ echo First > First.txt $ git add First.txt $ git commit -m "First" First.txt $ git log 07e9061 First 756281e First 13aba60 Second 7b49271 Typo 82f9a21 Initial Commit

What we'd like to do is fix the typo made in the first commit, join the two First commits into one, and reorder the Second so that it comes second in the list. To do this, we kick off an interactive rebase, which will give us an editor:

$ git rebase -i 82f9a21 @ pick 7bf9271 Typo @ pick 13aba60 Second @ pick 756281e First @ pick 07e9061 First

What this is saying is a sequence of cherry-picks to replay the history with the specific changes listed. We can re-order them to replay history in a different manner:

@ edit 7bf9271 Typo @ pick 756281e First @ fixup 07e9061 First @ pick 13aba60 Second

Git will then rewind history to the parent of the Typo commit, and drop us down into a shell which allows us to make changes:

$ echo Hello World > README.txt $ git add README.txt $ git commit -m "Readme" README.txt $ git rebase --continue

Here, we've stopped editing for a while and kept going through the rebase operation. We could insert more commits if we wanted to but we've just committed the current state as is. You might also see:

error: could not apply 07e9061... First hint: after resolving the conflicts, mark the corrected paths hint: with 'git add <paths>' and run 'git rebase --continue' Could not apply 07e9061... First

This is caused because we're changing the same line in the same file, and Git is asking if that's OK. We can simply add that file and continue, or (given that 07e9061 is a complete replacement for 756281e) just not have done it in the first place:

$ git rebase --abort $ git rebase -i 82f9a21 ... @ edit 7bf9271 Typo @ pick 07e9061 First @ pick 13aba60 Second

By removing the commit from the list, it is as if that commit never happened. This should run through and allow you to commit all the changes without having any conflicts.

Summary

Rebasing allows you to re-write history in an automated manner, instead of having to unwind and manually replay the changes yourself. It's often used with git rebase -i HEAD~5 (or some other small number) to fix up changes in your local history before merging or pushing to a central repository.

Rebasing also allows the transplantation entire sections of a tree, which we'll talk about another time.

Finally, remember that Git never loses commit data. If you're working against a branch, you've got the branch's reflogs to fall back on; what Git allows you to do is effortlessly rebuild new commit trees (whilst keeping the old commit trees around in your local cache) until you're happy with the result.

Come back next week for another instalment in the Git Tip of the Week series.

Adventures in Multi-Architecture Eclipse

2011-05-24T19:19:00.001+01:00

To bring to an end the sorry saga of trying to make a multi-architecture Eclipse build, the conclusion appears to be that it is impossible due to limitations in P2.

Although in my last post, I claimed to have success in having a per-architecture “configuration” directory and “eclipse.ini” file, the reality is that P2 leaps to the rescue in breaking what otherwise would be a fine set up. And I don't care how it's supposed to be capitalised.

The problem is twofold:

An Eclipse P2 profile, written into the eclipse.p2.data.area location is single-architecture by design

P2 doesn't check what the location of the configuration dir or eclipse.ini files are during updates; it just assumes they have the standard values

Earlier attempts of generating two profiles (on a per-architecture basis) succeeded in getting the application off the ground. However, since P2 also writes into configuration/org.eclipse.equinox.simpleconfigurator/bundles.info, we have to split the configuration directories as well.

Conventionally, the configuration directory is called configuration, but you can change it with a command-line switch (-configuration) or with an entry in the eclipse.ini file. This value is known at runtime (go to the 'About' page and you see it shown at the top) but P2 thinks it's still called configuration, regardless of what it actually is.

Not only that, but the eclipse.ini isn't always called eclipse.ini. The actual lookup is product.ini, with a fallback to eclipse.ini, if the product-specific one can't be found. This allows you to rename the launcher to something else, or (originally) to have multiple products in a single location.

Sadly, whilst you can launch a multi-architecture build using the previous case, you can't update it. P2 will write entries out to eclipse.ini and configuration, which if you've called them eclipse32.ini and configuration32 are of no use. Ironically, the plugins do actually get loaded and installed into the plugins directory; it's just the P2 metadata which is written to the wrong place. In The Good Old Days, you'd have been able to run eclipse -clean and it would have just worked – though to be honest, a multi-architecture would have just worked then as well.

All of this brings us to the conclusion that a single package, multi-architecture build is not going to happen with products in one folder. It might be possible to have one folder per executable with a shared plugin or feature store; but by the time you get to that, why not just stick all the plugins and features into a local M2_REPO type cache and just run from there?

Notes: yes, there's a bug, so please don't leave a comment asking whether I've raised one. And in all honesty, if you're going to leave a comment with P2 vs p2, then I'd much prefer you spent less time worrying about what people are calling your product and more time on getting it to work.

Git Tip of the Week: Git Revisions

2011-05-23T09:00:00.000+01:00

This week's Git Tip of the Week is about looking for commits with git revisions syntax. You can subscribe to the feed if you want to receive new instalments automatically.

Git Log

Git log is a versatile tool which can let you introspect the state of your repository. We've been implicitly using it in a number of other examples already; in this post, we're going to look at some of the other options that git log takes, as well as the ways in which we can refer to items in Git's history.

Last week, we looked at Git reflogs, which (together with the previous post on git stash) discussed a notation for looking at commits with the HEAD@{1} (c.f. stash@{1}) syntax. In fact, there's a lot of other mechanisms we can use to refer to items in Git's history, over and above the commit hash or branch name.

A git history is a directed acyclic graph of commits, from the HEAD backwards to one (or more) roots. In most cases, commits have a single parent; but merge commits have two (or more) parents. (Hg, by contrast, can only have one or two parents. Converting from a Git repository to an Hg repository is therefore not totally faithful.)

Since each commit may have more than one parent, the parent operator (^ as a suffix) allows you to disambiguate which parent you are referring to. Given that each merge node is represented as a pair (or more) of commits, the parents are numbered from 1 to n. (The special number 0 is used to refer to itself.)

# Dummy repository $ git log --oneline 77bc990 Third commit 25d4fc4 Second commit f0faab6 First commit $ git log --oneline HEAD^ 25d4fc4 Second commit f0faab6 First commit $ git log --oneline HEAD^ 25d4fc4 Second commit f0faab6 First commit $ git log --oneline HEAD^^ f0faab6 First commit $ git log --oneline HEAD^2 fatal: ambiguous argument 'HEAD^2': unknown revision or path not in the working tree.

Here, HEAD is pointing to 77bc990 Third commit, and so both HEAD^ and HEAD^1 refer to the same item (25d4fc4 Second commit). However, HEAD^^ gives a different answer to HEAD^2; in the former, finding the root of the history and in the latter giving an error.

That's because HEAD^^ means “the (first) parent of the (first) parent of HEAD”, whereas HEAD^2 means “the second parent of HEAD”. Generally, HEAD^n where n >= 2 only makes sense on merge nodes.

However, there is another useful reference; the grandparent selector. Instead of considering a breadth-based lookup, it does a depth-based lookup:

# Dummy repository $ git log --oneline 77bc990 Third commit 25d4fc4 Second commit f0faab6 First commit $ git log --oneline HEAD~ 25d4fc4 Second commit f0faab6 First commit $ git log --oneline HEAD~~ f0faab6 First commit $ git log --oneline HEAD~2 f0faab6 First commit

Like the parent operator, the grandparent operator can take a number as well; except instead of referring to the nth parent, HEAD~n refers to the nth grandparent.

Note that the ~ selects up the first parent (much like ^ does); but the two forms can be mixed if needed. In this case, both HEAD^~ and HEAD~^ have the same effect; but you can explicitly select which item you want with a numeric selector; for example, git log HEAD^2~10 gives you the current merge node's second parent's tenth ancestor.

Ranges and sets

As well as single references, it's possible to refer to ranges in Git as well. In fact, these are referred to as sets of commits (since they may not be contiguous on the commit tree). The most common way is to show the commits between one ref and another:

$ git checkout -b other f0faab6 Switched to a new branch 'other' $ touch file $ git add file $ git commit -m "Adding file" file $ git log --oneline 1762164 Adding file f0faab6 First commit $ git log --oneline other..master 77bc990 Third commit 25d4fc4 Second commit $ git log --oneline master..other 1762164 Adding file

The syntax is between..and, where both references can either be one of the symbolic branch/tag references, or a commit hash etc. What this is saying is “Show me all the commits which are in master but not in other” (and vice versa, for the second one).

However, whilst it looks like a range, really this is just a set selection. The syntax between..and is actually a shorthand for ^between and, which itself is a shorthand for and --not between. In other words, the above code is showing you what is in other but not in master.

What use is the long hand, when the shorthand is much more convenient? Well, the long hand allows you to specify more than two references. For example:

# Show all unmerged changes between features 1,2,3 and master $ git log ^master feature1 feature2 feature3 # Show changes in hotfix and release branches, but not in master $ git log ^master hotfix release-1.0 release-1.1

In most cases, the asymmetric diff (using two commits) is probably what you want. It's worth noting, finally, that there is a symmetric diff (using two commits) – instead of two dots, use three:

$ git log master...other 1762164 Adding file 77bc990 Third commit 25d4fc4 Second commit $ git log other...master 1762164 Adding file 77bc990 Third commit 25d4fc4 Second commit

Note that there's no difference in the order of the commits, since they're ordered by time (or however you want to order them in git log). And, since it's symmetric, the result is the same regardless of the order of operands.

Summary

Git allows you to specify single commits as well as a range of commits using a number of git revision identifiers, and these can be used to find out the state of and differences between trees. In this post, we covered the parent operator (^), the ancestor operator (~), as well as ranges with asymmetric and symmetric differences.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Reflogs

2011-05-17T09:00:00.000+01:00

This week's Git Tip of the Week is about recovering work with the reflog. You can subscribe to the feed if you want to receive new instalments automatically.

Reflogs

The Git reflog is very different from the Hg reflog of the same name. Hg reflogs are equivalent to CVS's ,v files. (Actually, they're RCS's, but never mind that now ...)

A Git reflog is a list of hashes, which represent where you have been during commits. Each time a branch is updated to point to a new reference, an entry is written in the reflog to say where you were. Since the branch is updated whenever you commit, the git reflog has a nice effect of storing your local developer's history.

Furthermore, the pointer in the reflog points to a commit object, which in turn points to a tree object, which represents a directory-like structure of folders and files. So whilst the reflog is active, you can go back and see what changes you have made – and even recover specific files from previous commit versions. With Git, you never really lose anything; even if you've done a filter-branch to re-write history, you're only a reflog entry away from getting it all back.

To see what the reflog is all about, run git reflog from an active Git repository. It might look something like this:

9bdbd83 HEAD@{0}: commit: Adding build script 86a7a39 HEAD@{1}: commit (amend): Updating commit message 325e0af HEAD@{2}: commit (amend): Example Project (fix typo) 06bf85e HEAD@{3}: commit (initial): Example project

The first number is simply the commit hash at the point the change was made. Even though these don't represent linear history (you'll see a couple of (amend) listed there), these are the sequence of actions taken on the local repository, in the order they were done.

The second is the state of HEAD, along with the number of changes. In this case, we have HEAD@{0}, which means where HEAD is now; HEAD@{1} is where HEAD was previously, and so on.

The final part is the type; whether it is a commit or an amended commit, and the commit subject. This is often helpful to remember where the code was, especially if it isn't part of the linear history. It also contains other operations, such as checkout, merge and reset:

abec02f HEAD@{0}: merge foo: Merge made by recursive. 9bdbd83 HEAD@{1}: 9bdbd83: updating HEAD 2d90ece HEAD@{2}: merge foo: Fast-forward 9bdbd83 HEAD@{3}: checkout: moving from foo to master 2d90ece HEAD@{4}: commit: hello 9bdbd83 HEAD@{5}: checkout: moving from master to foo

Reflog references

Most git commands accept a number of different references to point to a commit. For example, you can run git checkout master, git checkout abec02f and git checkout mytag. However, you can also checkout references by reflog as well.

In the example above, we can run git checkout 290ece, or we can refer to it as git checkout HEAD@{2}. Provided we haven't committed anything else (which would change the reflog), these two variations have the same effect.

You can use this to implement a crude form of undo:

git config --global alias.undo "reset HEAD@{1}"

This will cause you to revert to the previous action (whether it was a commit or otherwise).

Reflog references revisited

Although we've been using HEAD here, reflogs are more general than just HEAD. The general representation is name@{qualifier}.

In fact, stashes (covered last week) are a specific form of reflog, whose name is stash. Not only that, but other branches can be referred to by their reflog as well.

All of the branch reflogs are stored under .git/logs/refs/heads/. (There's also one under .git/logs/HEAD, as well as .git/logs/refs/stash if you've used a stash before.)

We can identify reflogs for a specific branch as well:

$ git reflog show master abec02f master@{0}: merge foo: Merge made by recursive. 9bdbd83 master@{1}: 9bdbd83: updating HEAD 2d90ece master@{2}: merge foo: Fast-forward 9bdbd83 master@{3}: commit: hello $ git reflog show foo 2d90ece foo@{0}: commit: hello 9bdbd83 foo@{1}: branch: Created from HEAD

Since all of these are valid git references, we can perform diffs against them, e.g. git diff foo@{0} foo@{1}.

Timed reflogs

Since each reflog has an implicit time associated with it, you can filter not only by history, but also on time as well. Various supported forms include:

1.minute.ago

1.hour.ago

1.day.ago

yesterday

1.week.ago

1.month.ago

1.year.ago

2011-05-17.09:00:00

The plural forms are also accepted (e.g. 2.weeks.ago) as well as combinations (e.g. 1.day.2.hours.ago).

The time format is most useful if you want to get back to a branch's state as of an hour ago, or want to see what the differences were in the last hour (e.g. git diff @{1.hour.ago}). Note that if a branch is missing, then it assumes the current branch (so @{1.hour.ago} refers to master@{1.hour.ago} if on the branch master.

Summary

Git never really loses anything, even if you perform filter branching or commit amending. The commit, tree and contents are still stored in the repository and the reflog still maintains pointers to those previous commits.

Each time you update a branch, it stores the set of past revisions in a reflog, which is both stored against HEAD and against a particular branch (including a special reflog for stashes).

The reflogs stay until expired (which can be done with the git reflog expire command). The default for unreachable commits is 30 days (or the gc.reflogExpireUnreachable config value) or, for reachable commits, 90 days (or the gc.reflogExpire config value).

Come back next week for another instalment in the Git Tip of the Week series.

Git, Gerrit and Jenkins for iOS development

2011-05-10T21:49:00.000+01:00

Last week, I gave a presentation at the London iOS Developer Group meeting at the Apple Store in Regent Street, London. The purpose of the talk was to cover the points I'd made previously in Someday... and the subsequent Git, Gerrit and Jenkins presentation. Links to resources are available from the Lanyrd site.
Since these were both Java specific, and I gave a short talk at NSConf earlier this year, I wanted to do something more iOS specific. The resulting demonstration evolved into the one seen here.
This video is an audio dub of the same presentation material, but re-recorded for the purposes of vimeo. Enjoy!

Git, Gerrit and Jenkins for iOS development from Alex Blewitt on Vimeo.
I have added this to those subscribed to the git tip of the week feed as a departure from the regular Tuesday series.

Git Tip of the Week: Stashes

2011-05-10T09:00:00.000+01:00

This week's Git Tip of the Week is about keeping work in progress safe, known as stashing. You can subscribe to the feed if you want to receive new instalments automatically.

Drop everything bugs

Sometimes, when you are working on a problem you need to drop it suddenly in order to work on a different problem, like a critical bug that has been reported against a production version of your code.

Whilst you could easily create a new checkout of your project with the specific branch, it's faster to switch branches in Git natively, and saves setting up other tools like an IDE or a test server.

The problem is, your work may not be in a fit state to commit at the time the bug comes in. Rather than committing a half working state, you ideally want to save the work in progress (including dirty, uncommitted files), to permit you to switch to the higher priority issue, and resume when necessary.

Git Stash

This is where git stash comes in handy. This takes a working tree and files it away in a location which can be retrieved at a later stage. The stash can then be popped to get the changes back as they were prior to the stash taking place.

To create a stash, just run the git stash command. You can see what stashes are present with git list:

$ git stash list $ touch example $ git add example $ git stash Saved working directory and index state WIP on master: dc3ea83 HEAD is now at dc3ea83 $ git stash list stash@{0}: WIP on master: dc3ea83 $ ls example $ # do emergency bugfix here, and afterwards $ git stash pop # or git stash apply $ ls example example

Here, we created a new stash which contained the example file. Once we'd done the stash, we were reset to the current commit (i.e. a clean workspace), so the example file is no longer present.

We can list the available stashes with git stash list, which gives them their identifier (in this case, stash@{0}) which can be used to identify the changes. (It defaults to the last stash if not specified.)

The stash identifier also includes what branch the stash was created from. Importantly, once the stash has been created, applying (or popping) the stash results in just the differences being added, rather than resetting back to a specific previous state. This allows subsequent changes to be made on the branch, followed by replaying the stash changes on the current version of the branch afterwards.

Git stash can be used to avoid merge conflicts with work-in-progress code. Simply stash before doing a pull, and unstash afterwards, as the example in the manpage shows:

$ git pull ... file foobar not up to date, cannot merge. $ git stash $ git pull $ git stash pop

Notes

The stash only applies to added files; it doesn't apply to untracked files. If you want to add untracked files as well, you need to run git add --all prior to running git stash operation.

The WIP stands for Work In Progress, since it's not obvious to non-English native speakers.

git stash pop will remove the stash from the list; git stash apply will keep the stash in the list

The {0} represents the last stash you did, the {1} represents the second to last, and so on. This syntax also turns up elsewhere (e.g. reflogs).

Individual stashes can be removed with git stash drop, or git stash clear to get rid of all of them.

Git stashes are stored as commit objects, but the branch HEAD isn't updated to point to them. Instead, they are referenced from a reflog entry in .git/logs/refs. You can use git show stash@{0} to see a stash.

Come back next week for another instalment in the Git Tip of the Week series.

Git Tip of the Week: Gollum

2011-05-03T09:00:00.000+01:00

This week's Git Tip of the Week is slightly off topic, talking about the Git-based wiki Gollum. You can subscribe to the feed if you want to receive new instalments automatically.

Wikis

Wikis have been around with us for over 15 years, and being able to edit a page from any location has been a massive timesaver in documentation generation. Most of the EGit User guide has been created through the use of wiki contributions.

One problem with most wikis is that you need to be on-line in order to interact with them. (There are some clients that cache pages from Wikipedia for off-line reading, but these are not able to manage changes.) Wouldn't it be great if we could have a system that allowed us to edit wikis off-line, and merge our changes back in when we are connected again?

Gollum

Enter Gollum from those fine people at GitHub. This is a ruby implemented wiki server which allows you to view, edit and save documents into a wiki on your local machine. (It's been around for a while but not widely known about, for some reason.)

As it's from GitHub (and this post is about Git tips, after all) then it should come as no surprise that Gollum is a wiki server for wiki pages backed by a git repository. Each save corresponds to an individual commit in the repository, and the “write a change message” box at the bottom is translated to the Git commit message.

Each wiki page is translated to a page specified in the --page-file-dir directory (or the repository root, if not set). Furthermore, the markup is user-configurable and defaults based on extension type (the default for new pages is Markdown). Other sane wiki formats (like MediaWiki) are also supported, though insane formats (like confluence) are not. In fact, the multi-wiki-format is supported through Jekyll, so whatever formats are supported there are likely to be usable.

Up and running

Installing Gollum is easy, provided you have Ruby installed (which you do if you have OSX). You can run:

sudo gem install gollum

… which will install it in your system's path automatically, or without sudo to install in a per-user path (which on OSX is in ~/.gem/ruby/1.8/bin, which you'll have to add to your path if you want to run it from the command line).

Once installed, you can create a repository, fire up Gollum, open a web browser, and you're off:

git init TestWiki gollum --page-file-dir wiki TestWiki openurl http://localhost:4567

This creates a new wiki (for test purposes) and fires up the Gollum server, pointing it to that Git repository. We have specified wiki as the subdirectory, so that when we commit a file, we're writing it into TestWiki/wiki/PageName.md.

Formats

There are many different types of formats that are supported by Gollum, but the parsers have to be installed separately in order to see any wiki content rendered as you expect.

Markdown gem install rdiscount

MediaWiki gem install wikicloth

If you want to pretty-print code, you can use Pygments with sudo easy_install pygments. This allows you to begin code with ```java and end with ``` to pretty-print the embedded code snippet.

So, if you try editing a page and the markup isn't being rendered appropriately, check that you have the appropriate renderer installed in order to work.

Identity

Git commits in the repository use the credentials that are associated with the user who launched Gollum. Whilst this works for open projects, if you want to have a recorded user from some kind of SSO, you'll need to integrate this. Gollum uses the Ruby Sinatra to generate the web-based front end, and this is used to determine whether authentication is used or not (see the FAQ).

However, it doesn't support passthrough of the committer's identity into the commits, or using the GIT_AUTHOR_EMAIL or GIT_COMMITTER_EMAIL variables (though this is an issue with Grit, the Ruby front-end to the Git repository). This, combined with lack of multi-project support somewhat limits it for production uses, but works fine for a local (single user) wiki.

Summary

Gollum provides a powerful web-based mechanism to edit wiki pages in a local git repository, using the local user's commit credentials. This allows a distributed wiki to be edited remotely (whilst disconnected) but managed by Git under the covers, including the ability to branch, tag, and distributed pushing.

Whilst it's lacking in some key use cases that make a distributed wiki front end for teams of users, for an anonymous wiki (or one where the committer credentials are of lesser importance) it's an incredibly easy system to get up and running. And for those that regularly write whilst disconnected, it can be a good way to build up a repository of information without needing to be connected yet still allow those changes to be merged into a repository when reconnected.

Come back next week for another instalment in the Git Tip of the Week series.

Adventures in Multi-Architecture Eclipse

2011-04-28T09:46:00.001+01:00

I've been working on a multi-architecture single build for Eclipse, as described previously, and have now got it to a state where I can generate a build for multiple architectures into a single install. Unfortunately, it's not easy to script with Tycho (yet) but it can be done.

The problem is largely with P2. Although the plugin soup can handle having plugins installed for both its platform and others simultaneously, P2 needs to be told (through the use of a profile) which subset it is to look at. Instead of doing something sensible (say, assuming that the current set of installed plugins represents the 'starting point' and going from there), P2 insists that the profile be built up by a series of installation requests.

The problem is this profile contains architecture-specific installation. Even if you install the same RCP base product, the profiles are different as they reference different fragments (so, on an x86 build you have SWT-x86, and on an x86_64 build you have SWT-x86-64). Due to this three character difference, you have to build the product twice.

The first approach – use a shared bundle pool – doesn't work. That's because P2 checks to see if a bundle has already been installed, and if so, skips it. This works fine for bundles without architecture-specific fragments, but fails for those with. Run P2 on a shared install for x86 and x86-64, and when the second install happens, it says “Hey! SWT is already present; nothing to do”. Of course, this misses the fact that SWT needs the SWT-x86-64 fragment in order to work and so you end up with a failed product.

The second problem is the configuration folder. Although this is 99% platform agnostic, the simpleconfigurator has a list-of-bundles to install upon startup. This too contains the architecture-specific fragments, so although it is almost identical, it differs in a few key places.

Fortunately, all is not lost. We can fix this by doing three things:

Have each architecture have its own profile name

Have each architecture have its own configuration directory

Rename the executables per platform (and corresponding ini files)

The first is relatively easy; in the P2 director, pass a different profile name. Unfortunately, Tycho doesn't currently let you have a different profile name per architecture so you either have to use two projects or a hacked version in order to achieve this goal.

The second is more challenging. There's no way of telling the director what the configuration directory should be at build time, although you can specify this at run time. The trick is to rename the configuration directory to something architecture-specific (say, configuration.x86-64) and then write a pointer to that from the ini file for the launcher; e.g. -config configuration.x86-64. That way, running Eclipse.x86-64.exe will load Eclipse.x86-64.ini, which points to configuration.x86-64.

This does mean that an installation of a plugin for one architecture won't make it visible in the other architecture (they share different configuration and profile directories) but at least both can be launched from a single image. A post-process cleanup script can be run on a per-installation basis to remove unnecessary products if desired.

Being able to serve a single platform from a single build is an important one in minimising download and maintenance costs across an organisation or disk usage across the internet as a whole. Adding launchers for other architectures doesn't add much to the installer size; though whether it makes sense to combine different platforms (Windows, Mac, Linux) may be up to the use cases and policies that require them. And whilst it's possible to merge the executables for those that support fat binaries, this isn't possible on all platforms.

However, there's really no reason why there isn't a single build for Windows (all platforms), Linux (all platforms) and Mac (all platforms). It shouldn't be necessary to have to download everything again just to get a slightly different fragment, and maybe this approach will be a way of allowing a transition to occur to a more sensible build strategy.

Git Tip of the Week: Tags

2011-04-26T09:00:00.000+01:00

This week's Git Tip of the Week is about working with tags. You can subscribe to the feed if you want to receive new instalments automatically.

Descriptive labels

Git provides a couple of mechanisms for identifying changes by labels instead of by unique hash values.

The first, we've already seen, is branches. When we switch between two branches, we're really using the descriptive label to identify a specific commit to switch to.

The second, which we'll introduce here, is tags. A tag is like a branch, in that it identifies a specific commit with a descriptive label.

Branches versus Tags

What's the difference between tags and branches? The workspace is (almost always) associated with a branch, called master by default. When it is, a commit will automatically update the master reference to point to that new commit; in other words, branches are mutable references.

A tag, on the other hand, is created to point to a specific commit and thereafter does not change, even if the branch moves on. In other words, tags are immutable references.

Annotated Tags

Git has two flavours of tags; annotated and non-annotated. When using them, there is little difference between the two; both will allow you to refer to a specific commit in a repository.

An annotated tag creates an additional tag object in the Git repository, which allows you to store information associated with the tag itself. This may include release notes, the meta-information about the release, and optionally a signature to verify the authenticity of the commit to which it points.

Examples

We can create a simple tag, based on the current repository's version, with:

$ git tag example

This creates a lightweight tag as a reference in .git/refs/tags/example, which points to the current commit. If we want to make it as an annotated tag, we need to supply -a, and a message with -m:

$ git tag -a v1 -m "Version 1 release"

This will create an (unsigned) annotated tag object, containing that message and a pointer to the commit object. Now the reference in .git/refs/tags/v1 will point to the tag object, which then points to the commit.

If we wanted to guarantee the authenticity of the tag, we could use -s on the git tag command. This uses gpg to sign, based on your email address – though you can use -u to specify a different gpg identity instead. You can verify the signature of an existing tag with -v.

To list the local repository's tags, run git tag without any arguments; or, for a pattern, use -l with * as a wildcard:

$ git tag example v1 v1s $ git tag -l *s v1s

Finally, to get rid of tags, you can delete them with -d:

$ git tag -d v1 $ git tag example v1s

Deleting tags are OK if you never made them publicly available, but you really should avoid deleting tags once you've pushed them to a publicly readable location. Similarly, you shouldn't change a tag once it has been released to the wild either.

Contents and Describe

In order to see what the tag contains, you can use git show, as you can with other git objects:

$ git show v1s tag v1s Tagger: Al Blue <alblue@example.com> Date: Tue Apr 20 09:00:00 2011 +0100 Version 1 signed -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) iF4EABEITke4AkyRUh8ACgkQWwXM3hQMKHZq5QD/esqKyinelXGM1TSzUqEzuBdI Ah2Cq/5TS3j4kiP4+UUA/2nN2SVoWwYryN9234kgWUvZIrV1P0FGTG+lAEN5avj3 =JICf -----END PGP SIGNATURE----- commit 91a2b24....

If the tag is an annotated tag, you'll see the message and the tag object, followed by the commit. If the tag is a lightweight tag, then you'll see only the commit object.

A key difference between annotated and non-annotated tags is in the use of git describe. This gives an identifier of the repository, based off of the nearest annotated tag. If we were to run now, we'd see a reference to the v1s annotated tag:

$ git describe v1s

If the current commit exactly matches that of a tag, then only the tag name is printed. If there are changes, then git describe will print out the tag name, a hyphen, the number of commits made, a hyphen, the letter 'g' and then the commit identifier. This allows anyone to use that explicit revision to identify the commit, through the hash at the end. As such, it is often useful to include that in file versions as a means of identifying it at a later stage.

# Add a commit $ touch file $ git add file $ git commit -m "Adding file" $ git describe v1s-1-g24242c3

The letter g is added to denote a git managed version; so other repositories can use the same format but substitute that letter for a different one.

If no annotated tags are found then it will print fatal: No names found, cannot describe anything. To allow describe to use non-annotated tags, run with git describe --tags. It's also possible to get it to describe against a branch using git describe --all, although this only makes sense if the branch is known remotely.

Pushing and Pulling

Since a tag (either annotated or lightweight) is just a reference on your local repository, it is not sent up by default to the remote repository during pushes. (This is one observable difference between Git and Hg.) Instead, you can git push the tag individually, or you can run git push --tags which will push all tags. For “release” tags (e.g. V1.0.0) it is conventional for these to be annotated tags; it is relatively rare that you will push a lightweight tag to a central repository.

For pulling, any tags associated with your current branch will be fetched when you check it out. This may result in not having all the tags in your local repository that the remote repository has. If you'd like to fetch them all, you can do git fetch --tags to pull them all in, or git fetch tag to pull a single one.

Summary

Tags in git are lightweight references that point to an SHA hash of a commit. Unlike branches, they are not mutable and once created should not be deleted. Tags may be lightweight (in which case they refer to the commit directly) or annotated (in which case they point to a tag object which points to the commit). Tags used to denote versioned releases typically use annotated tags, and for many open source projects, the tags will also be signed.

Come back next week for another instalment in the Git Tip of the Week series.

Cleaning up Google results

2011-04-19T17:27:00.001+01:00

If you search for information with Google, you might occasionally find that your results are infiltrated with useless copy-and-paste sites. Fortunately, you can block these from your results if they turn up, provided that you're signed in to a Google account when you're searching.

It's also possible to define a list of known bad sites even before they appear in a search, by going to google.com/reviews/t. Anything you search for will automatically exclude results from those sites mentioned. You also use the same page to unblock sites that you have accidentally blocked (or that you want to include back in, if it cleans up its act).

There's more information in the help page and the blog, which announced it last month. The manual site list is a useful way of populating against known bad sites; note that the block option only gets shown if you first navigate to a site, then return back again; since Google's algorithms expect you to have gone to the site to discover its uselessness.

Broadband speed

2011-04-19T10:08:00.001+01:00

Tomorrow, I get fibre broadband installed. Well, fibre to the cabinet (rather than to the home). Got to be an improvement over where I am today:

Here's what it looks like after the install:

Consider me a happy easter bunny :)

Git Tip of the Week: Merging

2011-04-19T09:00:00.000+01:00

This week's Git Tip of the Week is about working with branches. You can subscribe to the feed if you want to receive new instalments automatically.

Merging

So far, we've just looked at committing content to the git repository. In other words, we've been building up linear histories, one commit at a time. Although there's nothing wrong with this – after all, most other version control systems you've been used to may work in exactly that way – there's a lot of power that Git bestows upon you with branches.

A lot of people avoid branching, if they've only been used to lesser version control systems like CVS or SVN. That's not because branching is especially different (although still an order of magnitude slower than Git) – rather, it's the lack of good merging that kills branching. As a result, branches are reserved for product releases, and then, begrudgingly.

Git, on the other hand, makes branching and merging free – so much that branching becomes a way of life in a Git workflow.

A merge brings together one more commits onto the current branch. Typically the commits are branch names, but they can be any hash in the repository.

A common workflow is to branch off an item of development to implement a new feature. In non-distributed systems, this is often represented as the “check it out but don't commit it” phase; but in Git, creating a branch is free and committing as you go is encouraged – a bit like autosaving your source files. When you're ready, you can merge it back onto the main development branch:

$ git checkout -b feature # implement feature $ git commit -m "Feature part 1 implemented" # implement feature $ git commit -m "Feature part 2 implemented" # test, check everything is OK # Switch back to master: $ git checkout master $ get merge feature # Optionally, delete the feature branch $ git branch -d feature

What we've done here is create a new feature branch feature, worked on it for a while, then switched back to master and merged the contents in. It's fairly common for this pattern to occur frequently; sometimes, with features overlapping each other. Don't worry about it; Git always keeps track of your changes.

If there's no problems, then git merge will automatically create a commit for you. If not…

Conflicts

Nothing in life is free, and merging is no exception. If you've changed the same file in two or more divergent commits, then you might end up with a merge conflict. This is like other version control systems; you get <<<<<<< ======= >>>>>>> markers in the code, like you'd see in other version control systems. However, you see these a lot less frequently in Git, because commits tend to be smaller, and in many cases, Git can replay re-orderings in your code.

To fix the merge conflict, edit the file, removing the merge conflict markers and resolving on a conflict-by-conflict basis, and git add the file. You can use git status to show you the list of files; those with merge conflicts will be shown separately, so you can step through these individually, fixing and git adding them as you go.

Once you're done, running git commit will commit all your merge changes. Unlike previous cases, where you have a single parent mentioned in the commit metadata at the top, this time you'll see two parents. These represent the previous version of this tree and the merged tree that you're bringing in.

There's more interesting things we can do with merged trees and merges, but we'll cover those in more detail another time.

Come back next week for another instalment in the Git Tip of the Week series.

Associated Business Solutions, Inc

2011-04-18T16:41:00.001+01:00

To anyone thinking of hiring the services of Associated Business Solutions, Inc:

Don't.

These guys have been spamming the MacFUSE forum repeatedly over the last few weeks advertising for positions that are utterly unrelated to OSX development or filesystems, and don't seem to take a hint in responses that their services are not required in this forum.

Anyone you hire from them is likely to have been picked up by scamming through forums and the quality of such candidates are likely to be unrelated to the position that they have been offering.

Specifically, they have been posting as “Faheem Ghouri” and “Abdul Aziz” from the domain name “absolutions.us” which, for obvious reasons, I'm not linking to directly here.

Please feel free to trawl the MacFuse mailing list to verify.

Git Tip of the Week: Branches

2011-04-12T09:00:00.000+01:00

This week's Git Tip of the Week is about working with branches. You can subscribe to the feed if you want to receive new instalments automatically.

Branches

So far I've only talked about code which is on a single branch, or in Git terms, the master branch. That's fine for getting started, but you will soon find that you want to create branches for parallel development.

First, it's worth briefly mentioning how Git handles branches, because it's a fairly fundamental part of how Git works. Every Git repository is a tree of commits, with the initial commit being the root of the tree. Every subsequent commit has one or more parents; so from any given commit, you can walk up its parent tree(s) to find the initial commit of a repository.

Since each of these commits is represented as a hash, like 9ab532… then any point on the Git repository's tree of commits can be uniquely identified by this number alone.

As a result, every reference mechanism in Git basically boils down to storing this hash value somewhere. For (local) branches, these are stored in the files refs/heads/*; you'll find a master file in there with a 40-character hash as its contents.

Working with branches

To display a list of branches in Git, run git branch. This shows the list of branches that are currently in your repository. The asterisk shows you which branch you're on at any time.
To create a new branch, based off the current branch, you use git branch name. If you want to base it off a different commit, you can use git branch hash name. To switch to a different branch, you use git checkout name. Here's what it looks like when put together:

$ git branch * master $ git branch new $ git branch * master new $ git checkout new $ git branch master * new

There's a shorthand for creating a branch and checking it out, which is git checkout -b name [hash]; so in the above case, we could have done git checkout -b new:

$ git branch * master # Shorthand for: git checkout -b new master $ git checkout -b new $ git branch master * new

You'll find that what's happened is there has been a file created, refs/heads/new, which contains the new pointer to the tree. Since we created one from the other, they've both got the same contents. However, if we were to commit a change to the new branch, then they would be different. The master branch would stay where it was, whilst the new branch would be one commit ahead.

If you were to checkout the master branch instead, and create a commit, then the branches would start to diverge. Alternatively, we might want to bring in that change from new to master, which we'll do with a merge next time.

Come back next week for another instalment in the Git Tip of the Week series.