Alex headshot

AlBlue’s Blog

Macs, Modularity and More

Why OSGi qualifiers aren't working

2012, eclipse, osgi

In my previous post, I argued that the decision not to support SNAPSHOT ranges in OSGi was a missed opportunity for the platform, perhaps one that will limit its adoption in the future. Despite several vocal critics (largely arguing from a point of not having used Maven and its SNAPSHOT versioning scheme), I decided to see how much of a problem it really was.

The arguments of ‘Maybe this would cause problems’ don’t really seem to be borne out by the Maven repository, which has over 400G of versioned artefacts that have gone through the -SNAPSHOT process during development and rarely, if ever, has any problems in versioned code. (That’s not to say mistake can never happen; but it’s easy enough to publish a new asset with a new patch version if required, as has been done occasionally before such as the javax.jms releases.)

The problem with the existence of the OSGi build qualifier is that it means Eclipse developers don’t actually take care to version code properly. Instead, the build process just spits out recompiled bundles, often without a corresponding bump in version number where required or needed.

Conversely, JARs published to Maven Repo (many of which already are bundles, thanks to the maven-bundle-plugin and others such build tools) all have the correct version defined outside of a -SNAPSHOT space – not the least of which is that there’s a process in place which prevents the publication of -SNAPSHOT releases to Maven central (or ‘The Central Repository’) as it is now known. This means that once a bundle is published with its final version number, and its corresponding SHA hashes generated, can never be changed. Immutable bundles are good.

So, I took a look at the Eclipse Indigo composite repository (i.e. including 3.7.0 as well as 3.7.1 and 3.7.2) to see how widespread the problem was. The results surprised even me. Of the 9000 bundles that make up Indigo 3.7.0–3.7.2, over 950 of them were published with differences in the qualifier only. This, above all, seems to violate both the intent and the practice of good versioning processes. The semantic versioning specification requires the patch version to be bumped when the bundle is changed, as does the OSGi recommendations and the Eclipse recommendations. Clearly, this is not happening when over 10½% of bundles in Eclipse – including many in the ‘core’ bundles such as PDE, Equinox and Core – are not bumping versions appropriately.

When I tweeted about this finding, BJ Hargrave of IBM and OSGi director claimed that this indicated OSGi versioning was working well, and that the bundles had ‘simply been rebuilt’. This seems to be missing the wood for the trees; if there were no changes, why did it need to be rebuilt? Conversely, if there were changes, why wasn’t the version bounced?

I looked into the Equinox framework admin bundle, which was rebuilt, and looked at the differences between them (org.eclipse.equinox.frameworkadmin.equinox_1.0.300.v20110506.jar vs org.eclipse.equinox.frameworkadmin.equinox_1.0.300.v20110815-1438.jar). The only changes in this bundle were a stamping of the Bundle-Version, the Eclipse-SourceReferences element, and a change in the api_description to point to the new number. There were no changes in the bundle code itself. So this 60k bundle is completely meaningless when compared to its predecessor; it’s just extra bits sitting in everyone’s Eclipse instance, on Eclipse update servers and taking additional bandwidth when updating. OK, so 60k on its own isn’t much; but if, as BJ argues, all of these 950 bundles are ‘just rebuilds’, what is the point?

I argue that this is not only a flaw in the build process, but also in the OSGi versioning as well. If these were built using -SNAPSHOT style versions, the version would have been bumped to 1.0.300 in May 2011, and only if required would a 1.0.301 be needed. Since the process when dealing with -SNAPSHOT style versions is to bump,release,bump then it would have been clear if any changes were needed, and the timestamp would not have been necessary. You don’t get multiple releases of Felix where ‘it has just been rebuilt’, for example.

The same effect appears to be visible with the org.eclipse.pde.runtime_3.4.201 differences; again, it appears to be due to a commit of the api_description, Bundle-Version, Eclipse-SourceReferences and so on. Again, the use of the bundle qualifier is masking the real problem; there’s a build process which is updating tags on files via Eclipse-SourceReferences and api_description needlessly, then releasing the changes into the update site. This also causes a knock-on effect with the performance of the P2 repository, which stores all of these duplicated entries and is queried whenever Eclipse is updated; worse, whilst the bits of the duplicated JARs are downloaded only once (at install/update time), the P2 metadata is updated even when you don’t install that bundle.

Having a .qualifier extension where you can spin up a new build of a bundle every time masks the real problem, which is that it’s possible to generate a new bundle with potential changes without changing the version number. Furthermore, if you do build a new version, it will automatically be picked up by P2 even if the content is nearly identical (as the above changes show). Picking up the latest version is necessary during development phases – the -SNAPSHOT phase in Maven terms – but once the content is locked down, there’s no need to ever rebuild a new version.

Finally, there are changes which should have required a change in version number which – like the others above – have slipped through the net. The org.eclipse.emf.ecore bundles ( 2.7.0.v20110912-0920 vs 2.7.0.v20120127-1122) show a change in the (public) org.eclipse.emf.ecore.resource/impl/URIHandlerImpl class, although this actually corresponds to no significant change in the source code itself.

All of these errors have been hidden by virtue of the automatic build process spitting out (in essence) permanent SNAPSHOT-esque qualifiers from the Eclipse PDE build system. Even if there are no changes, rebuilding a bundle causes a bump in the qualifier, which in turn leads to additional downloads even where none is required. This would not happen if OSGi supported real -SNAPSHOT version ranges, because the bump-to-prod would have identified that there was no need to rebuild that bundle, having already been published at a specific version.

Unfortunately, even though the ‘early access’ moniker is designed to make the specifications available prior to publication, there is no way this will be changed at this point. Ironically, it was snapshot version ranges which brought in the reason for bumping the OSGi release version to release 5; now that the SNAPSHOT releases have been removed there is little reason to still view this as R5 instead of an R4.4 release.