Alex headshot

AlBlue’s Blog

Macs, Modularity and More

Multiple Architecture, Single Build

2011, eclipse, rant

One of the things you take for granted on a Mac is that applications will Just Work, regardless of what the hardware you're running on is based on. OSX has always been able to create fat binaries – that is, binaries which have more than one processor variant type – and it selects the one appropriate for the system you're running on. In the days during the PPC to Intel switchover, this gave a seamless transition – but even now, we still use fat binaries which contain both 32-bit and 64-bit executables.

OSGi has the ability to be packaged with more than one processor type, and either using OS-specific fragments or headers in the Bundle-NativeCode will select the appropriate version for a given native library. Some libraries (like SWT) are highly coupled to their native library implementation, whilst others (filesystem resources) provide additional implementations but may fallback gracefully if not present.

Per-platform builds

Why, then, do we waste terabytes of the Internet's bandwidth by packaging up multiple types of Eclipse applications, each customised for precisely one combination of operating system and yet every one of them containing the same 98% of pure Java code? Even multiple windowing systems can co-exist peacefully in an Eclipse installation; if the bundle for the required software is not applicable to that platform, it doesn't get installed.

All of this is bundled into a minor “native launcher” that's dependent on the OS, the size, and the windowing system (and the latter only because it has to ask for a default workspace and show a pretty splash screen). And the mechanisms which churn out these products (which largely have remained the same since before Eclipse was OSGi based) still spew out that same 98% of Java code for each build before writing out a platform-specific eclipse.exe. Oops, downloaded the 64 bit Eclipse instead? Back you go, wasting everyone's bandwidth with a 300M+ download just to get a new 70k eclipse.exe.

Fortunately, sanity can be restored. If you are in the position to create an Eclipse-based product, then do yourself a favour – when it has finished building its variants for all of the different systems, just merge the folder contents together. There's no reason why win32.win32.x86 and win32.win32.x86_64 can't sit together in the plugins directory; after all, only one of them is valid at any one time.

Bundle configurator

The only thing you need to be mindful of is the bill-of-materials launcher, stored in configuration/org.eclipse.equinox.simpleconfigurator/bundles.info. This is a text file, containing all the bundles that will be installed into the runtime. It doesn't matter if the plugins directory contains things; unless they're in bundles.info then Equinox doesn't load them. Think of it as the simplicity of Apache FileInstall coupled with the ls-lr of yesteryears' FTP sites.

The only reason I mention it is that when you run the build-everything-a-few-times step to get your almost-identical-build-folders, there are some subtle differences. You'll notice that the SWT for 64-bit Windows is different from the SWT for 32-bit Windows, for instance.

If you just do a blind copy over of everything, then what will happen is you'll end up with a list of plugins that contains the right contents, but the bundles.info from only one of them. When you try and launch on the other platform, P2 will helpfully tell you that SWT doesn't exist, despite the fact that you can plainly see it in the folder. P2 only believes what you tell it to, not evidence of existence, so you have to tell it to believe that there's an item in the plugins directory for it to work.

Fortunately, the bundles.info is a plain text file with key=value pairs; so you can in fact just concatenate the two files together. This will leave duplicates; but in fact, it's a line-oriented text file so you can simply do a sort and uniqueify operation to ensure that every bundle is mentioned once. (You need to be careful in general about different versions; that optimisation is left as an exercise for the reader.)

The other thing you have to take into account is the native launcher itself. Whilst each native launcher has a corresponding library (which can co-exist), there can be only one eclipse.exe at the top level. The simplest way around this is to rename it Eclipse32.exe for the 32-bit version and Eclipse64.exe for the 64-bit version (and correspondingly, other versions such as the wpf windowing system).

Cut down launcher

With this, we have the ability to make a cut-down launcher, which can run on multiple systems, in a single download. Yes, we would add slightly to the total installed download size versus a single OS but the overall savings would be immense. All it really needs is to be based on org.eclipse.platform, along with org.eclipse.equinox.p2.ui and possibly the marketplace client, and you have an all-in-one download which will be identical for every Eclipse IDE based user.

You can then switch over to the P2 based installer for getting the latest and greatest; and when someone needs to upgrade to the latest version, even if they've got the multi-OS older version, they can still use that to bootstrap a P2 installer client with the right data in place. For example, the RCP binary is only 56MB.

Furthermore, we can lose the source bundles for this all-in-one download, whose pack'ing is utterly pointless (non-Java JARs don't benefit from packing versus just GZipping in essence) and just download executable code. For those that need it, source on demand is the way forward.

The same argument extends to other operating systems. Whilst a Windows and Linux build could cohabit (their launchers are eclipse.exe and eclipse), the Unix variants might need to have their own locations. Mac users would be fine; we could just have the Eclipse.app and ignore the *.exe cruft in the top level.

Bloody Large Zips

So, who's with me in trying to stop the proliferation of Bloody Large Zips on the Eclipse mirrors containing the same content repeated time after time after time? It's really no wonder that Eclipse mirrors are pulling out due to disk space requirements (or why Denis is campaigning for more disk storage) – the problem is the culture of Eclipse builds producing the same stuff over and over again.

All we really need is a cross-platform, all-in-one IDE solution with P2/Marketplace client, and a P2 repository to point it to. EPP would then become a feature selection choice on the Welcome page for you to install your own content. Look at the list of downloads for the Eclipse 3.6.2 platform. I count around 50 installs of the base platform; so that's 50 copies of org.eclipse.osgi all the way up to org.eclipse.ide at a minimum. And that's without the EPP builds doing exactly the same thing. Just look at the SDK:

InstallSize
eclipse-SDK-3.6.2-aix-gtk-ppc64.zip 170.9M
eclipse-SDK-3.6.2-aix-motif.zip 169.9M
eclipse-SDK-3.6.2-hpux-motif-ia64_32.zip 169.8M
eclipse-SDK-3.6.2-linux-gtk-ppc.tar.gz 170.5M
eclipse-SDK-3.6.2-linux-gtk-ppc64.tar.gz 170.2M
eclipse-SDK-3.6.2-linux-gtk-s390.tar.gz 170.1M
eclipse-SDK-3.6.2-linux-gtk-s390x.tar.gz 170.2M
eclipse-SDK-3.6.2-linux-gtk-x86_64.tar.gz 170.6M
eclipse-SDK-3.6.2-linux-gtk.tar.gz 170.5M
eclipse-SDK-3.6.2-linux-motif.tar.gz 172.8M
eclipse-SDK-3.6.2-macosx-carbon.tar.gz 169.9M
eclipse-SDK-3.6.2-macosx-cocoa-x86_64.tar.gz 170.0M
eclipse-SDK-3.6.2-macosx-cocoa.tar.gz 170.1M
eclipse-SDK-3.6.2-solaris-gtk-x86.zip 170.2M
eclipse-SDK-3.6.2-solaris-gtk.zip 170.3M
eclipse-SDK-3.6.2-win32-x86_64.zip 171.0M
eclipse-SDK-3.6.2-win32.zip 171.0M

This is almost 3G of data; when even assuming a 2M platform specific size would indicate it could probably all be compressed into a single 200M download. (In comparison, the size of the Helios P2 mirror is around 5G; and that's including the 3.6.0, 3.6.1 and 3.6.2 releases.) Not only that, the downloads are likely to be faster; instead of having to download from a single server, downloads from P2 repositories have the option to create multiple connections to multiple mirrors to resolve the data.

Conclusion

We need to stop the packaging proliferation that is damaging Eclipse's mirrors, and make it easier to have a bootstrapped environment install on demand the content that's needed. A suitable welcome/intro screen will probably be the fastest way to market; and though packaging has been a success in terms of advertising to different types of users, we should move this advertising page into the Eclipse welcome screen once the bootstrap process has run, and not as a set of pre-prepared zips on the server.