Apple finally kill off ZFS

Earlier this year, I wrote about ZFS' demise on Mac OSX and its disappearance from the (as then) up-and-coming Snow Leopard. Since then, there had been little public communication on the ZFS mailing lists (as is Apple's wont for unreleased products) - but even once Snow Leopard was released, there was still precious little communication.

As of a few hours ago, an empty page on the old http://zfs.macosforge.org page has removed all the content and left the message "The ZFS project has been discontinued. The mailing list and repository will also be removed shortly." At least there's a final conclusion to what once was a highly regarded project, even if it was an abrupt and faceless ending to the project.

The last released version of ZFS was 119, though you may be able to find some bits for 10a286 which have been leaked that were part of the earlier developer builds for OSX (of which I have no idea, so don't ask me). Alledgedly the 119 bits still work on Snow Leopard, though there are still some bugs (similar to those on the Leopard install, one assumes).

In the vain hope of getting ZFS to live on, I've created the Mac ZFS project at Google. This is used as a project placeholder; the source, which is available under a CDDL (and APSL) license can't be held as part of the project. There's a zfs-macos mailing list, as well as some data rescued from the old wiki (including the Getting Started page). There's even a cached version of ZFS-119.pkg installer and zfs-119_src.tar the on the zfs-macos files page. Fortunately, the contents of the SVN repository was captured on GitHub; git:github.com/alblue/zfs-mac.git was cloned from peaceful who imported the SVN repository before its demise.

There have been many reasons as to why ZFS was killed off by Apple; whatever the reason, it seems that this is indeed the final curtain on OSX. A number of people are looking into both Free BSD and Open Solaris as replacements for their file-serving needs; after all, it's almost certainly better to have a lesser operating system that understands the filesystem data than a greater operating system that doesn't. Which all in all is a shame, because the latest Mac Mini Server has all the hallmarks of a system that would work well with ZFS support.

The most likely reason for the abrupt demise is stability. ZFS119 had been known to crash, particularly if there were multiple snapshots/scrubs/drops concurrently running. In addition, if there's a problem with a ZFS device, the system freezes; not an insurmountable one, but annoying none the less. There were also some reports of problems in the developer builds as well.

There's a couple of other theories relating to the file system. One is that there were patents with NetApp lawsuit that Sun was defending; though that was back in 2008 when that started. The other, probably more likely, is the (impending) takeover from Oracle; the Big O is not known to play softball when it comes to licensing; but again, the acquisition was announced back in April 2009 and ZFS wasn't yanked until July 2009. The most likely thing was that it wasn't ready for prime-time in time for Snow Leopard's golden master, which was already in danger of running late.

Not all has gone well though; in fact, now Apple is looking for a new file system engineer. Could it be that Apple is trying to re-invent ZFS on top of the beleaguered HFS? Or are they just going to bolt-on extras, like they've done with journaling, case-sensitivity, and more recently, compressed data sections and attributes. What of the existing filesystem team? Who knows, but Noel, if you're reading this, thanks for all the hard work you put into ZFS - I'm sure I speak for many people on the list when I say thanks for all the hard work, even if the corporation decided to go in a different direction.

However, it may be that there were one problem too many in making it happen. I'm going to conclude by citing a post from an Apple e-mail address to the list:

Without implying anything about why ZFS was not shipped with SnowLeopard, the discussion that starts out at minute 22 [of this video presentation by Jeff Bonwick] exposes some continued mis-understanding by Mr. Bonwick and crew. I have a great deal of respect for Jeff but I am a bit surprised that he's not aware of these issues.

At around 23:45 or so he says that "laptops never experience problems with flush-track-cache because they have a battery and thus never lose power". While it is true that laptops have batteries, machines still crash and/or get force rebooted. When a reboot happens the bus is reset and what the drive will do with dirty data sitting in its cache is undefined at that point. Regardless of what the specs say, some drives toss any unwritten data while others will keep it and eventually write it after the reset.

To be fair, we guarantee that all internal drives Apple ships will honor flush track cache so you're safe on a Mac laptop. However if you're running Solaris on some other brand of laptop you may or may not be out of luck.

Last, I do not believe that the crash protection scheme used by ZFS can ever work reliably on drives that drop the flush track cache request. The only approach that is guaranteed to work is to keep enough data in a log that when you remount the drive, you can replay more data than the drive could have kept cached. This is effectively what the journal on HFS has done since Leopard.

Scala is (still) not enterprise ready

A couple of months ago, I posted the question: “Is Scala ready for the enterprise?” with the answer ‘no’ at the time. I called out specific mention to the fact that 2.8 was once again not going to be backward compatible with the previous release; the post generated a number of comments on the subject.

Hoping to help Scala move towards becoming more suitable for the enterprise, I proposed modularity for scala. The goal was to take the existing runtime, break it down into component modules, and set up versioning between them. The idea was to be able to evolve modules independently of each other, so that a core kernel of Scala library support would be usable and then Scala systems could import modules on an as-needed basis.

However, despite posting a set of requirements (and for a while, hosting them in Wiki form on GitHub), it became clear that modularisation in Scala won't work whilst basic issues like versioning are swept under the carpet. Sadly, this just highlights the fact that the release engineering process of Scala is not sufficiently mature to base any kind of serious application on it; at least, not one that would track the core libraries changing going forwards.

The issue at stake is the subsequent version of Scala to 2.7. At present, it has been described as Scala 2.8 and brings a number of new features (named and default arguments) which themselves require a binary incompatible change to the way that methods are invoked.

There's a currently accepted definition of version numbers which almost everyone (bar Sun and Oracle) uses. That is formalised in the OSGi specification; major.minor.micro, where major increments represent a step-change in compatibility, minor increments represent backwardly compatible behaviour but often bring new features, and micro which adds no new functionality but fixes bugs. Sun shot themselves in the foot a long time ago with the Java 2™ trademark, which at a marketing stroke killed off any possibility of using a Java 2.x version in the future. (I've covered the folly of confusing marketing numbers and version numbers before.)

Although Scala has suffered glitches before (2.5 actors were incompatible with 2.6, for example) the ongoing progression of versions had largely worked. The introduction of generics caused the last biggest headache of Scala (in 2.7) but like the previous time it happened, it was seen as the last time we'd need to change backward compatibility.

Normally, a project would bump up its major version number to signify these changes. By rights, we should be on about Scala 4.0 by now. Consumers of Scala should then be able to determine whether they should upgrade to the latest-and-greatest, or accept the fact that they were on a binary incompatible version and continue on. Hiding such changes behind a 2.x stream is following exactly the same mistake that Java made back in 1997; an irrational fear of bumping the major version number. In fact, when asked on the scala-internals list, the responses included “We keep to the name 2.8 mainly because it was announced that way for some time. Switching to 3.0 now would cause confusion” and “I agree that too much has been said and printed about Scala 2.8 to, at this point, change it to Scala 3.0” – the latter of which was cited as “An even stronger argument”. It's pretty unlikely that a bump would cause any confusion whatsoever; after all, it's the case that both 2.8 and 3.0 would supersede 2.7. However, the interpretation of those semantics is important; someone choosing to use 2.8 might think that it is backwardly compatible, but would think 3.0 isn't necessarily.

This of course boils down to the key problem with Scala as it stands today; the version numbering system is based on number of previous uses of a term and gut feeling rather than a more objective measurement of the (in)compatibility of code. None of this will matter to the (early) adopters of Scala; they'll just notice when they upgrade their runtime and discover that it suddenly doesn't work.

So, back to modularity. Unfortunately, with no sane versioning system in place for the Scala runtime as a whole, it will be impossible to create a set of modules for the Scala language that are anything more than a shim which splits up the library.jar into a number of tightly coupled dependants. The only way for Scala modularity to survive would be by evolving modules independently; but since these are tightly coupled to the version of the compiler and runtime used, such parallel evolution will not happen in any meaningful way. In fact, the existence of implicits and the possibility of misuse of traits (without proper evolution procedures) will mean that instead of allowing modular development, the Scala compiler and runtime implicitly enforces a total lockdown of any versions of code that may run. Even now, the head of the Scala build is being co-compiled against both Java 5 and Java 6 because the output (and runtime requirements) for both are different. In any case, it doesn't make sense to pursue modularity in Scala whilst version numbering and binary incompatibilty between releases run rampant throughout the Scala codebase.

Scala is an interesting language. It challenges a lot of the preconceived notions that other languages have avoided; and it's surpassing the slowly fossilising Java language thanks to its ability to support functional constructs like lambdas and filter/map operations. However, the whole release process and long term viability of the language and runtime remains questionable even in the face of high-profile adopters. For the enterprise, the choice is clear – Scala is not ready for the enterprise.

iTunes LP and Apple TV

A recent report on Apple Insider on the exorbitant cost of iTunes LPs led to me thinking – why can't we just develop our own iTunes LPs? The format isn't documented at this stage, but it's essentially just a http://theappleblog.com/2009/09/14/itunes-9-deconstructing-lps-extras/">web application in a can. That led me to think that perhaps using iTunes LP to deliver a more interactive experience over Apple TV (or FrontRow) would be a good idea.

However, it appears that Apple TV doesn't support iTunes LPs. In fact, they can only be viewed inside iTunes and not standalone. That's a bit of a let-down and means that the iTunes LP format isn't quite as useful as it might have been.

Ripping media for Lexus 400h SEL

The documentation of how/what to find (in terms of ripping media) for getting video onto a Lexus multimedia system (like in our 400h) is non-existent. Furthermore, the UI is an abomination and is more Lidl than Lexus. If you've managed to save yourself from buying the SEL and/or multimedia kit that comes from Lexus, congratulations – if you haven't yet, then please save yourself the bother and buy something else.

Anyway, we've got this now and so I wanted to put some media onto a USB stick (since the kids keep losing the DVDs, and most have got these naff menu systems) but needless to say, no-one at Lexus had a clue as to what I could put in the USB stick. Eventually, I managed to get it to work, by using a FAT-formatted USB stick and HandBrake to convert the media. I needed to set the output format to AVI (and end the filename with .avi, as otherwise it didn't show) as well as use the MPEG-4 (XviD) codec. I used an average bitrate of 1000kbps which looked reasonable on the small screen - the audio was set to MP3 128kbps using the LAME encoder.

Unfortunately, there is an overlay which shows a countdown of how far you are into the current track, which I've not managed to disable yet; if I ever figure out where the manual is then I'll update the post.

Finding download location of firmware

I've been asked how I find the download of the Apple firmware for iPhone updates. There's no magic in it - all you need to do is run an HTTP proxy, and configure your network preferences to go through that, and you can read off the logs where you're accessing it from. However, there's an easier way.

The Apple software update site URL is http://phobos.apple.com/version, which returns an XML file containing the locations of products that are downloadable. There's quite a lot of XML verbosity in here, but you can easily find out from this list which ones are relevant to you. For example, if you wanted to look for iPhone updates, then you can use grep to find out. Here's what I do:

• Get the current version file

  curl -L http://phobos.apple.com/version

• Grep it for iPhone installs

  egrep iPhone.*ipsw

• Strip out the tags

  awk -F'>|<' -- '{print $3}'

• Sort and uniquify

  sort -u

You can do this all in one line by piping, or write intermediary results to files to taste. Here's what it looks like when it comes together:

curl -L http://phobos.apple.com/version | egrep iPhone.*ipsw | awk -F'>|<' -- '{print $3}' | sort -u

Writing the blog entry takes a little longer ...

Apple releases iPhone OS 3.1.2

Just a couple of numbers short of Pi, Apple have released iPhone OS 3.1.2 which fixes the (quite frankly) horrible bugs that crept in with the 3.1 update. I've been suffering from the wake-from-sleep mode hanging the connection, as well as the odd delays and jitter (not to mention random menubar-sized gaps at the top of some screens). Given that 3.1 has been the buggiest release so far in terms of usability, I'm really hoping that 3.1.2 fixes all of these problems, and that I can update in peace. I'll update this post once I've updated and used it for a few days to confirm whether it does, in fact, behave as it should afterwards.

Links to downloads:

• iPhone1,1 (Original)
• iPhone1,2 (3G)
• iPhone2,1 (3Gs)