The third and final day of QCon London started on-time with a brief overview of the track hosts and then jumped into Resilient Response in Complex Systems. An ongoing theme of the conference is ‘Fail happens; deal with it’ and the optimisation of Mean Time to Recovery is far more important than Mean Time Between Failure. It’s also important to regularly drill debugging so that people build the skills necessary to cope with the operation under pressure; in fact, there’s a lot of similarity with other high-stress situations, such as in Naval, Electrical or Air Traffic environments – except, as John Allspaw noted, that “Other than the killing people part, they’re not all that different from web startups”.
An important part of learning is to both learn from mistakes (especially with retrospectives/post-mortems) but also learn from near-misses and success stories. There’s more opportunity to learn from success (with any luck, they happen more often) so taking the time to understand what went right (and what didn’t go wrong) is a valuable learning exercise.
Perhaps unsurprisingly, any talk with ‘GitHub’ in the name was destined to be popular. This talk was no exception, with about 250 people cramming themselves into a room destined for 150 people maximum. This one was courtesy of Zach Holman, who has worked for GitHub for the past couple of years.
In it, he highlighted the fact that GitHub is a family-friendly company, with a flexible approach to working and one focussed on delivery rather than hours, or as he put it, ”Hours are Bullshit”.
GitHub’s view is to enable creativity, and recognises that creativity is unlikely to happen between restricted hours. In addition, it also realises that people around the globe have different working hours, and as such, that not everyone is going to be in the same place at the same time. In fact, being asynchronous is the only way of solving that problem.
There were also some interesting nuggets; for example, for video conferencing they have a couple of iPads hooked up to a few flat-screen TVs for connecting others in different regions with the main office. They also have a couple of iPods hooked up to TVs dotted around the office, such that the web browser can be remotely driven and showing a web page, which can be instantly updated from anywhere to show the same information across the globe. This also extends to the music jukebox, which has an automated DJ playing a music track, allowing those in other parts of the world to listen to the same music playing – even if they aren’t in the same office as others – to build a company culture.
A consistent approach to measurement means that anyone can see how the servers are responding under load, and if there is a requirement to drill in to details then these are only a couple of clicks away. As part of their open process, Zach highlighted the recent SSH problem and showed the spike in SSH failures and failed pushes immediately following the reset and the period immediately thereafter.
The final point of that talk is that GitHub optimises for happiness, and that although you may not be able to make all the changes in your own organisation that at least it is possible for you to try and enact some of the changes.
Scala and Functional Programming
This track got much more attention than some of the others (primarily because it mentioned the word ‘Scala’) and even had Philip Wadler dress as Lambda Man. (Photo from @anagri.)
The Guardian has been an early adopter of Scala, and pointed out some of the successes and challenges that they had on their way. (A quick bit of audience research revealed about 5% utilisation of Scala in production at the current time, about 10% evaluating it and around 70% using Java for production use.)
Their previous incarnation was based on Java, Spring, Hibernate and Velocity, and included around 185k of Java, 35k XML, 72k Velocity templates and 250k Java test suite. With a two-week release cycle and a monolithic application, they found the size of the environment started to cause problems.
Having heard their earlier talk on modularisation, it was clear that the benefits they got were primarily as a decoupling of the systems, with a clear interface (HTTP+HTML) between them. Their micro-app SSI framework meant that requests were being processed and then farmed out to multiple individual components, and that the side-effects of the re-write were advantageous but mainly due to the redesign.
They moved from Java/Guice/Solr to testing in Scala (as a way of trying the waters before adopting wholesale) and then incrementally replacing Java classes with equivalent Scala classes as they were amended. IntelliJ was used to replace Java with Scala on demand until the critical point had been passed, when a final walk-through of the code converted the remains to Scala.
Throughout this process, the existing Maven build performed the hybrid Java/Scala compilation, although once everything was fully transitioned they switched to SBT to avoid problems with the otherwise slow Scala compiler. They also highlighted an issue with SBT in that whenever a new version came out, they had to rev all of the existing infrastructure to deal with any binary incompatibilities the upgrade may have brought.
Since most of their work is compiled from source (and thus, relatively little in the way of shared Scala libraries) they were not often burned by the lack of backward compatibility; in addition, since each of their network components is in an isolated process, each micro-app effectively has its own classpath and doesn’t need to be consistent with the other apps that may be running at that time.
Finally, some other points; they don’t hire experienced Scala developers (mostly because of scarcity, but also because of price) – however, they do hire developers with the right attitude and passion who can then become proficient in Scala. In their case, they aren’t using functional programming aspects but mostly view Scala as a more compact Java, without the semi-colons and with automatically declared constructors.
Graham pointed people to Scala School as being an interesting set of exercises for those wishing to learn Scala, as well as their open-sourced components at GitHub, including the Open Platform data APIs available from http://content.guardianapis.com.
He concluded with a slide which values:
- Clarity of intent over ceremonial abstraction
- Architectural evolution over revolution
- Craft over patterns
- Passionate developers over turning the crank
- Clarity over cleverness
The slides are available at slideshare:
The less said about LinkedIn, the better. They had a NoSQL and BigData talk which explained how they use a huge Oracle database at the back end, with Voldemort as a read-only cache sitting on top for common queries. Fortunately it finished about half an hour early so I was able to get some time back for more practical uses.
NoSQL and Grids
The final talk was by John Davies on grids, and how much of the finance industry has been doing this with utilities like GemFire, Terracotta, Gigaspaces and Coherence. His point was that SQL is very well suited for some tasks (like batch processing tabular data) but that it struggles with some of the more hierarchical data; and that SQL as a language isn’t that bad, but rather the uses to which it is put.
As an example, he showed a Java NIO Grep example which was performing a filter
of CSV records whose 7th field was Think; on a standard developer laptop, it
buckled at 100m data files and took 1.2s to process; but on a 1G data file, the
cut -d ',' -f 7 | grep Think performed the same search in under
He highlighted the uses to which grid computing is put; either as a large distributed memory cache (thereby taking a lot of memory distributed over many machines) or as a large compute farm (thereby taking a lot of CPU distributed over many machines). He also observed that invariably it’s easier to spin up a new VM to acquire this (possibly returning it when it’s not being used) than it is to go through the procurement process to acquire new hardware.
This has been my third QCon in London, and it’s getting bigger and better each year. Part of the attraction of QCon over other kinds of conferences is the diversity of the presentations over what might be seen in a language-specific or tool-specific conference. It’s possible to see a presentation on hard-core Java, go to a presentation on the Lambda calculus followed by an agile development process and round of the day with how a startup handles high volumes of data or adapts their working practices around the happiness of their employees. There isn’t another tech conference like it, and if you’ve been following the #qconlondon hashtag then you’ll have a feeling of what it’s like to be there too. See you next year!