Alex headshot

AlBlue’s Blog

Macs, Modularity and More

QCon London 2014 Day 2

2014 Qcon Conference

Day 2 at QCon started off with a talk by Tim Lister (co-author of Peopleware). One of the great things about QCon is that you get to meet interesting peope from all walks of life, and I had the pleasure of chatting with Tim at drinks the night before.

Forty Years of Teams

In 1987, the book “Peopleware: Productive Projects and Teams” was published, written by Tim Lister and Tom DeMarco. The third edition was published in June 2013, and it’s a book which has become a standard for understanding team dynamics.

Tim’s main observation was that most problems with projects are not a result of any single or sudden change, but of problems with people. He used the phrase “Dead Fish” to identify projects which were clearly identifiable as failures for years before the ultimate fail point, but despite being common knowledge the project is kept alive. Economic interests, power plays and a need to meet bugetary quota are some examples of where clearly failing projects are kept alive to achieve side-effects other than the project’s success.

He mentioned the benefits of working in a great team – “it’s great to know your colleagues have your back” was the phrase he used. Working for and with people smarter than you can result in successful delivery and learning; on the other hand, “if you don’t have the trust of the team or you aren’t excited to go to work then you should use your feet.”

Finally he inspired many by talking about software and being one of the biggest changes in the lsat 50 years, and how conferences such as QCon are about moving an idea from one thick skull to another. He also suggested that writing is a great way to distill ideas and opinions (which reminds me, I need to blog more …)

Maneuverable Architecture

Michael Nygard talked about maneuverable architecture, and started by talking about maneuverability based on the investigation of John Boyd who looked at aircraft performance and how quickly they could respond in dogfights by converting kinetic energy to potential energy and vice-versa.

In addition, he proposed an Observe, Orient, Decide, Act loop (also known as OODA) which is the process of deciding what to do; specifically, by speeding this up or reducing the time taken to perform the actions can result in being able to react faster.

The same thing can happen in a business context, by being able to reduce the design/implement/test/release cycle. By being more agile (or maneuverable) it is possible to reduce time to market and thus costs.

One of his observations was that immutable data is easier to share and scale than mutable data. By using identifiers consisting of hashes of the content, the state of a system can be externalised into a list of identifiers. This is similar to how Git works in terms of referencing content. His example was of a shopping cart, with an identifier based on its content, and then a state reference from a user to point to the current version of a basket (referenced by its ID). By namespacing IDs it is possible to avoid content collisions based on different types.

This allows a system of micro services (aka μservices) to be built and a standardised API to be able to interact with them. By layering the services together it is possible to implement security or filtering to the requests.

The argument is that by decomposing services and making the feature an emergent property of the system instead of a hard-coded one (for example; a process which periodically calls a URL or a service that sends a mail, instead of a single process which walks through and mails based on content of the server).

APIs and Platform as a Service

Paul Fremantle talked about APIs and platform as a service (PaaS) as being the next stage in the evolution of virtualisation. Instead of VMs and OSes being virtualised (Infrastructure as a Service) the next layer up (Platform) is being virtualised. An API is like a virtualised service itself; by providing a connection point between two services. He also linked to A company without an API is like a computer without internet.

Paul also observed that having an API and a way of monetising that API is necessary for being able to ensure growth. He used Twitter as an example of a service whose API got massive growth but without an initial monetising plan, which then had to be retrofitted. But even with a monetisation plan in place, having a free sign-up for developers (or at least access to a developer or test environment) is a way of ensuring that developers can get started with as little friction as possible.

To build an API-first system, start with developing an API and ensure that all clients go through that API. That way, the system’s contents is exposed via the same API that everyone uses, which means that there aren’t first-class versus second-class applications. In addition, as new features are added to the API then all clients can take advantage of it.

One example of the APIs he referred to was OAuth2 and OpenID connect (and linked to a PDF report prepared by his company on Building an Ecosystem for API Security).

He concluded by noting that everything is evolving to virtualisation.

  • APIs are the virtualisation of functions
  • PaaS is the virtualisation of application development
  • Forge/Cloud toolchain is the virtualisation of development
  • DevOps is the virtualisation of deployment
  • Enterprise Store is the virtualisation of application delivery

Migrating to Micro Services ##

Adrian Cockroft talked about his experience of migrating Netflix to micro services (whilst wearing Google Glass). His key enabler is that speed wins in the marketplace; by skating to where the marketplace will be you can out-innovate the competition.

To do this, reducing friction from product development is the key. Automating everything and trusting developers allows them to innovate without needing meetings, approvals, sign-offs, cross-checks, or anything else which gets in the way of the desired result – providing functionality to serve the business.

Importantly, by focussing on what you are good at (and what’s core to your business) you can let go of things that you aren’t; including using open-source tools instead of trying to fork/wrap your own APIs, and making the non-secret sauce parts of your environment as open-source applications. Doing so increases your reputation of the company – your company’s repositories on GitHub is its on-line resume;. By licensing permissively (Apache, MIT, BSD etc.) you trade-off control for ubiquity.

By doing this, Netflix were able to take full advantage of Amazon’s cloud infrastructure, and provisioned 192TB of SSD storage based on a hallway conversation in 20 minutes, and within the day had performed load testing across two data centres with replicated content totalling 18TB of data replicated in less than 5 hours between regions, and then load testing with less than half a second delay in replication between the east and west coast.

Typical reactions to the prior Netflix talks:

  • You guys are crazy! Can’t believe it (2009)
  • What Netflix is doing won’t work (2010)
  • It only works for Unicorns like Netflix (2011)
  • We’d like to do that but can’t (2012)
  • We’re on our way using Netflix OSS code (2013)

By reducing all friction in the decision cycle, developers can innovate faster and get changes into production.

In Netflix’s case, they achieved this by automating their deployment and testing processes, so a full stack could be provisioned on the fly. By building microservices, and having each developer team responsible for their own services, it is possible to build a complex ecosystem from modular parts. These services can be versioned so that all calls are using a fixed version; this allows a production version of the service to be used whilst test versions are used by a sample of users or for testing live in production.

The recommendation is to provide solid client APIs for connecting to services as well as the service itself; this reduces friction in the users who need to connect to the service to interact with the runtime. Adrian suggested that MongoDB was successful over Cassandra because the former had plenty of good drivers in many languages, although with investment the Cassandra service may be better in different circumstances. Using non-blocking calls with Reactive Java and using Netty for non-blocking IO gave an increase in performance without being overly concerned about threading issues.

Netflix’s code is available on GitHub and they blog extensively.

Lambdas and Streams

Simon Ritter talked about Lambdas and Streams in Java 8, due out at the start of EclipseCon 2014 in a couple of weeks.

For those that aren’t familiar, the Lambda expressions will be able to provide a similar change as inner classes, allowing in-line functions to be assigned to a Single Abstract Method (SAM) type. These will actually be functional types rather than instances of classes, so they don’t have this.

Runnable stuff = () -> System.out.println("Hello Lambda");

He also covered the new streaming API which allows a sequence of collection elements to be processed in a stream, much like an iterator allows elements to be processed. However, by moving this into a stream it’s possible to perform stream combinations such as map, filter and flatMap to construct different streams on the fly. Importantly, these streams are lazy and so aren’t evaluated until the end of the process (the terminal condition) such as sum(), collect() etc.

Java 8 is shaping up to be the most significant release since generics were added in Java 5. Although not as functional or flexible as languages like Scala, it will be a great help to developers who continue to use Java.

Back to the Future

Emma Langman explained why my train home to Milton Keynes was delayed two weeks ago. Time for a drink.