buckd

Mon 18 August 2014 by Jim Purbrick

BuckGraffiti

One of the things I’ve been working on since joining Facebook is Buck, an open source Android & Java build tool which is significantly faster than many other Java build tools for a number of reasons.

As well as being fast, Buck gains a lot of power and flexibility by using Python to generate build rules. Once projects become very large, however, this can become a problem as Buck has to execute thousands of python scripts to build its dependency graph before it can start its parallel build process. When I started working on Buck this parse phase could last tens of seconds. Buck was already much faster than Ant, but test driven development could be painful.

Our initial work focussed on making the parsing step faster and after some experimentation with Jython we discovered that bigger improvements could be made by running a long lived Python process which could be handed build files to execute as required.

As is often the case, the bulk of the improvements could be made by caching. Build files change far less often than source files, so caching the build file output avoids the need to spend a lot of time parsing in the common case when only a small number of source files change. After spending some time looking at serialising the build file output to disk it became clear that a more effective approach would be to cache the output in memory by running Buck as a long lived server process using Nailgun.

Nailgun is a client, protocol, and server for running Java programs without incurring the JVM startup overhead. Nailgun makes converting Java applications to client-server architectures as simple as passing the name of the class containing your Main method to the nailgun Server and client application. Early experiments running Buck with Nailgun showed a lot of promise, allowing us to reduce parse time to close to zero, but running buck as a server invalidated several assumptions that required a non-trivial amount of work to fix.

The environment had to be threaded through from the client and calls to System.getenv() replaced, System.exit() could no longer be used for garbage collection, so resource lifetimes had to be managed with try-with-resources blocks and Nailgun needed to be extended to detect client disconnections which could be thrown as InterruptedExceptions to ensure that killing the Nailgun client cancelled builds as expected. It’s also worth noting that for large, long running applications like Buck the JVM started overhead saved by Nailgun is not significant, but the time saved by the long running process being able to maintain a JIT cache of Java class files is.

With Buck running as a long running server process the next step was to make it correctly invalidate cached build rules when build files changed. In order to avoid building outputs each time a file is touched, buck hashes the contents of input files to see if they have actually changed. While this saves a lot of time when switching between similar source control branches it requires reading each input file each time a build is run: something which was adding several seconds to the per-build overhead that we were trying to reduce.

To avoid this overhead we switched to a composite approach which watches the file system for changes and then checks the hashes of the contents of changed files. In the case where a few files are edited only a few hashes are generated and compared, in the case where source control systems touch many files without changing their contents comparing hashes avoids unnecessary rebuilding.

Initially we used the standard Java WatchService to generate file change events, but found that in practice the latency between changing a file and the FileWatcher generating events was far too high. Luckily wez, sid0 and friends had built Watchman which provides very low latency file change events and an easy to use JSON based interface which only took a day to wire in to Buck. Watchman is an amazing piece of technology, but requires some tweaking of OS settings to work well, so if you notice Buck taking a long time to parse project files you may need to check the system specific preparation.

When combined with exopackage and a number of other optimisations, the benefits of the Buck daemon are significant. Trivial builds now take a small fraction of the time they used to and in some cases it’s possible to incrementally build and install of an app in less time than it takes some build systems to do a no-op build.

I’ve had a great time working with the amazingly talented Buck team and I’m very happy to see buckd improving build times within Facebook and beyond.

Now it’s time to go back to writing a test, watching it fail and making it pass: with a lot less waiting around.


Organisational Structures

Thu 20 March 2014 by Jim Purbrick

There have been a number of blog posts recently about exciting new organisational structures. As Cory points out “Every early stage company thinks it has reinvented management”: a very dangerous belief when betting on a new organisational structure can be much riskier than betting on the wrong product.

It starts ...

read more

Beyond Time Dilation?

Wed 29 January 2014 by Jim Purbrick

The Battle of B-R5RB

EVE online is a remarkable game. On Monday over 2000 people spent over 20 hours destroying virtual spaceships worth 200,000 USD in real money in what was the likely the largest battle in a video game ever. That EVE is capaple of supporting such large engagements is an amazing ...

read more

Osprey Therian

Sun 15 December 2013 by Jim Purbrick

In mid-2004 I first started exploring Second Life. Version 1.4 had just been released and Philip Rosedale had said in the press release “My fantasy is to be Uma Thurman in Kill Bill, and now I can. I’d pay $10 for her yellow jumpsuit and sword moves and ...

read more

Parse By The Sea

Sat 19 October 2013 by Jim Purbrick

#parsebythesea

A few weeks ago Facebook London hosted the Parse By The Sea hackathon at the Brighton Dome as part of the Brighton Digital Festival. The idea was to take one of our internal hackathons on the road and invite members of the public to join us, turning a hackathon in ...

read more

Facebook Hackathons

Mon 16 September 2013 by Jim Purbrick

I’ve been a big fan of hackathons since one of the first Yahoo! Hack Days I attended at Alexandra Palace was struck by lightning. The lightning caused the fire alarms to go off which opened the roof to let the torrential rain pour on to hundreds of geeks and ...

read more

Brighton Digital Festival

Wed 04 September 2013 by Jim Purbrick

The Brighton Digital Festival starts this week and I’m very happy to be helping out with Facebook London‘s contributions: Parse By The Sea, a mobile app Hackathon featuring Parse on the 26th of September, and helping to Connect The Brighton Digital Festival by sponsoring Metranet to provide high ...

read more

Final Score

Thu 04 July 2013 by Jim Purbrick

Google Reader

Using Reader on my HTC Wizard on the loo was probably responsible for my biggest increase in clue ever.

Goodbye Reader, you’ll be missed.

read more

One Universe, Many Scales

Thu 10 January 2013 by Jim Purbrick

One epic meta-game design I first remember talking about a decade ago while working on Warhammer Online is the multi-scale online game: a system of interconnected games in which you choose to be a solo operative, work in a small group, or command epic forces or huge space fleets and ...

read more

Creatarr

Wed 09 January 2013 by Jim Purbrick

cc image by vdu, j4mie

One of the things I’ve been tinkering with since leaving Linden Lab is Creatarr: a creative, collaborative social game. Creatarr’s goal is to bring some of the magical collaborative creation found in Second Life to a wider audience and to push creativity in ...

read more

Following In My Father’s Footsteps

Mon 12 November 2012 by Jim Purbrick

Tintin Hair

From 2 years before I was born, until just before I started working on Second Life at Linden Lab, my Dad worked at an innovative technology company with a large consumer photography business: Kodak. From January next year I’ll be working at an innovative technology company with a large ...

read more

Caching Shared, Private Data With Ningx

Sun 11 November 2012 by Jim Purbrick

As with many other social services, a large amount of the data in EVE Online and Dust 514‘s New Eden universe is shared between subsets of users. Some corporation data should only be accessible to the corporation’s members, market prices should only be accessible to capsuleers and infantry ...

read more

Adding Vary Header Support To Nginx

Sun 14 October 2012 by Jim Purbrick

Although Nginx supports proxy caching it doesn’t provide support for the HTTP Vary header out of the box. This is a problem if you want to use Nginx to proxy different versions of the same URI which Vary on Content-Language or proxy different representations of a RESTful resource specified ...

read more

Load Balancing Stateful Services With Nginx

Mon 30 July 2012 by Jim Purbrick

The EVE online network architecture uses stateful proxy servers which manage sessions for players connected to the cluster via the EVE client. The client sends requests to the proxy which are forwarded on to sol servers maintaining the game state and the sols send notifications to the proxy which are ...

read more

Brighton Mini Maker Faire: The Movie

Thu 24 May 2012 by Jim Purbrick

A great video of the Brighton Mini Maker Faire last year by Andrew Sleigh showing the making of You’re The Boss 2. Applications for this year’s Maker Faire are now open and I can’t wait to see what everyone comes up with this year!

read more

Super Hyperpolyglot

Sat 05 May 2012 by Jim Purbrick

A few years ago nearly all the code I wrote was in C++, but increasingly I’m finding myself writing in a variety of mostly C-style languages and having to perform crunching mental gear changes as I switch between them.

In the interests of making these language switches less painful ...

read more

100 robots Vs The Audience

Wed 04 January 2012 by Jim Purbrick

A couple of years ago I had great fun putting together the London Geek Community iPhone OSCestra at Open Hack London and I’ve been controlling Ableton Live with iPhone tapped to my guitar as part of 100 robots for a couple of years now so when @andybudd suggested I ...

read more

100 robots Attack!

Fri 09 December 2011 by Jim Purbrick

Lots of exciting 100 robots news! Our debut album, Attack!, has been professionally mastered by Chris at Melograf Mastering who has done an amazing job and made the album sound incredible. The new version is already available at bandcamp and will be available on itunes, amazon and many other download ...

read more

The JavaScript Jungle

Mon 03 October 2011 by Jim Purbrick

There was a slide in the early talks that Cory Ondrejka used to give about Second Life about alien abductions in Second Life. One of the most exciting moments in Second Life for the early Lindens was when a resident constructed a UFO and flew around the world abducting other ...

read more

Data Is Not Art

Sat 01 October 2011 by Jim Purbrick

This week I experienced two remarkable combinations of music and the moving image.

Natures 3B from Quayola on Vimeo.

This evening I watched Nature — Mira Calix and Quayola’s audio visual piece which took video footage of flowers blowing in the wind and used motion tracking technology to generate music ...

read more
Fork me on GitHub