As well as being fast, Buck gains a lot of power and flexibility by using Python to generate build rules. Once projects become very large, however, this can become a problem as Buck has to execute thousands of python scripts to build its dependency graph before it can start its parallel build process. When I started working on Buck this parse phase could last tens of seconds. Buck was already much faster than Ant, but test driven development could be painful.
Our initial work focussed on making the parsing step faster and after some experimentation with Jython we discovered that bigger improvements could be made by running a long lived Python process which could be handed build files to execute as required.
As is often the case, the bulk of the improvements could be made by caching. Build files change far less often than source files, so caching the build file output avoids the need to spend a lot of time parsing in the common case when only a small number of source files change. After spending some time looking at serialising the build file output to disk it became clear that a more effective approach would be to cache the output in memory by running Buck as a long lived server process using Nailgun.
Nailgun is a client,
protocol, and server for running Java programs without incurring the
JVM startup overhead. Nailgun makes converting Java applications to
client-server architectures as simple as passing the name of the class
Main method to the nailgun Server and
client application. Early experiments running Buck with Nailgun showed
a lot of promise, allowing us to reduce parse time to close to zero,
but running buck as a server invalidated several assumptions that
required a non-trivial amount of work to fix.
The environment had to be threaded through from the client and calls
could no longer be used for garbage collection, so resource lifetimes
had to be managed with
blocks and Nailgun needed to be extended to detect client
disconnections which could be thrown as
to ensure that killing the Nailgun client cancelled builds as
expected. It’s also worth noting that for large, long running
applications like Buck the JVM started overhead saved by Nailgun is
not significant, but the time saved by the long running process being
able to maintain a JIT cache of Java class files is.
With Buck running as a long running server process the next step was to make it correctly invalidate cached build rules when build files changed. In order to avoid building outputs each time a file is touched, buck hashes the contents of input files to see if they have actually changed. While this saves a lot of time when switching between similar source control branches it requires reading each input file each time a build is run: something which was adding several seconds to the per-build overhead that we were trying to reduce.
To avoid this overhead we switched to a composite approach which watches the file system for changes and then checks the hashes of the contents of changed files. In the case where a few files are edited only a few hashes are generated and compared, in the case where source control systems touch many files without changing their contents comparing hashes avoids unnecessary rebuilding.
Initially we used the standard Java WatchService to generate file change events, but found that in practice the latency between changing a file and the FileWatcher generating events was far too high. Luckily wez, sid0 and friends had built Watchman which provides very low latency file change events and an easy to use JSON based interface which only took a day to wire in to Buck. Watchman is an amazing piece of technology, but requires some tweaking of OS settings to work well, so if you notice Buck taking a long time to parse project files you may need to check the system specific preparation.
When combined with exopackage and a number of other optimisations, the benefits of the Buck daemon are significant. Trivial builds now take a small fraction of the time they used to and in some cases it’s possible to incrementally build and install of an app in less time than it takes some build systems to do a no-op build.
I’ve had a great time working with the amazingly talented Buck team and I’m very happy to see buckd improving build times within Facebook and beyond.
Now it’s time to go back to writing a test, watching it fail and making it pass: with a lot less waiting around.
There have been a number of blog posts recently about exciting new organisational structures. As Cory points out “Every early stage company thinks it has reinvented management”: a very dangerous belief when betting on a new organisational structure can be much riskier than betting on the wrong product.
It starts ...read more
Beyond Time Dilation?
EVE online is a remarkable game. On Monday over 2000 people spent over 20 hours destroying virtual spaceships worth 200,000 USD in real money in what was the likely the largest battle in a video game ever. That EVE is capaple of supporting such large engagements is an amazing ...read more
In mid-2004 I first started exploring Second Life. Version 1.4 had just been released and Philip Rosedale had said in the press release “My fantasy is to be Uma Thurman in Kill Bill, and now I can. I’d pay $10 for her yellow jumpsuit and sword moves and ...read more
Parse By The Sea
Brighton Digital Festival
The Brighton Digital Festival starts this week and I’m very happy to be helping out with Facebook London‘s contributions: Parse By The Sea, a mobile app Hackathon featuring Parse on the 26th of September, and helping to Connect The Brighton Digital Festival by sponsoring Metranet to provide high ...read more
One Universe, Many Scales
Following In My Father’s Footsteps
From 2 years before I was born, until just before I started working on Second Life at Linden Lab, my Dad worked at an innovative technology company with a large consumer photography business: Kodak. From January next year I’ll be working at an innovative technology company with a large ...read more
Caching Shared, Private Data With Ningx
As with many other social services, a large amount of the data in EVE Online and Dust 514‘s New Eden universe is shared between subsets of users. Some corporation data should only be accessible to the corporation’s members, market prices should only be accessible to capsuleers and infantry ...read more
Adding Vary Header Support To Nginx
Although Nginx supports proxy caching it doesn’t provide support for the HTTP Vary header out of the box. This is a problem if you want to use Nginx to proxy different versions of the same URI which Vary on Content-Language or proxy different representations of a RESTful resource specified ...read more
Load Balancing Stateful Services With Nginx
The EVE online network architecture uses stateful proxy servers which manage sessions for players connected to the cluster via the EVE client. The client sends requests to the proxy which are forwarded on to sol servers maintaining the game state and the sols send notifications to the proxy which are ...read more
Brighton Mini Maker Faire: The Movie
A few years ago nearly all the code I wrote was in C++, but increasingly I’m finding myself writing in a variety of mostly C-style languages and having to perform crunching mental gear changes as I switch between them.
In the interests of making these language switches less painful ...read more
100 robots Vs The Audience
100 robots Attack!
Lots of exciting 100 robots news! Our debut album, Attack!, has been professionally mastered by Chris at Melograf Mastering who has done an amazing job and made the album sound incredible. The new version is already available at bandcamp and will be available on itunes, amazon and many other download ...read more
Data Is Not Art
This week I experienced two remarkable combinations of music and the moving image.
This evening I watched Nature — Mira Calix and Quayola’s audio visual piece which took video footage of flowers blowing in the wind and used motion tracking technology to generate music ...read more