Modern software development is sometimes colourfully described as being similar to firing tracer bullets at a target. Rather than spending time doing a lot of research, design and specification up front, the smallest, simplest version of the software is built and the feedback gathered from its use is used to improve the next version. Being able to continuously integrate, test and deploy software means that we can make this feedback loop incredibly tight but because the same engineers are developing the software and the tests we constantly have to think carefully about the testing investments we make and their opportunity costs.

Software engineers allow themselves to use knowledge of how the software works to pick good white box tests. They rely on code reviewers to catch important test cases that they’ve missed. They rely on unit tests to exercise pathological cases which are hard to exercise in end-to-end or integration tests. They use bugs discovered in previous releases to inform tests cases developed for the next. The goal is to ensure that the next version of the software provides feedback on how close it is to the target rather than just exploding on impact.

The decisions around which tests to write are hard because tests have costs. They take time to write and maintain and need to be changed and updated along with the software they test. What would happen if some or all of the tests could be free?

This question has tantalised computer scientists since Turing’s work on the halting problem in the 1930s and while academic progress has been made, tools based on that work have tended to either only work on small codebases; perform trivial analysis; generate a high percentage of false positives; produce difficult to understand reports or impose other overheads on the development process that make the resulting testing pretty far from free. As a result while we theoretically know how to do sophisticated analysis, in practice we tend to rely mostly on linters doing trivial checks alongside automated tests which check a handful of paths through the software written by engineers.

So it was wonderful when my mixture of hope, anticipation and scepticism gave way to delight when the Infer static analyser delivered on the promise of useful, non-trivial static analysis at Facebook. While Infer currently only reports on a small subset of the errors that it can detect, for those classes of errors Infer’s output is generally as useful as seeing a failing test case: Infer’s 80% fix rate on the non-trivial bugs it finds and reports on is extraordinary in an environment like Facebook’s.

Infer and tools like it won’t completely replace test cases written by engineers. To paraphrase Cristiano, without application specific information, no formula will be able to determine whether a piece of software is working as intended. As a companion to test suites though, static analysis will be transformative. It’s likely that sophisticated static analysers will soon be used by everyone from the smallest software engineering teams to the biggest tech companies. When high quality tests are free, why wouldn’t you run them?

Working with Infer in production at Facebook over the last year has been like living in the future. It has changed the way I think about testing and about the testing investments that should be made. I’m incredibly excited to see its public release as an open source tool that can be used by everyone and very grateful to have been able to help distribute this future more widely.


Investing In Testing

Wed 10 June 2015 by Jim Purbrick

Droidcon London

Last year I was talking to an engineer at Droidcon London who was working on an Android app with 100% test coverage. I immediately asked whether he thought 100% test coverage was worthwhile: many software engineering teams strive to achieve 100% test coverage, but few succeed because it’s an ...

read more

buckd

Mon 18 August 2014 by Jim Purbrick

BuckGraffiti

One of the things I’ve been working on since joining Facebook is Buck, an open source Android & Java build tool which is significantly faster than many other Java build tools for a number of reasons.

As well as being fast, Buck gains a lot of power and flexibility by ...

read more

Organisational Structures

Thu 20 March 2014 by Jim Purbrick

There have been a number of blog posts recently about exciting new organisational structures. As Cory points out “Every early stage company thinks it has reinvented management”: a very dangerous belief when betting on a new organisational structure can be much riskier than betting on the wrong product.

It starts ...

read more

Beyond Time Dilation?

Wed 29 January 2014 by Jim Purbrick

The Battle of B-R5RB

EVE online is a remarkable game. On Monday over 2000 people spent over 20 hours destroying virtual spaceships worth 200,000 USD in real money in what was the likely the largest battle in a video game ever. That EVE is capaple of supporting such large engagements is an amazing ...

read more

Osprey Therian

Sun 15 December 2013 by Jim Purbrick

In mid-2004 I first started exploring Second Life. Version 1.4 had just been released and Philip Rosedale had said in the press release “My fantasy is to be Uma Thurman in Kill Bill, and now I can. I’d pay $10 for her yellow jumpsuit and sword moves and ...

read more

Parse By The Sea

Sat 19 October 2013 by Jim Purbrick

#parsebythesea

A few weeks ago Facebook London hosted the Parse By The Sea hackathon at the Brighton Dome as part of the Brighton Digital Festival. The idea was to take one of our internal hackathons on the road and invite members of the public to join us, turning a hackathon in ...

read more

Facebook Hackathons

Mon 16 September 2013 by Jim Purbrick

I’ve been a big fan of hackathons since one of the first Yahoo! Hack Days I attended at Alexandra Palace was struck by lightning. The lightning caused the fire alarms to go off which opened the roof to let the torrential rain pour on to hundreds of geeks and ...

read more

Brighton Digital Festival

Wed 04 September 2013 by Jim Purbrick

The Brighton Digital Festival starts this week and I’m very happy to be helping out with Facebook London‘s contributions: Parse By The Sea, a mobile app Hackathon featuring Parse on the 26th of September, and helping to Connect The Brighton Digital Festival by sponsoring Metranet to provide high ...

read more

Final Score

Thu 04 July 2013 by Jim Purbrick

Google Reader

Using Reader on my HTC Wizard on the loo was probably responsible for my biggest increase in clue ever.

Goodbye Reader, you’ll be missed.

read more

One Universe, Many Scales

Thu 10 January 2013 by Jim Purbrick

One epic meta-game design I first remember talking about a decade ago while working on Warhammer Online is the multi-scale online game: a system of interconnected games in which you choose to be a solo operative, work in a small group, or command epic forces or huge space fleets and ...

read more

Creatarr

Wed 09 January 2013 by Jim Purbrick

cc image by vdu, j4mie

One of the things I’ve been tinkering with since leaving Linden Lab is Creatarr: a creative, collaborative social game. Creatarr’s goal is to bring some of the magical collaborative creation found in Second Life to a wider audience and to push creativity in ...

read more

Following In My Father’s Footsteps

Mon 12 November 2012 by Jim Purbrick

Tintin Hair

From 2 years before I was born, until just before I started working on Second Life at Linden Lab, my Dad worked at an innovative technology company with a large consumer photography business: Kodak. From January next year I’ll be working at an innovative technology company with a large ...

read more

Caching Shared, Private Data With Ningx

Sun 11 November 2012 by Jim Purbrick

As with many other social services, a large amount of the data in EVE Online and Dust 514‘s New Eden universe is shared between subsets of users. Some corporation data should only be accessible to the corporation’s members, market prices should only be accessible to capsuleers and infantry ...

read more

Adding Vary Header Support To Nginx

Sun 14 October 2012 by Jim Purbrick

Although Nginx supports proxy caching it doesn’t provide support for the HTTP Vary header out of the box. This is a problem if you want to use Nginx to proxy different versions of the same URI which Vary on Content-Language or proxy different representations of a RESTful resource specified ...

read more

Load Balancing Stateful Services With Nginx

Mon 30 July 2012 by Jim Purbrick

The EVE online network architecture uses stateful proxy servers which manage sessions for players connected to the cluster via the EVE client. The client sends requests to the proxy which are forwarded on to sol servers maintaining the game state and the sols send notifications to the proxy which are ...

read more

Brighton Mini Maker Faire: The Movie

Thu 24 May 2012 by Jim Purbrick

A great video of the Brighton Mini Maker Faire last year by Andrew Sleigh showing the making of You’re The Boss 2. Applications for this year’s Maker Faire are now open and I can’t wait to see what everyone comes up with this year!

read more

Super Hyperpolyglot

Sat 05 May 2012 by Jim Purbrick

A few years ago nearly all the code I wrote was in C++, but increasingly I’m finding myself writing in a variety of mostly C-style languages and having to perform crunching mental gear changes as I switch between them.

In the interests of making these language switches less painful ...

read more

100 robots Vs The Audience

Wed 04 January 2012 by Jim Purbrick

A couple of years ago I had great fun putting together the London Geek Community iPhone OSCestra at Open Hack London and I’ve been controlling Ableton Live with iPhone tapped to my guitar as part of 100 robots for a couple of years now so when @andybudd suggested I ...

read more

100 robots Attack!

Fri 09 December 2011 by Jim Purbrick

Lots of exciting 100 robots news! Our debut album, Attack!, has been professionally mastered by Chris at Melograf Mastering who has done an amazing job and made the album sound incredible. The new version is already available at bandcamp and will be available on itunes, amazon and many other download ...

read more
Fork me on GitHub